Add ImageNet example

[mkuchnik]

ImageNet is a common workload for benchmarking. This PR adds a recipe for running ImageNet in PyTorch with a Croissant loader, which can be useful for both system and ML characterization.

Current WIP status is:

• ImageFolder to Croissant prototype.
• Functional equivalence of Croissant loader with validation loader.

TODO:

• Add training loader with shuffling.
• Optimize system performance (e.g., startup for validation loader is ~30 seconds compared to near instant).

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[mkuchnik]

@marcenacp Thoughts on the structure so far? I'm not sure if there is a better way to read a local directory---I essentially emulate a zip/tar by creating a no-op "extract" function. I plan to simplify this further to avoid the symbolic links. I am also wondering if you have any thoughts on how to add shuffling/sharding of rows in mlcroissant, which would help with various node and worker-level parallelism strategies.

[marcenacp]

Why do you need this function if _soft_link == os.symlink?

[marcenacp]

Do we need to copy this from pytorch into mlcroissant? Could you alternatively refer the user to https://github.com/pytorch/examples/blob/2d725b6ab255e05c55e0b08925f06f171aaedc0c/imagenet/main.py?

[marcenacp]

What's the difference with

croissant/datasets/1.0/recipes/read_from_directory.json

Lines 51 to 59 in e532c4c

	"distribution": [
	{
	"@type": "cr:FileSet",
	"@id": "files",
	"name": "files",
	"encodingFormat": "text/plain",
	"includes": "data/file*.txt"
	}
	],

?

[marcenacp]

@mkuchnik Sorry for the late review... I think it's super interesting and I left a few comments:

• There is already a way to import files from a specific folder (data/). I linked to an example.
• Maybe it's confusing to copy a whole Python file from pytorch. Can you give more instructions? Maybe it's worth having a Jupyter Notebook recipe with an end-to-end example on ImageNet!