VideoClassificationDataset does not support frame video folders

[Skylion007]

I realized some of the helper code around the pytorch video does not allow the videos to be frame_video objects, which is disappointing and is supported by the underlying pytorch video.

https://pytorchvideo.readthedocs.io/en/latest/_modules/pytorchvideo/data/labeled_video_dataset.html#labeled_video_dataset

Discussed in #1490

^{Originally posted by Skylion007 November 26, 2022}
Decoding videos from AVI / MP4 is eating up the CPU on my GPU server, is there a way to have the VideoClassificationData folder accept a folder of frames and maybe some metadata as the video instead? Or recommendation for a video codec that would be faster to decode?

[Skylion007]

So it looks like it does work if specify the folders manually from_files by globbing those dirs. However, the more intuitive from_folders classmethod appears to filter out all the folders that only contain JPEGs. Loading small samples from these frame_videos is also less efficient than I would have expected, which is a tad disappointing. It can even be slower than loading AVI encoded videos (but faster than H264 ones it seems).

[Borda]

do you have a specific suggestion on how to improve the user/developer experience?