Community

Idea for next machine learning series - Keras with large datasets using generators/augmentation - nobody currently has a good guide for this online

Dear sentdex,

Love your videos.

Like you I previously used tflearn/tensorflow for my ML and am migrating to Keras.
One thing I've found is there is nowhere on the net that shows how to use Keras with a huge (as in many multiples GPU-sizes e.g. ImageNet) dataset with data-augmentation etc.

Keras has generators which can use HDF5 datasets for this but all of the examples I've found rely on infinite loops, are messy, and make data-augmentation very tricky.

This post is one of many examples of people struggling with it: https://github.com/keras-team/keras/issues/1627
Another: https://github.com/keras-team/keras/issues/68
And another: https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly.html
One user even asks about data augmentation and the response notes:
"I just checked the docs and it looks that Keras' ImageDataGenerator class is an end-to-end black box. No worries though as there is still the possibility to add these functionalities manually to DataGenerator."

After considerable searching I am yet to find a good tutorial and I suspect it would give you serious traffic to your excellent YouTube channel.

A big fan,

James