Size drastically increased when images converted to hdf5 format


I converted Kaggle Dog vs Cat dataset from image to hdf5 format, in order to fasten the learning time. To my surprise, 540MB training data converted to more than 30GB of data in hdf5 file format. Why is this so. Also I have not much space in my laptop, is there any other method to fasten my deep learning process without consuming too much space.


Posted 2021-01-07T07:50:07.410

Reputation: 544

1You probably didn't use any compression – couka – 2021-01-07T08:30:02.453

How much storage your data takes does not necessarily correlate with how fast your training takes. Also, for image data, it is highly discouraged to try to load all data in memory anyway due to their size. – Valentin Calomme – 2021-01-07T10:13:32.207

No answers