Strange out of memory while loading lots of pictures before input CNN for deep learning

Question

I’m using the TFlearn, and want to classify pictures to two category. But the strange out of memory while loading lots of pictures before input CNN for deep learning. The RAM is 64 G in my deep learning box. I using 482426 pictures(resize to 224 x 224) for training (241213 are A category, the other 241213 are B category ). The process shows the '%MEM: 98.4' whenever the I load these images, so that I can't complete the process. I would like to know how can I edit my code for this situation?

imgs = []

for filename in glob.glob(Learning_Data_Path+"/Training/A/"+"*.tif"):
    img = load_image(filename)
    img=img.resize((224,224))
    img_arr = np.asarray(img) 
    imgs.append(img_arr)


for filename in glob.glob(Learning_Data_Path+"/Training/B/"+"*.tif"):
    img = load_image(filename)
    img=img.resize((224,224))
    img_arr = np.asarray(img) 
    imgs.append(img_arr)


imgs = np.array(imgs)


y_data = np.r_[np.c_[np.ones(Training_A_num), np.zeros(Training_A_num)],np.c_[np.zeros(Training_B_num), np.ones(Training_B_num)]

score 0 · Answer 1 · answered Apr 05 '20 at 11:50

You should load images in batches. 480K of 224*224 sized images is too large for any machine let alone a 64GB one.

You can use image_generators from keras or tf.keras

https://www.tensorflow.org/tutorials/load_data/images#load_using_keraspreprocessing

https://keras.io/preprocessing/image/

score 0 · Accepted Answer · answered Apr 05 '20 at 14:24

You have about 700K of 224 X 224 X3 images. If you try to put all these into memory at once you will get a resource exhaust error. The solution for handling such large data sets is to feed the training data to your network in "batches". Keras provides utilities to make this process easy. Use the Keras ImageDataGenerator.flow_from_directory to fetch data of a specified batch size from a directory containing your training data and provide it to your network. Documentation is here. Use the Keras model.fit_generator to train your model. Documentation for that is here

Strange out of memory while loading lots of pictures before input CNN for deep learning

2 Answers2