CPU and GPU memory

[ptrblck]

@Hoang_Nam_Bui

When we start training the model (already in GPU memory), our input will be transferred from the CPU RAM to GPU memory right?

Yes

@Hoang_Nam_Bui

Then, will all the intermediate outputs (outputs after each Convolution-Activation-Pooling layer) be stored in GPU after forward propagation?

Yes, the outputs will use the same device as the inputs of an operation by default.

@Hoang_Nam_Bui

If so, can we store those outputs back in CPU memory in order to save the GPU memory?

Also yes. You can use CPU offloading as described here.