Question
Due to the limitation of GPU memory, a deeplearning network can't learn like 16 samples in a batch. So can I compute the gradients for a batch of 8 samples, and update the network gradients with 2 batches' gradients? If I compute the gradients of a deeplearning network by [gradients,state,loss] = dlfeval(@modelGradient,dlNet,xTrain,yTrain); So after 2 batches, I get gradients1, gradients2, state1, state2, loss1, and loss2. For my instant opinion, I think the total gradients should be the mean of gradients1 and gradients2. But how can I compute the state values? Is it also the mean of state1 and state2?
Expert Answer
Neeta Dsouza
PhD Expert
Answered Aug 24, 2026
Yes, absolutely, just sum the gradients until your batch size is the size you want, then update the model. The principle is exactly the same is training a model on multiple GPUs (or CPUs).
The State update depends on the State. This example shows you how to aggregate batch norm state. The function aggregateState is of interest here. Instead of using gplus you would just be aggregating over your 'sub'-iterations.
function state = aggregateState(state,factor)
numrows = size(state,1);
for j = 1:numrows
isBatchNormalizationState = state.Parameter(j) =="TrainedMean"...
&& state.Parameter(j+1) =="TrainedVariance"...
&& state.Layer(j) == state.Layer(j+1);
if isBatchNormalizationState
meanVal = state.Value{j};
varVal = state.Value{j+1};
% Calculate combined mean
combinedMean = gplus(factor*meanVal);
% Caclulate combined variance terms to sum
combinedVarTerm = factor.*(varVal + (meanVal - combinedMean).^2);
% Update state
state.Value(j) = {combinedMean};
state.Value(j+1) = {gplus(combinedVarTerm)};
end
end
end
100% Run Guarantee
3-Hour Fast-Track Delivery
Need a Custom Version or Complete Simulation for This Problem?
Our 500+ PhD engineers build, debug, and optimize working MATLAB scripts and Simulink (.slx) models tailored to your exact assignment rubrics with zero plagiarism.
Tested on MATLAB R2024b / R2026a
Turnitin 0% Plagiarism Report
Free 7-Day Revisions Guarantee
Have a different question? Ask here
Related deep learning Questions & Solutions
Browse All →
Explore similar technical troubleshooting questions and verified MATLAB solutions:
Ready-to-Run MATLAB & Simulink Projects
Browse All Projects →