Finding best neural network structure using optimization algorithms and cross-validation
Hi. I'm using optimization algorithm to find best structure+inputs of a 'patternnet' neural network in MATLAB R2014a using 5-fold cross validation. Where should i initialize weights of my neural network? *Position_1(for weight initialization)* for i=1:num_of_loops *Position_2(for weight initialization)* - repeating cross validation for i=1:num_of_kfolds *Position_3(for weight initialization)* - Cross validation loop end end I'm repeating 5-fold cross validation (because random selection of cross validation) to have more reliable outputs (average of neural network outputs). Which part is better for weight initialization (Position_1,Position_2 or Position_3) and why?
John Michell answered .
2025-11-20
Nw = (I+1)*H+(H+1)*O
exceed the number of training equations
Ntrneq = Ntrn*O
This will occur as long as H <= Hub where Hub is the upperbound
Hub = -1+ceil( (Ntrneq-O) / (I+O+1) )
Based on Ntrneq and Hub I decide on a set of numH candidate values for H
0 <= Hmin:dH:Hmax <= Hmax numH = numel(Hmin:dH:Hmax)
and the number of weight initializations for each value of H, e.g.,
Ntrials = 10
If the training target is ttrn = t(indtrn), the mean-square-error of a naïve constant output net (independent of the input) is
MSEtrn00 = mean(var(ttrn',1));
Then using MSEtrn00 as a normalization reference, I use the following double loop format
rng(0)
j=0
for h = Hmin:dH:Hmax
j=j+1
net = ...
net.divideFcn = 'dividetrain';
...
for i = 1:Ntrials
net = configure(net,x,t);
...
[ net tr y e ] = train(net,x,t);
...
R2(i,j) = 1-mse(e)/MSEtrn00;
end
end