Why do I see a drop (or jump) in my final validation accuracy

Question

Why do I see a drop (or jump) in my final validation accuracy when training a deep learning network?

Neeta Dsouza · Accepted Answer

If the network contains batch normalization layers, the final validation metrics are often different from the validation metrics evaluated during training. This is because the network undergoes a 'finalization' step after the last iteration to compute the batch normalization layer statistics on the entire training data, while during training the batch normalization statistics are computed from the mini-batches.
 
If in addition to batch normalization layers the network contains dropout layers, the interaction between these two layers can aggravate this issue, as described here: https://arxiv.org/abs/1801.05134
If one removes the batch normalization (and dropout) layers from the network, the 'final' accuracy should be the same as the last iteration accuracy.
 
Increasing the size of the mini-batches can also alleviate this issue, since the statistics from a larger mini-batch may be better estimates of the entire training data statistics.

Why do I see a drop (or jump) in my final validation accuracy

Need a Custom Version or Complete Simulation for This Problem?

how can I overcome on algebraic loop in simulink

How to create an attention layer for deep learning networks?

How do I generate a pulse train which starts at the origin using functions

Nvidia NVLink in MATALB using multiple Geforce 2080 ti cards

How to train labels for image blocks using neural network?

Using a python trained neural network

Modeling and Simulation of an Induction Motor Using Sliding Mode Control in MATLAB

Solar Inverter with BMS in MATLAB Simulink Complete Simulation Project

Simulation and Kinematic Analysis of a 5-DOF RRRRRP Robotic Manipulator Using MATLAB

Get a Free Consultation or a Sample Assignment Review!