How evaluate multi-step forecasting performance for large dataset?

Question

How evaluate multi-step forecasting performance for large dataset?   What I means is that to use NARX for forecasting I have to:   1. create closed loop network with   netc = closeloop(net); 2. construct target prediction data with targetSeriesPred = [targetSeries(end-delay+1:end), con2seq(nan(1,N))], i.e. use a number of nan equal to forecasting horizon 3. prepare data with [Xs,Xi,Ai,Ts] = preparets(netc,inputSeriesPred,{},targetSeriesPred); 4. get simulation/prediction results with yPred = netc(Xs,Xi,Ai); 5. evaluate forecasting performance (regarding olny one step-ahead prediction) with perf = perform(net,yPred,targetSeriesVal); Now suppose I have a large dataset of data concerning 3 years of hourly sampled data, I can utilize only the first year for traning, validation and test phase ('divideblock' of course). I don't know how to get a unique performance value for the remaining data, 2 years, for a forecasting horizon for example of only 6 samples. I hope I to have been clear enough. Thanks in advance for your replies/suggestions.

Kshitij Singh · Accepted Answer

How evaluate multi-step forecasting performance for large dataset? Asked by EanX 14Apr2014
 
 A. Design an openloop net
  1. Use training data and nncorr or fft to estimate
   a. Significant autocorrelation output feedback lags
   b. Significant input/output crosscorrelation input lags
  2. Use trial and error to determine number of hidden layer nodes
  3. Use divideblock in narxnet to obtain multiple designs
  4. Choose the best design based on validation set error.
  5. Use the test set on the best design to obtain an UNBIASED 
     estimate of performance 

 B. Convert to a closeloop design
  1. [ netc Xci Aci ]= closeloop(neto, Xoi, Aoi)
  2. Use preparets and test netc on the original data
  3. If performance is unsatisfactory, train netc intialized with 
     the initial conditions Xoi, Aoi and final weights of neto.
  4. To predict beyond the original data, use the data at the end of 
     the test set to intialize the delay buffer and a new external input.
  5. Use preparets and netc to predict beyond the test data.

How evaluate multi-step forecasting performance for large dataset?

Need a Custom Version or Complete Simulation for This Problem?

Prediction using narx Network

Error Message when using configured ANN (mismatch number of inputs)

Predictions using NARX Network

future value prediction using narx

How to train NARX neural network in closed loop

How to get R squared values in NARX Neural Network?

Modeling and Simulation of an Induction Motor Using Sliding Mode Control in MATLAB

Solar Inverter with BMS in MATLAB Simulink Complete Simulation Project

Simulation and Kinematic Analysis of a 5-DOF RRRRRP Robotic Manipulator Using MATLAB

Get a Free Consultation or a Sample Assignment Review!