Alpha_bravo - 2021-07-06T10:07:09+00:00
Question: How can I assess the reliability of my machine learning model on unseen data?
I have a model of a system that can detect some abnormalities and then react accordingly. Now, I want to analyze how reliable is our model in predicting these abnormalities. So far, I have manually analyse certain situations and assess whether the system reacted correctly or incorrectly. This is very time consuming and I would like to know how we could adopt supervised machine learning to train a neural network to make this assessment automatically.
Expert Answer
John Williams answered .
2025-11-20
In general, to create a machine learning model, you would:
1. Collect data.
2. Split the data into training, test and validation sets.
3. Train a machine learning model using both the training and test sets.
4. Validate that your trained model on the validation set to verify that it can still reliably predict "unseen" data.
5. Use the model to predict real world data.
From the workflow above, you can see that we can only assess the accuracy of the model (before really using it in real world) by evaluating the prediction it outputs on the validation set.
If the predicted values on the validation set is within some reasonable accuracy that you desire, then, you can use the model to predict real world data with the assumption that it would also predict these new data with the same level of accuracy.
Yet, the validation set itself had to first be manually collected and labeled.
Furthermore, it is counter-intuitive (if not impossible) to be able to *automatically *assess the accuracy of your model on new unseen (and unlabeled) data. If you have another model that can assess whether your existing model is predicting new data correctly vs. incorrectly, you would certainly have used that model instead.
Not satisfied with the answer ?? ASK NOW
