User permissions required: 'View Sources' AND 'View Labels'
Validation within Re:infer is the process by which the platform evaluates the performance of the model associated with a dataset by testing how well it is able to predict each label in the taxonomy, using a sub-set of training data from within that dataset.
How does it do this?
Re:infer first splits the reviewed (i.e. labelled) verbatims in the dataset into two groups; a majority set of training data, and a minority set of test data. In the image below, the coloured dots represent the labelled verbatims within a dataset. This split is determined by the verbatim ID when the verbatims are added to the dataset, and remains consistent throughout the life of the dataset.
The platform then trains itself using only the training set as training data.
Based on this training, it then tries to predict which labels should apply to the verbatims in the test set and evaluates the results for both precision and recall against the actual labels that were applied by a human user.
The validation page then publishes live statistics on the performance of the latest model version, but you can also view historic performance statistics for previously pinned model versions.
When does the validation process happen?
Every time you complete some training within a dataset, the model updates and provides new predictions across every verbatim. In parallel, it also re-evaluates the performance of the model. This means that by the time the new predictions are ready, new validation statistics should also be available (though one process can take longer than the other sometimes).
Please Note: The platform will always show you as default the latest validation statistics which have been calculated, and will tell you if new statistics are yet to finish being calculated.