Using the 'Check label' and 'Missed label' training modes is the part of the Refine phase where you try to identify any inconsistencies or missed labels in the verbatims that have already been reviewed. This is different to the 'Teach' (Unreviewed) step, which focuses on unreviewed verbatims that have predictions made by Re:infer, rather than assigned labels.
'Check label' shows you verbatims where Re:infer thinks the selected label may have been misapplied - i.e. it potentially should not have been applied.
'Missed label' shows you verbatims that Re:infer thinks may be missing the selected label - i.e. it potentially should have been applied but wasn't. Here the selected label will typically appear as a suggestion, as shown in the image below.
'Missed label' mode with the 'Bedroom > Beds' label selected
The suggestions from Re:infer in either mode are not necessarily correct, these are just the instances where Re:infer is unsure based on the training that's been completed so far. You can choose to ignore them if you disagree with Re:infer's suggestions after reviewing them.
Using these training modes is a very effective way of finding occurrences where the user may have not been consistent in applying labels. By using them you are able to correct these occasions and therefore improve the performance of the label.
When to use 'Check label' and 'Missed label'?
As a rule of thumb, any label that has a significant number of pinned examples but has low average precision (which can be indicated by red label warnings in the Validation page or in the label filter bars) will likely benefit from some corrective training in either 'Check label' and 'Missed label' mode.
When validating the performance of a model, Re:infer will determine whether it thinks a label has often been applied incorrectly, or where it thinks it's been regularly missed, and will prioritise whichever corrective action it thinks would be most beneficial for improving a label's performance.
For more detail on improving label performance and when you may need to use 'Check label' or 'Missed label', see here.
'Missed label' is also a very useful tool if you've added a new label to an existing taxonomy with lots of reviewed examples. Once you've provided some initial examples for the new label concept, 'Missed label' can quickly help you identify any examples in the previously reviewed verbatims where it should also apply. See here for more detail.
How to use 'Check label' and 'Missed label':
To reach either of these training modes, there are two main options:
- If it is a recommended action in Validation for a label, the action card acts as a link that takes you directly to that training mode for the selected label
- Alternatively, you can select a label from the label filter and then select either 'Check label' or 'Missed label' from the dropdown menu in the top right-hand corner of the page
Explore training mode dropdown once a label has been selected
Please Note: You must first select a label before either 'Check label' or 'Missed label' will appear in the dropdown menu. Both of these modes also disable the ability to filter between reviewed and unreviewed verbatims, as they are exclusively for reviewed verbatims)
In each mode, Re:infer will show you up to 20 examples per page of reviewed verbatims where it thinks the selected label may have been applied incorrectly ('Check label') or may be missing the selected label ('Missed label').
In 'Check label' review each of the examples on the page to confirm that they are genuine examples of the selected label. If they are, move on without taking action. If they are not, remove the label (by clicking the 'X' when hovering over it) and ensure you apply the correct label(s) instead.
Review as many pages of reviewed verbatims as necessary to identify any inconsistencies in the reviewed set and improve the model's understanding of the label. Correcting labels added in error can have a major impact on the performance of a label, by ensuring that the model has correct and consistent examples from which to make predictions for that label.
In 'Missed label' review each of the examples on the page to see whether the selected label has in fact been missed. If it has, click the label suggestion (as shown in the image above) to apply the label. If it has not, ignore the suggestion and move on.
Just because Re:infer is 'suggesting' a label on a reviewed verbatim, does not mean the model considers it to be a prediction, nor will it count towards any statistics on the number of labels in a dataset. If a suggestion is wrong, you can just ignore it.
Review as many pages of reviewed verbatims as necessary to identify any examples in the reviewed set that should have the selected label but do not. Partially labelled verbatims can be very detrimental to the model's ability to predict a label, as when you do not apply a label to a verbatim, you essentially tell the model 'this is not an example of this label concept'. If it is in fact a correct example, this can be very confusing for the model, particularly if there are other very similar examples that DO have the label applied.
Adding labels that have been missed can therefore have a major impact on the performance of a label, by ensuring that the model has correct and consistent examples from which to make predictions for that label.
Once the model has had time to retrain after your corrective training in these modes, you can check back in Validation to see the positive impact your actions have had on the Model Rating and the performance of the specific labels you've trained.