User permissions required: ‘View Sources’ AND ‘Review and label’

Please Note: Users will be able to see verbatims in Explore if they have ‘View sources’ AND labels if they have ‘View labels’ permissions, but they will require the ‘Review and label’ permission in order to actually use Explore to apply labels.


Introduction

 

In the second stage of the model training process, you will use Explore to further build your taxonomy and help the model better understand each of the labels by providing a wider variety of examples. This is the core phase of training and requires the most time and effort. 

 

The Explore page lets you filter by label, search for key terms, and toggle between reviewed and unreviewed verbatims. It also gives you an indication of which labels need more training examples to make better predictions. To use Explore, we advise the approach outlined below.



Key steps


 1.  Training using 'Shuffle'

 

  • Shuffle mode in Explore shows you a selection of 20 random verbatims that you can then review and apply labels to


  • Training using this feature is an important part of the training process, as it ensures that you label a representative sample of verbatims within your dataset, and do not bias the model too much by only training it on very specific and targeted concepts that you've searched for.


  • Using the dropdown in the top right-hand corner of the Explore page, you can select ‘Shuffle’ mode. This will show you 20 random verbatims. You can also then filter between ‘Reviewed’ or ‘Unreviewed’ using the left-hand filter menu

 

 

Dropdown menu to access ‘Shuffle’

 

  • When training a new model in Re:infer, we would recommend going through and labelling at least 5 - 10 pages worth of verbatims in shuffle, to help create a representative training set of reviewed examples and to ensure that your model performs well


Shuffle is a useful for a few other reasons:


  • It’s an easy way to get a sense of how the overall model is doing, and can be referred to throughout the training process. In a well-trained taxonomy, you should be able to go through any unreviewed verbatims on shuffle and just accept predictions to further train the model. If you find lots of the predictions are incorrect, you can see which labels require more training


  • Going through multiple pages on shuffle later on in the training process is a good way to check if there are intents or concepts that have not been captured by your taxonomy and should have been. You can then add existing labels where required, or create new ones if needed



2.  Reviewing label predictions

 

  • Explore lets you see all of the verbatims in the dataset, both reviewed and unreviewed. It also lets you search by individual labels or search for key terms, as well as set a variety of filters to restrict which verbatims appear

 

  • By clicking 'unreviewed' as shown below, you are presented with verbatims which have not been labelled by a human, but for which Re:infer may predict or suggest labels

 

 

Unreviewed verbatim filter

 

 

  • If you then select an individual label from the left-hand label filter bar, Re:infer will present you with unreviewed verbatims, presented in order of Re:infer’s confidence that this label applies, i.e. with the most confident first

  

 

 

Explore page filtered to ‘Rejection’ label

  

 

  • For example, the ‘Rejection’ label above has a low number of training examples, and so the top unreviewed verbatim only has a confidence rating of 51%

 

  • The transparency of the predicted label provides a visual indicator of Re:infer’s confidence. The opaquer it is, the higher Re:infer’s confidence

 

  • To pin a predicted label, simply click on it. To add a different or additional label, click the ‘+’ button and type it in

 

  • If you add a label in error, you can just hover over it and an ‘X’ will appear, click this to remove the label


 


Example label

 


  • Next to labels with less than 25 training examples you will notice a red training health circle, indicating that more training examples are required for Re:infer to be able to accurately estimate the performance of the label. The fuller and brighter the circle, the fewer the training examples there are

  

Training health circles



  • You should follow the steps just described to make sure that you’ve trained at least 25 verbatims per label, so that the platform can accurately evaluate how well it is able to predict that label


  • Once you reach 25 pinned examples you should see one of the below label performance indicators in place of the training health circle:
    • The grey circle is an indicator that the platform is calculating the performance of that label - it will update to either disappear, or an amber or red circle once calculated
    • Amber is an indicator that the label has slightly less than satisfactory performance and could be improved
    • Red is an indicator that the label is performing poorly and needs additional training / corrective actions to improve it
    • If there is no circle, then this means that the label is performing at a satisfactory level (though still may need improving depending on the use case and desired accuracy levels)
    • To understand more about label performance and how to improve it, you can start here




Label performance indicators

 

  • As you train the model, the number of pinned (or reviewed) verbatims per label is shown next to the label name

 

  • If you click the button at the top of the label filter bar, you can switch to the number of predicted verbatims with that label from the whole dataset. This is indicated by the  icon

 

 


Pinned vs. predicted Label count



Please Note: the predicted number is an aggregation of all the probabilities that Re:infer calculates for this label. For example, 2 verbatims with a confidence level of 50% would be counted as 1 predicted label



3.  Training using 'Search'

 

  • Explore, like Discover, also lets you search by key terms or phrases and then label verbatims that contain these. 

 

  • This is a really important way of finding a wider variety of examples that can apply to each label. If you know of key terms and phrases that would typically be associated with a label, you can search for them and then add the label to the search results.

 

  • To do this, simply type what you want to search into the search bar at the top and hit enter. Re:infer then highlights your search terms in yellow within the verbatims that match. You’ll notice that the mode in the dropdown in the top right-hand corner of the page automatically changes to ‘Text Search’


  • If you want to force an exact search simply add quotation marks (" ") around your search terms. 


  • While searching in Explore, Re:infer will also return an approximate number of verbatims that contain your search terms (see figure below), which is a really helpful way of gauging whether there will be lots of examples for a particular label, if there are some obvious words or phrases that would indicate that the label in question should apply.

 

 

 

Example search returning approximate matches

 


4.   Training using 'Low confidence'

 

  • Similarly, using the dropdown in the top right-hand corner of the Explore page, you can also select ‘Low confidence’ mode

 

 

Dropdown menu to access ‘Low predictions’



  • This will show you 20 verbatims for which Re:infer has either no predicted labels or labels predicted with a very low confidence. This means that for each individual label, the model typically has less than 10% confidence that the label applies to these verbatims

 

  • This is a useful tool to assess how well your current taxonomy and training covers the verbatims in your Dataset 

 

  • If you see verbatims which should have existing labels predicted for them, this is a sign that you need to complete more training for those labels 

 

  • If you see verbatims for which no current label is applicable, you may want to create new labels to capture them

 


Useful tips

  • The model will start to make predictions with only a few labelled verbatims, though for it to make accurate predictions, you should label at least 25 verbatims per label. Some will require more than this, it will depend on the complexity of the data, the label, and the consistency with which the labels have been applied


  • When labelling a verbatim, focus on the mentality of the person who has written it. You should be labelling the content based on the problem they are trying to solve (what is in the verbatim), not based on how you would solve it (using your own expertise). Re:infer needs to be able to infer from what is in the verbatim what the label should be


  • In Explore, you should also try and find verbatims where the model has predicted a label incorrectly. You should remove incorrect labels and apply correct ones. This process helps to prevent the model from making a similar incorrect prediction in future.



Next: Using Teach