User permissions required: 'View Sources' AND 'View Labels'


Overview

 

Please Note: This section assumes an understanding of precision and recall. For a full explanation of these, please see here.

 

The Validation page is a useful tool for helping you to improve the performance of the model associated with your data.

 

The page shows a chart of average precision per label versus their training set size, so you can easily see which labels are not performing as well as others, and then do some further training to improve their precision.

 

By hovering over any of the points on the chart, you can see the label name, its average precision, and the training set size. 

 

Whilst it obviously varies between datasets, any label with an average precision above ca. 75% is considered ‘healthy’, but a label’s performance should also be viewed relative to that of the other labels in the taxonomy. Labels with an average precision below ca. 75% should be worked on to improve their scores (see below).

 

Poorly performing labels will typically be in the bottom left-hand section of the chart, such as the ‘Request > Agreement’ example shown below.

  

 

 

Average precision per Label chart

 


Understanding and improving label performance:

 

Below are outlined the main reasons why a label may be performing poorly (typically indicated by a low average precision score), as well as a suggested solution to improve it:

 

1.  The training set size may be too small

  • If the training set size is quite small, it may be that you just need to provide more training examples for the model
  • Continue training the label using the methods outlined in the model training guide

 

2.  The label may have been applied inconsistently or incorrectly to some of the verbatims

  • It can often be the case that a user’s definition of a label changes over time, and older reviewed verbatims with that label may need revisiting to see if the label still applies
  • Alternatively, if there are multiple users training a dataset, they could have interpretations of what each label means, and send mixed signals to the model
  • To determine whether this is the case, users can use the Teach function to go through the reviewed verbatims for the label, and see where a label has been applied inconsistently or incorrectly
  • Users can then correct any errors and update labels to ensure consistency
  • Going forward, if there are multiple users training a dataset, they should ensure that they are fully aligned on how they define the intents or concepts covered by each label

 

3.  The intent or concept that the label is intended to capture may be vague or very broad and hard to distinguish from other labels

  • If a label is used to capture a very broad or vague intent or concept, it can be hard for the model to identify why that label should apply to a verbatim – it may then try to apply it to far too many verbatims
  • Try not to be too generic when creating a label; it needs to be identifiable and distinguishable from other labels

 

4.  Alternatively, the intent or concept could be very specific or have too many layers in its hierarchy

  • Trying to be too specific or adding many layers to a label’s hierarchy can make it too difficult for the model to detect, or distinguish it from previous layers
  • The level of specificity for a label should match the content of the verbatims. If it is too specific to realistically distinguish from other similar labels in the hierarchy, the model may get confused
  • In most cases, it is best practice to have three layers or less in a label’s hierarchy – i.e. [Root label] > [Connecting label] > [Leaf label]

 

5.  There may be several labels in the taxonomy that heavily overlap and the model struggles to distinguish between the two

  • If you have two labels that are very similar and hard to distinguish from one another, it can confuse the model, as it won’t know which of the two labels applies
  • In these instances, consider merging the labels
  • Alternatively, go through the reviewed verbatims for each and make sure that the concepts are applied consistently and are distinct from one another

 

6.  The verbatims with that label applied may mostly be very similar or identical, and the model struggles to detect different ways of expressing the same intent or concept

  • You should ensure that for every label you provide the model with multiple training examples that include various different ways of expressing the intent or concept that the label is intended to capture

 

7.  The intent or concept captured by that label is not semantically inferable from the text of the verbatim or it’s supporting metadata

  • It is common for users to label a verbatim based on their own business knowledge of the context or process that would follow, and not on the actual text or metadata of the verbatim
  • For example, an SME user may know that because the communication has come from a certain individual, it must be about a certain topic, even though nothing else in the text or metadata clearly indicates that the label should apply
  • In this instance, users should only apply the label if the model would be able to detect it from the text or metadata, without this inside knowledge



Previous: Understanding Model performance      |     Next: Precision & recall explained