Below are outlined are some of the main reasons why a label may have low average precision, as well as a suggested solution to improve it:

 

1.  The training set size may be too small

 

• If the training set size is quite small, it may be that you just need to provide more training examples for the model
• Continue training the label using the methods outlined in the Explore phase, particularly 'Shuffle' and 'Teach label'

 

2.  The label may have been applied inconsistently or incorrectly to some of the verbatims

 

• It can often be the case that a user’s definition of a label changes over time, and older reviewed verbatims with that label may need revisiting to see if the label still applies
• Alternatively, if there are multiple users training a dataset, they could have interpretations of what each label means, and send mixed signals to the model
• To determine whether this is the case, users can use 'Check label' and 'Missed label' training modes to go through the reviewed verbatims for the label, and see where a label has been applied incorrectly, or missed unintentionally
• Users can then correct any errors and update labels to ensure consistency
• Going forward, if there are multiple users training a dataset, they should ensure that they are fully aligned on how they define the intents or concepts covered by each label

 

3.  The intent or concept that the label is intended to capture may be vague or very broad and hard to distinguish from other labels

 

• If a label is used to capture a very broad or vague intent or concept, it can be hard for the model to identify why that label should apply to a verbatim – it may then try to apply it to far too many verbatims
• Try not to be too generic when creating a label; it needs to be identifiable and distinguishable from other labels

 

4.  Alternatively, the intent or concept could be very specific or have too many layers in its hierarchy

 

• Trying to be too specific or adding many layers to a label’s hierarchy can make it too difficult for the model to detect, or distinguish it from previous layers
• The level of specificity for a label should match the content of the verbatims. If it is too specific to realistically distinguish from other similar labels in the hierarchy, the model may get confused
• In most cases, it is best practice to have three layers or less in a label’s hierarchy – i.e. [Root label] > [Connecting label] > [Leaf label]

 

5.  There may be several labels in the taxonomy that heavily overlap and the model struggles to distinguish between the two

 

• If you have two labels that are very similar and hard to distinguish from one another, it can confuse the model, as it won’t know which of the two labels applies
• In these instances, consider merging the labels
• Alternatively, go through the reviewed verbatims for each and make sure that the concepts are applied consistently and are distinct from one another

 

6.  The verbatims with that label applied may mostly be very similar or identical, and the model struggles to detect different ways of expressing the same intent or concept

 

• You should ensure that for every label you provide the model with multiple training examples that include various different ways of expressing the intent or concept that the label is intended to capture

 

7.  The intent or concept captured by that label is not semantically inferable from the text of the verbatim or it’s supporting metadata

 

• It is common for users to label a verbatim based on their own business knowledge of the context or process that would follow, and not on the actual text or metadata of the verbatim
• For example, an SME user may know that because the communication has come from a certain individual, it must be about a certain topic, even though nothing else in the text or metadata clearly indicates that the label should apply
• In this instance, users should only apply the label if the model would be able to detect it from the text or metadata, without this inside knowledge

 

---

Previous: Understanding and improving model performance     |      Next: Training using 'Check label' & 'Missed label'