Predicted entities appear as colour highlighted text, with a different colour appearing for each different entity type. Once an entity has been confirmed by a user, by either manually applying it or accepting a prediction, the entity will appear as highlighted text with a bold, darker outline, as shown below.
Entity format example
Please Note: There are two very important best practices to remember when accepting, rejecting or applying entities within verbatims:
1. Don't split words
It’s important not to split words – the highlighted entity should cover the entire word (or several) in question, not just part of it (see the incorrect example on the left below, and the correct application on the right)
Incorrect (left) and correct (right) examples of the ‘Organisation’ entity being applied
2. Don't partially label paragraphs
When labelling, if a user assigns one label to a verbatim, they should apply ALL labels that could apply to that verbatim, otherwise you teach the model that those other labels should not apply. For entities, the same is true, except entities are reviewed or applied at the paragraph level, rather than the whole verbatim.
Paragraphs in a verbatim are separated by new lines. The subject line of an email verbatim is considered its own single paragraph.
Make sure to review or apply all of the entities within a paragraph across all entity kinds if you review or apply one of them. Applying, accepting or rejecting entities in a paragraph means that the paragraph is treated as ‘reviewed’ by the platform from an entity perspective. Therefore, it’s important to accept or reject ALL of the predictions in that paragraph.
The example below shows the different paragraphs that have been reviewed within the email verbatim.
Figure: Example email verbatim showing reviewed entities across multiple paragraphs
The verbatim shown below shows an example where a user has not accepted or rejected all of the entity predictions in a single paragraph. This is incorrect, as the model will falsely treat the date entity prediction as an incorrect prediction.
Figure: Example email verbatim that has not been properly reviewed
Accepting and rejecting entity predictions
Once entities are enabled (see here), Re:infer will automatically start predicting them within the verbatims throughout your dataset. Users can then accept the predictions that are correct or reject them where they are incorrect. Each of these actions sends training signals to Re:infer that will be used to improve the platform’s understanding of that entity.
For the entities that are trained offline (all excluding ‘Organisation’ and ‘Person’), it is more important from an improvement perspective for users to reject or correct wrong predictions than it is for them to accept correct predictions.
For the entities that train live in the platform (currently ‘Organisation’ and ‘Person’), it is equally important to accept correct predictions as well as reject incorrect predictions. You do not, however, need to accept more than a few correct examples per unique entity for these kinds (e.g. Example Bank Ltd. is a unique organisation entity). The caveat to this is that if an entity is listed many times in the same paragraph, you need to accept/reject ALL of them.
To review an entity prediction, hover the mouse over the prediction and the entity review modal will appear, as shown in the example below. To accept it, click the icon, to reject it, click the icon.
Please Note: It’s important that you reject incorrect entity predictions, but if the highlighted text was in fact a different entity (this would be more common for date-related entities) that you apply the correct one afterwards (see below on how to apply entities).
Figure: Example verbatim with entity predictions and entity review modal
To apply an entity to some text where Re:infer may not have predicted it, users simply need to highlight the section of test like you would if you were going to copy it.
A dropdown menu will appear, as shown below, containing all of the entities that you have enabled for your dataset. Simply click the correct one to apply it.
Figure: A generated example verbatim showing entity application modal
Once an entity has been applied, it will be highlighted in colour with a bold outline (see below). Each entity type will have its own specific colour.
Figure: A generated example verbatim showing an applied ‘Value Date’ entity