User permissions required: ‘View Sources’ AND ‘Review and label’

 

Predicted entities appear as colour highlighted text, with a different colour appearing for each different entity type. Once an entity has been confirmed by a user, by either manually applying it or accepting a prediction, the entity will appear as highlighted text with a bold, darker outline, as shown below.



Entity format example



Best Practice

 

Please Note: There are two very important best practices to remember when accepting, rejecting or applying entities within verbatims:

 

1. Don't split words


It’s important not to split words – the highlighted entity should cover the entire word (or several) in question, not just part of it (see the incorrect example on the left below, and the correct application on the right)



 

 

Incorrect (left) and correct (right) examples of the ‘Organisation’ entity being applied


 

2. Don't partially label paragraphs


When labelling, if a user assigns one label to a verbatim, they should apply ALL labels that could apply to that verbatim, otherwise you teach the model that those other labels should not apply. For entities, the same is true, except entities are reviewed or applied at the paragraph level, rather than the whole verbatim.

 

Paragraphs in a verbatim are separated by new lines. The subject line of an email verbatim is considered its own single paragraph.

 

Make sure to review or apply all of the entities within a paragraph across all entity kinds if you review or apply one of them. Applying, accepting or rejecting entities in a paragraph means that the paragraph is treated as ‘reviewed’ by the platform from an entity perspective. Therefore, it’s important to accept or reject ALL of the predictions in that paragraph.

 

The example below shows the different paragraphs that have been reviewed within the email verbatim.

 

 

A screenshot of a cell phone

Description automatically generated 

Figure: Example email verbatim showing reviewed entities across multiple paragraphs

  

  

The verbatim shown below shows an example where a user has not accepted or rejected all of the entity predictions in a single paragraph. This is incorrect, as the model will falsely treat the date entity prediction as an incorrect prediction.


 

A screenshot of a social media post

Description automatically generated

 

Figure: Example email verbatim that has not been properly reviewed

 


Accepting and rejecting entity predictions

 

Once entities are enabled (see here), Re:infer will automatically start predicting them within the verbatims throughout your dataset. Users can then accept the predictions that are correct or reject them where they are incorrect. Each of these actions sends training signals to Re:infer that will be used to improve the platform’s understanding of that entity. 

 

For the entities that are trained offline (all excluding ‘Organisation’ and ‘Person’), it is more important from an improvement perspective for users to reject or correct wrong predictions than it is for them to accept correct predictions. 

 

For the entities that train live in the platform (currently ‘Organisation’ and ‘Person’), it is equally important to accept correct predictions as well as reject incorrect predictions. You do not, however, need to accept more than a few correct examples per unique entity for these kinds (e.g. Example Bank Ltd. is a unique organisation entity). The caveat to this is that if an entity is listed many times in the same paragraph, you need to accept/reject ALL of them.

 

To review an entity prediction, hover the mouse over the prediction and the entity review modal will appear, as shown in the example below. To accept it, click the   icon, to reject it, click the   icon.

 

Please Note: It’s important that you reject incorrect entity predictions, but if the highlighted text was in fact a different entity (this would be more common for date-related entities) that you apply the correct one afterwards (see below on how to apply entities).


 

A screenshot of a social media post

Description automatically generated


Figure: Example verbatim with entity predictions and entity review modal

 


Applying entities

 

To apply an entity to some text where Re:infer may not have predicted it, users simply need to highlight the section of test like you would if you were going to copy it.

 

A dropdown menu will appear, as shown below, containing all of the entities that you have enabled for your dataset. Simply click the correct one to apply it.

 

 

A screenshot of a social media post

Description automatically generated


Figure: A generated example verbatim showing entity application modal

 

 

Once an entity has been applied, it will be highlighted in colour with a bold outline (see below). Each entity type will have its own specific colour.


 

A screenshot of a cell phone

Description automatically generated


Figure: A generated example verbatim showing an applied ‘Value Date’ entity

 


Previous: Requesting new entities