Health Scores for Context Themes

What’s covered in this guide? 

1. What is Context Theme Health and why does it matter? 
2. How we measure Context Theme Health 
3. How to check Context Theme Health
4. How to train Themes to improve Health 

Psst, if you need a reminder on how to set up Context Themes before getting stuck into Context Theme Health, head to our section on Creating Context Themes.

1. Context Theme Health

What is Context Theme Health?

Context Theme Health is a measure of how well your Context Themes are performing, showing how effectively they can identify items in your data that correspond to the relevant Themes. 

Think of our own minds - if we don’t continue working on them, training them to understand the mountains of new information we take in on a daily basis, they’re not going to stay healthy. Context Themes work in exactly the same way. As data is continually fed into the platform, we have to continually train Context Themes to understand it correctly. 

That’s where Context Theme Health comes in. It gives you visibility of all the magic the Wordnerds platform is doing in the background to process your data, meaning you can see what needs to be done to ensure your Themes stay relevant and accurate.

Why does Context Theme Health matter?

Context Theme Health saves time when training Themes, because it’s a metric that gives visibility of when they are trained to optimum levels. This means you know when you’re ready to enable a Theme. 

By improving your Theme Health, the platform will correctly sort your data to draw useful insights from. You’ll find more of what you’re actually looking for within your data. 

Theme Health helps you get the most out of your data, more efficiently. 💪

2. How we measure Context Theme Health

The key metrics

F1 Score	A measure of accuracy; a harmonic mean of Precision and Recall. A higher F1 score means items in your Theme are accurately balanced between relevant and precise (highest possible score is 1).
Precision	Precision tells you how many of the tagged items are actually relevant to your Theme. A high Precision score means the data being positively tagged correctly matches the Theme.
Recall	Recall indicates how many of the relevant items you've successfully tagged. A high Recall score means your Theme is identifying a high proportion of the relevant items from the Sample Batch.
Coverage	Coverage is determined by comparing the similarity of the positively and negatively data tagged in your Theme to the untagged data in the Sample Batch. A high Coverage score means high similarity between the tagged data in the Theme and the untagged data in the Sample Batch and therefore indicates a good balance and volume of positive and negative tagged items.
Effort	Effort indicates the volume of items you need to tag to create an effective Theme. A low effort score means you need to tag more items.
Overall Health Score	The Overall Health Score is the lowest score of the F1, Coverage, and Effort scores for that Theme.
Health Breakdown	The individual scores (F1, Coverage, Effort) making up your Overall Health Score.

 

3. How to check Context Theme Health

Refer to the Overall Health Score for a metric which summarises the health of each Theme. Use this score to benchmark your Themes and determine which Themes you need to work on with more training. 

Click on Health Breakdown to see the individual scores making up the Overall Health Score for a Theme. This will show you what to focus on to improve Theme Health. Your Coverage score may be top notch, but if Effort is letting the team down your Theme’s Overall Health Score will be affected.

4. How to train Themes to improve Health

So, now you know your Theme Health scores, what can you do to improve them?

It’s time to get training!

Click Edit Theme to view the Theme Training Screen, where you can tag your Context Theme to train it.

When training your Themes, the ultimate aim is to reach an Overall Health score of above 75%. 

Where to start

Check the Health Breakdown to see which metrics to work on to boost the theme’s Overall Health Score. 

 

Effort

The Effort score allows you to find the balance between overtraining and undertraining your themes. To improve your Effort score, go through more rounds of tagging for your Theme. Aim to complete between 8-10 rounds of training and try to hit the 100% mark! 

Without a high effort score, F1 score isn’t truly meaningful because volume and quality of data are unbalanced.

 

Coverage

Completing additional rounds of tagging will boost your Coverage score. Coverage can be affected when you generate a new Sample Batch - if you see your Coverage score drop, you may need to tag more items, even if your Effort score is at 100%.

You could also use the Examples section to paste in specific sentences that you’ve seen being misclassified in the data and tag them accordingly.

 

F1

You guessed it… completing additional rounds of tagging will boost your F1 score. It’s worth noting that there’s always a trade-off between optimising both Precision and Recall (which make up your F1 score). Based on the specific needs of your project, decide whether you're willing to accept some noise in your theme to capture every possible item (which will lower Precision), or you prefer a cleaner theme even if it misses some items (lower Recall).

 

Ready to go!

Once you’ve finished training, remember to click Enable Theme. After you’ve gone through training rounds, your tagged items will be stored but not applied to the project data until you click Enable Theme. 

 

More questions?

We’re always happy to chat. Reach out using the help bubble at the bottom of your screen or contact your Customer Success Manager. 

 

✍️ Article written by: Zoe, Customer Success Manager