Skip to content
English
Contact us
  • There are no suggestions because the search field is empty.

Train a context theme

Context themes are great when an issue could be described using different language, or the context in which the language is used is crucial to understanding what is being discussed

  1. Train a context theme
  2. Tip: How to write the best theme training prompts 
  3. Theme Health
  4. Context Theme Magic
  5. Tip: Choosing the best threshold for your theme
  6. Walkthrough video

🙋‍♀️ Before you get started

Before training a new context theme, you'll need to add a theme to your project. You'll also need to make sure you have some data in your project as context themes are trained using a sample batch of your own data. 

Train a context theme

1. To start training your context theme, you'll need to provide two examples as prompts: one example of what you want to find with your theme, and one example of something you don't want. For the positive example (what you do want to include in your theme), use a simple sentence of a customer describing the issue you are looking for. For the negative example (what you don't want in your theme), use a simple sentence of a customer talking about an issue that is not relevant to your theme and that you don't want included. Enter your prompts and click Save.

Remember, when theme training, a negative example is an example of what you don't want in your theme, it is *not* an example of something you *do* want in your theme being discussed negatively. 

💡 Tip: How to write the best theme training prompts 

⭐️ Keep your examples explicit; they should not require you to infer what the issue is

⭐️ Examples are not a list of keywords or instructions; they are examples of what a customer would say when discussing the issue, in their own words 

⭐️ Examples should be short, simple and include a single concept only

⭐️ If you want to find customers talking about two issues together, you will get more accurate results if you train separate context themes for each concept and analyse the overlap

⭐️ Although your negative prompt (the example of something you don't want) can be an example of anything that is not relevant to your theme, it can be more effective if you use an example of the same language being used in a context that makes it not relevant to your theme

⭐️ If you have good examples available, you can use direct customer verbatim as your prompts

Here's my example: Prompts for a context theme to find customers talking about the comfort of seats in airline reviews

The prompt used for my positive example is how I would expect a customer to discuss this issue in simple language: 'the seats were not comfortable'. Customers will often use different language to discuss the same issue - don't worry we'll find it, but for the first example, keep it simple.

The prompt used for my negative example is something else a customer could discuss in an airline review, but is not relevant to my theme of seat comfort. In this case, I've written a prompt with some of the same language ('not comfortable'), but used in a context that completely changes what the customer is actually talking about, and not relevant to my theme: 'I am not comfortable paying extra for hold luggage'. 

2. The two examples given as prompts in step 1 formed the first round of training for your context theme. In step 2, you will be given a Training List to manually code, which will complete the rest of your theme training. The Training List (on the left-hand side of the training screen) is made up of examples from your sample batch of training data. To complete a round of training, select the Yes or No icon for each example in the Training List to confirm whether this represents an example of what you want included in your theme, or not and click Retrain. 

Exclude confusing or ambiguous examples 

Examples that are confusing or ambiguous can mislead the theme's definition. In these cases, it is best to exclude the example from theme training. To do this, select the Maybe icon when you are tagging examples in your Training List. 

Training themes using your data 

The Training List, located on the left side of the training screen, consists of excerpts from your own data. Within this list, you will encounter a combination of examples you want in your theme and those you don't want. It is essential to code both negative ('No': examples you don't want) and positive ('Yes': examples you do want) examples, as this helps your theme comprehend not only what to include but also what to exclude.

3. On the right-hand side of the training screen, you will see Results. Results are examples from your sample batch training data, sorted by most (Top) to least (Bottom) likely to match your theme definition based on the training so far. You can see the Hit/Miss threshold by selecting Boundary. With each round of training, the results will refresh. 

4. Continue rounds of training until the Effort score on the Health Breakdown is at 100%, usually after 8-10 rounds. A high effort score indicates you have coded enough examples.

5. After you have coded enough examples and achieved an Effort score of 100%, you will also see how balanced the health of your theme is based on the Health Score. You are aiming for a score in the green range (55%+) and ideally over 75%. The Health breakdown will show how the health score is calculated. 

Theme Health 

Health Score

A health score shows you how healthy your theme is. To find out the Health Score of a theme, click the Edit pencil to enter the training screen for the relevant theme. The Health Score will display in a bar with a percentage indicating how effective your theme is based on the health metrics we measure, which you can view by opening the Health breakdown. 

Health breakdown

F1 Score: A measure of accuracy; a harmonic mean of Precision and Recall. When it comes to classifying text data into themes, there are two ways you can be wrong: you can categorise the wrong thing (precision fail) or miss the right thing (recall fail). 

Precision: Precision tells you how many of the tagged items are actually relevant to your Theme. A high Precision score means the data being positively tagged correctly matches the Theme.

Recall: Recall indicates how many of the relevant items you've successfully tagged. A high Recall score means your Theme is identifying a high proportion of the relevant items from the Sample Batch. 

Coverage: Coverage is determined by comparing the similarity of the positively and negatively data tagged in your Theme to the untagged data in the Sample Batch. A high Coverage score means high similarity between the tagged data in the Theme and the untagged data in the Sample Batch, and therefore indicates a good balance and volume of positive and negative tagged items.

Effort: Effort indicates the volume of items you need to tag to create an effective Theme. A low effort score means you need to tag more items.

Overall Health: The Overall Health Score is the lowest score of the F1, Coverage, and Effort scores for that Theme. 

Health Breakdown: The individual scores (F1, Coverage, Effort) make up your Overall Health Score. 

 Further reading: Health Scores for Context Themes

 

5. In addition to relying on the Health Score to indicate how well your theme is likely to perform once it is applied to the rest of the data in the project, check the Results, including the Boundary, to make sure you are satisfied with how your training has been applied to the sample batch training data. If everything looks good to you, great: skip ahead to step 8.

✨ Context Theme Magic ✨

When reviewing the Results you'll see how effective your theme is getting at finding examples in your sample batch training data that match your theme, even if completely different language is used. If you see something in the Results that you're not happy with, you can manually change the coding using the same Yes/Maybe/No options.

Here's my example: Results for a context theme to find customers talking about the comfort of seats in airline reviews

After my initial prompts and a few rounds of training the results show that the theme is doing a great job of finding customers talking about the comfort of seats, even when the language used varies 'the seating is a downside', 'seats are cramped', and so on. This means I will much more accurately be able to size the number of customers who discuss the comfort of seats in their airline review, meaning customer voices are heard whether they describe the issue in language I'd predict, or not. 

6. For some themes, you may want to prioritise precision over recall, or vice versa. To do this, you can move the Hit/Miss boundary by changing the Threshold Choice. Once you have changed your threshold choice, click Retrain to apply the change. 

💡Tip: Choosing the best threshold for your theme (unsurprising spoiler: Best Overall is usually best)

Best Overall 

Best overall considers precision and recall to be equally important and, in most cases, is the most suitable threshold for a theme. 

Best Precision

Best precision prioritises keeping noise out of your theme (higher precision), but it's at the expense of missing things (lower recall). 

Best Recall 

Best recall prioritises finding as many things that match your theme as possible (higher recall) at the cost of noise (lower precision).

7. To view, edit and even add to the examples used for theme training, select Examples on the tab next to the results. What you'll see here are all of the examples that have been used to train your theme so far, including your initial prompts. To change an example between Yes/No, simply re-code it, or use the amber bin icon to remove it (the equivalent of selecting 'Maybe'). To add a new example, type your example into the text bubble at the top of the example tab and code it Yes/No. Once you have changed your examples, click Retrain to apply the change. 

8. Once you are happy with your theme and want to apply it to the rest of your data, click Enable Theme. You will have the option to apply the theme historically - this is recommended (so, tick the box!) as it will mean that your new theme will look for relevant examples in the data you already have in your project, which is super helpful for finding previous mentions of a newly identified issue. Once you've made your choice, click Confirm.

Here's a handy walkthrough video

More questions? 

We’re always happy to chat. Reach out using the help bubble at the bottom of your screen, email support@wordnerds.ai, or contact your Customer Success Manager directly.

✍️ written by: Katie, Customer Success Executive