Spotlight on SJTs: Five tips to minimise group differences in SJTs

4 young people stood with laptops and notebooks

We continue our Spotlight on SJTs series with a blog giving tips on ensuring an SJT is not working against an Equity, Diversity and Inclusion agenda. Please subscribe to our newsletter or check back here to follow along as more content in our series is published.

Fairness by design

The aim of any assessment is to give us information that allows us to differentiate between individuals. In a selection process, this differentiation helps us decide who to hire. But what happens when your assessment decisions seem to benefit certain groups and exclude others? 

The scrutiny given to this and other concerns about inclusion and belonging in organisations is – rightly – high. Businesses are investing heavily in Equity, Diversity and Inclusion initiatives to reduce the likelihood that individuals are disadvantaged in any way due to their personal characteristics and background.

One way to achieve this in selection is to ensure that all assessments and processes used are fair by design, rather than as an afterthought.

The implications for Situational Judgement Tests (SJTs)

SJTs are highly versatile tools that can measure behaviours, values, decision-making skills and other attributes, all while providing candidates with a unique, realistic insight into what it might be like to work at an organisation. 

It is well-documented that SJTs typically show less group differences compared to cognitive assessments, but more than personality assessments. This means they are often viewed favourably from a fairness perspective. For more information on this, see our first blog in our Spotlight series: 8 ways in which SJTs can add value, However, not all SJTs are created equal!

Cognitive and personality assessments are usually ‘off the shelf’ products, so ED&I considerations typically focus on the specific tests that are chosen and the processes to deliver them. SJTs are much more likely to be bespoke to specific settings. This enhances their attractiveness to candidates and utility in selection processes. It also provides an opportunity to ensure that fairness is baked into the SJT by design, every time.

Keep reading to find out about our top 5 design considerations to help minimise group differences in SJTs.

1. Build diversity in from the start

What to do:

When designing an SJT, and all the materials related to it, it’s important to seek input from individuals with diverse backgrounds and experiences. SJT question content is more likely to be realistic to the role and will be interpreted and answered consistently by all applicants if a range of perspectives have contributed to its creation. 

Remember:

This requires an investment in time to engage these individuals to participate in development and implementation of the SJT, and usually requires these stakeholders to take time out of their day jobs. It’s essential to explain why their input is so important. These stakeholders can also provide useful input in other areas, such as designing practice materials, FAQ documents and troubleshooting guides that will be accessible and helpful to as many prospective applicants as possible.

2. Language dos and don’ts

What to do:

The language used in your SJT scenarios and responses can play an important part in minimising group differences. Things you may wish to think about include:

  • The simplicity of the language used
  • The length of the questions
  • The use of any terms that are knowledge-dependent or require experience
  • The cultural sensitivity of the language used

Remember:

The language used in SJTs can differ based on the role being recruited for and the typical profile of the candidate group. If in doubt, make the language simpler rather than harder, and ask someone from outside the SJT design team to review the content and check it makes sense. 

3. Please provide your response below…

What to do:

SJT design goes beyond the language used in the questions. Different styles of SJT question style can be harder or easier for applicants to understand. Picking the best action from three options is simpler than picking the best action from eight options. Think about if your candidates will have seen an assessment like this before, and design accordingly.

It’s also important to consider time allocation. Make sure applicants have a sufficient amount of time to answer each question, and no time limit at all when reading instructions and answering practice questions.

Remember:

You may not get this right straight away: development of good SJTs is often an iterative process. Test out some questions, look at your data, and think about how groups are performing. Then adjust if you need to.

4. Promoting preparation

What to do:

If you want every applicant to have the best chance of success, they all must have access to good quality, free, preparation materials. This helps ‘level the playing field’ – applicants all understand what an SJT is, what it aims to measure, and how to answer the questions. Think about:     

  • How many applicants will have seen an SJT before? If this is likely to be a novel format, give them information about what to expect and what the purpose of an SJT is. 
  • How will applicants know what is expected of them when completing the SJT? Make sure the practical considerations about technology and the test platform are made clear.
  • How and when will applicants have access to this preparation material? It is useful to send it as early as possible and draw applicants’ attention to it. 

5. Data, Data, Data!

What to do:

When designing a new SJT, it always pays to try it out before using it to make selection decisions. This gives you useful data on how well it differentiates between applicants and the extent to which groups perform differently on it. Make sure you have captured data on personal characteristics to allow you to compare the groups. 

Remember:

How an SJT performs as an assessment can change over time (like all assessments!). It is useful to repeat this analysis at set times for bigger selection processes (e.g. annually for graduate schemes). Checking group differences are still within acceptable levels should be regularly monitored to ensure the SJT adds value in the selection processes.

Who are WPG?

WPG have decades of expertise in selection and assessment and places equity, diversity and inclusion goals at the heart of what we do. Our methods for SJT design, implementation and evaluation are built on best practice principles for recruitment and supported by the latest research evidence. We understand assessment design and use varies greatly across organisations depending on the context and desired outcome of using the SJT, which is why we take a bespoke approach to ensure that your SJT helps you to differentiate between high volumes of applicants fairly and reliably.

  • If you are interested in how we can support your organisation in understanding and minimising group differences in selection and assessments, please get in touch.
  • Do you want to learn more about the history of group differences, or how they may impact different selection methods? Feel free to start a discussion with us on LinkedIn!
  • To learn more about what we do, subscribe to our newsletter, where we regularly share insights into selection and examples of what is being done elsewhere.
  • For more information on using SJTs for effective recruitment, please see our other blogs in the Spotlight on SJTs Series.