We’re drawing our Spotlight on SJT blog series to a close by considering the future of SJTs in an AI-enabled world. The series has discussed how SJTs can add value in selection, how fairness can be embedded in SJT design, how SJTs might be used to tackle different recruitment challenges, and how you might go about implementing an SJT in your context. But do SJTs have a future in a world where Generative AI tools are widely and freely available?
We hope you’ve enjoyed our 2024 series on SJTs. If so, please subscribe to our newsletter to keep up to date with our latest thoughts in #assessment, #businesspsychology, and #peopleatwork.
What’s the panic?
The explosion of Generative AI tools in the last couple of years has completely changed how people interact with AI. It’s evolved from an abstract concept that most people didn’t consciously encounter to a practical set of tools that can be used in a multitude of different ways to improve learning and productivity. And with that has come a growing concern that it will (or maybe already has) sounded the death knell for ‘traditional assessments’. If candidates can simply use a Generative AI tool to generate a response to an interview question, complete a work sample test, or answer questions on a situational judgement test or cognitive ability test, how can we have faith that those assessments are still reliable and valid to use in selection and development?
I know AI can do well on cognitive tests, but aren’t SJTs different?
While our series has demonstrated how well designed SJTs can add value to the selection process, the growing evidence is that more developed generative AI technologies such as ChatGPT4 can perform as well, or better, than the majority of real candidates without support on an unsupervised (or unproctored) Situational Judgement Test. Despite the fact these assessments are designed to measure more nuanced perceptions of what is or isn’t appropriate workplace behaviour (rather than knowledge tests like cognitive ability assessments), research shows that AI tools are fairly accurate in defining the best responses to workplace dilemmas (Borchert et al, 2023; Sareen et al, 2023; Grela et al, 2024). So, if candidates are using generative AI to answer unsupervised SJT assessments, then it may be more difficult to obtain an accurate measure of their own understanding of those dilemmas.
However, this doesn’t spell the end of SJTs. We’ve compiled a set of practical steps that can be taken to protect the integrity of your assessments in an AI-enabled world, focused on the 4 areas shown in the diagram.
Design of assessments
There are a number of ways the design of SJTs (and assessments more broadly) can be adapted to mitigate the risks of AI use:
- Questions requiring candidates to use applied judgement, or problem-solving are harder for an AI to answer accurately. Assessment content should be specific, not generic, and related to the role in question.
- Multi-step questions (previous answers inform future answers) make it hard for an AI tool to answer consistently.
- Designing assessments so questions are delivered in a random order, or to different groups of candidates, can help protect the security of assessment content.
- Open-ended questions, and some types of closed-response questions, generate more variable AI responses.
Delivery of assessments
The issue with increasing use of AI by candidates is more related to the method of assessment delivery than the nature of the assessments themselves (there is a strong evidence base about the value of SJTs in effective assessment across a range of sectors). If there is agreement that situational judgement or other skills are still important to assess, then consideration needs to be given to how those assessments are delivered in a secure way:
- Provision of different kinds of ‘proctoring’ (monitoring/supervision) can safeguard against inappropriate AI use by candidates. If your test is delivered on demand, virtual proctoring tools can be used to monitor candidate behaviour and screen activity. Reports can show if additional checks should be performed for certain candidates based on flags logged.
- Locked-down browsers for assessment provision can prevent candidates copy/pasting assessment content into an AI tool, or stop them from using any other sites while taking the assessment.
- In-person proctoring massively reduces the risk of cheating for high-stakes assessments.
- Applying time limits to tests can reduce the possibility candidates use an AI tool to generate answers, as long as reasonable adjustments are available as required.
- Some delivery precautions may not be as necessary if assessments are used for employee development purposes, rather than for selection.
Whole system review
It’s important to consider the challenge AI poses to assessment in a holistic way, thinking about whole processes rather than single assessments in isolation. Other strategies can be used to complement good assessment design and secure delivery:
- Use of multiple assessments can be helpful in validating performance across different methods. For example, an in-person or live interview could be designed to validate results from previous automated assessment stages.
- AI detection software can be used to review written responses to detect patterns that might indicate use of an AI (although be cautious with this as the tools aren’t very consistent yet!).
- Analysing response timing can indicate if candidates have answered questions rapidly and still scored well, or spent an unusually long time on the test (implying potential inappropriate use of AI despite a locked-down browser).
- Reviewing average performance on each assessment after delivery can show if scores have meaningfully changed year on year.
Transparency and education
Continuing the theme of whole system review, we need to ensure everyone involved in recruitment (including candidates) is clear on how AI tools should be responsibly used:
- It’s critical that all stakeholders are aware of how AI is applied in assessment. Providing candidates with an overview of the human touchpoints embedded in assessment processes can build candidate trust that systems have appropriate oversight.
- Candidates should be given meaningful guidance on when to use or not use AI tools. This may vary based on the type(s) of assessment they are taking.
- Use of honesty statements, or being explicit with candidates about the implications of AI misuse, can also be a useful strategy to reduce inappropriate AI use.
- Gather candidate feedback, including how clear they were on what constitutes appropriate AI use, to refine processes in future.
Where do we go from here?
There are many steps that assessment users and providers can take to ensure SJTs and other assessments remain effective tools in an AI-enabled world. But we also need to reflect on how we balance the implementation of AI guardrails with other important challenges in selection and assessment; we want assessment to be a good predictor of performance in role, not advantageous to certain groups over others, and an efficient way of making consistent decisions about potential.
AI evolution does not mean traditional assessments are dead, or that there are alternative tools immediately available which provide a similarly robust measure of critical skills. However, AI does provide us with a positive opportunity to reflect on our assessment use, make continuous improvements to processes, and be proactive in thinking about to measure skills of interest in an AI-enabled world.
What do we offer?
WPG have decades of expertise in selection and assessment. We understand assessment design and use varies across organisations; we take an individualised approach to ensure that your assessment process works for you and your stakeholders.
- If you are interested in discussing your own selection processes or use of assessments, please get in touch.
- Do you want to learn more about what we do, and what our colleagues in the Occupational and Business Psychology field think? Feel free to start a discussion with us on LinkedIn!
- To get regular insights into our current projects and things going on at WPG, subscribe to our newsletter/blog.
- For more information on using SJTs for effective recruitment, please see our other blogs in the Spotlight on SJTs Series.
