In a healthcare setting, over the past year assessors have had to rely solely on remote digital selection methods.
But how fair and valid are these new assessment processes? And what lessons can we learn from this unprecedented time in healthcare selection?
This was one topic of discussion at this year’s International Medical Education Conference (IMEC), which was held virtually earlier this month.
International Medical Education Conference
The IMEC provides a platform to exchange ideas, experiences and showcase innovations within the medical field.
WPG’s Consultant Psychologist, Emma-Louise Rowe, spoke at the conference about the strengths and limitations of digital selection procedures based on available research. Plus also what we can learn from this natural ongoing experiment in the world of healthcare selection.
Emma-Louise said: “It is encouraging to see so many institutions keen to take a step back to evaluate their assessments and consider the evidence available for these remote options, before deciding how they will assess candidates going forward.”
The strengths and limitations of digital selection procedures
During the talk, Emma-Louise explored a handful of strengths and limitations of current digital assessment procedures. These included:
- Their accuracy – The switch to remote digital tools has seen a reduction in test length and a reduction in contact time with the assessors. Emma-Louise said: “This threatens the reliability and validity of the tool. So, this would need to be thoroughly evaluated if a digital solution were to be adopted long term.”
- A candidate’s digital familiarity – At what level does each candidate feel confident using the technology? Emma-Louise said: “When you go digital-, the candidate is then carrying an additional load as they have to manage the tech whilst being interviewed. All of which may affect their performance.”
- How people present – People may present differently over video, rather than in person. Emma-Louise said: “Communication and empathy are areas that healthcare assessors look for. But body language and non-verbal cues can be difficult to assess online.”
- Changes to applicant pool – “A big positive with digital assessments is that more applicants may have increased access to them. There’s not the need for travel and the costs – both financial and time – that involves,” Emma-Louise said. “But there will also be those who may not have the hardware or appropriate environments to partake in digital selections procedures fairly. So it’s a real balancing act for assessment teams when it comes to setting future strategy.”
- Cost and efficiently – There are many cost savings with reduced travel and no venue hire. But there are also new costs to think about. This includes additional resources to manage digital platforms, brief candidates and provide tech support.
- Stakeholder acceptance and feedback – At present, there is very limited evidence here. Emma-Louise said: “Before the pandemic, candidate feedback suggested they were very accepting of digital selection procedures as a short–listing tool for medical school assessment. There is also limited evidence on assessor feedback and whether assessors can adequately assess candidates only using digital selection tools.”
The next steps
To ensure we continue to provide a fair and balanced selection process, we must continue to:
- Build evidence – better understanding of remote assessments and interviews for high stakes selection.
- Compare face to face vs remote processes – a better understanding of the differences in assessments will enable evidenced decisions about future selection processes.
- Longitudinal impact – consider the long–term impact and unintentional consequences of changes to medical selection during Covid–19 restrictions.
- Stakeholder perceptions – we must work to understand how applicants view these new online tools when face to face options are available again.
Emma-Louise added: “Right now, the validity of such high stakes on-line selection is unknown. Neither can we understand candidates’ perceptions of these processes when face–to–face restrictions are lifted. It is essential to gain an understanding of likely sources of bias in digital testing formats. However, we must also adapt to changing times and embrace the tech around us.
“The priority is now to invest in high quality evidence to ensure that digital selection procedures meet the key evaluation criteria and then we will be able to build on our existing research-based understanding of high stakes selection. And we need to conduct research outside of Covid-19 restrictions to understand the use of digital selection procedures when face to face delivery is available.”
