Technology is undoubtedly changing the way the assessment part of recruitment is delivered. Gamification and the immersive candidate experience are perhaps the most visible innovations in today’s psychometric landscape. But technology could also transform the development of assessment tools. Specifically, crowdsourcing could be about to take assessment development by storm.
What is Crowdsourcing?
Crowdsourcing is a way for businesses to engage a ‘crowd’ in a common goal. This type of collective contribution is inspired by our growing connectivity through social media platforms such as Twitter, Facebook and – in particular – LinkedIn. And it allows organisations to harness the power of the crowd for ideas, time, expertise, or funds.
At Work Psychology Group we recently collaborated with Deloitte and the Department for Education to design the content for a high-stakes test for the accreditation of social workers. The aim of this was to target professional knowledge and appropriate judgement through a sophisticated test that requires participants to answer questions and make judgements across a range of typical situations faced in modern-day social work.
The challenges we faced were common to many assessment development projects:
- We needed a large volume of good quality test content developed quickly;
- Contribution from social work experts was essential to ensure test items were relevant and correct, but these busy professionals couldn’t commit to hours of item writing;
So, in conjunction with Deloitte, we decided to work with crowdsourcing guru Wikistrat. The team at Wikistrat offered a unique online collaboration platform, which allowed social work experts, drawn from their global network of 3000+ experts and client organisations, to collaborate and co-create in real time. Through the platform, experts could access guidance, upload their test items and critically review each other’s work. As a result, 700 test items were written and reviewed in just two weeks, rather than the several months it may have taken using traditional methods.
The Wikistrat crowdsourcing approach draws on principles borrowed from social media and gaming to create an environment of ‘collaborative competition’. Experts can connect and provide feedback through ‘comments’ and ‘likes’, providing an experience that is familiar and engaging but that also gets the job done. Plus, gathering contributions in this way ensures all voices are heard, minimising professional bias or that single focus group member dominating the discussion.
For this project, crowdsourcing was an innovative approach but one that was not without risk. The experts had no experience of writing assessment content, so clear and accessible guidance was crucial. But the approach has been a huge success for Deloitte.
Neil Hepworth, Managing Consultant at Deloitte said: “We needed lots of ideas and it was vital for us that leading thinkers from the sector contributed. In a short period of time, we were able to use the collective wisdom of a community of experts to create a vast number of high quality assessment questions. This work will have a direct impact on the improvement of social care, which is currently delivered by 35,000 child and family social workers in England to 394,000 children in need.”
Stuart Martin-Jay, CPsychol, Senior Consultant at Work Psychology Group, said: “Crowdsourcing in assessment design is still very innovative, and as with any innovation there is still much to learn. But as a process it’s engaging, democratic and holds clear potential for efficient assessment design, particularly where new material is crafted and where correct answers are judgements rather than facts.
“This said, a bespoke crowdsourcing solution comes with a price tag. Plus, we found that despite clear guidance, review by assessment expert Psychologists was still essential. In particular, it was challenging for inexperienced item writers to create plausible incorrect options. So still a need for expert training and guidance!”
If you’ve had experience you’d like to share or would like to start a conversation about the possibilities of crowdsourcing, then we’d love to hear from you.
