Institute of Brewing & Distilling
How the Institute of Brewing & Distilling implemented On Screen Marking (OSM) when the heat was turned up in the hottest of summers in 2018.
by Steve Curtis, Technical and Publications Manager, www.ibd.org.uk, firstname.lastname@example.org
This study tells the story of how the Institute of Brewing & Distilling (IBD) moved from a paper solution for its annual essay and short answer exams to an online marking solution in just six weeks during the summer of 2018.
At the IBD, we have a strategic mission:
To provide the most respected and widely recognised qualifications, training and development in brewing and distilling globally
Due to the increase in popularity of craft beers and spirits, and brewing in particular, we are growing quickly. We provide a range of examinations in brewing, distilling, packaging and malting many of which were completed on paper and marked on paper – a process which was under strain due to the increased numbers. There is one main week of exams, in early June each year.
Originally, we were thinking of implementing the project in two stages: this year scan the scripts so we can present and store them electronically; and the following year move to online marking. We mentioned it in passing to Gavin Busuttil-Reynaud, Director of AlphaPlus who suggested we might want to speak with Calibrand.
Both Calibrand and AlphaPlus whilst being very supportive of IBD’s project plans, warned that sometimes the process ‘gets stuck’ i.e. the first part of the process, in this case the scanning, gets implemented well, but in a way that prevents a smooth transformation to online marking at a later date.
We spoke to Calibrand in early May and they made some suggestions with respect to the bar-coding of the scanned papers and the formatting of the scans which would help in moving across to On Screen Marking as and when IBD decided to do that.
IBD visited Calibrand and we liked what we saw, after several conversations and discussions about the time-tabling; the decision to go ahead was made according to the following schedule:
- First contact 3rd May 2018
- First meeting IBD/Calibrand 17th May 2018
- Exams 5th-8th June 2018 globally
- System ready early July 2018
- Scanning done July 10-14 2018
- Marking from 24th July
- Marking completed 29th August
- Moderation early September
- Results published early September
Although the meetings went into a lot more depth the essentials of On-Screen Marking are shown on this video.
From getting the set-ups right at the start of June, then on it was all about co-operation with the scanning company, the IBD markers, and the IBD team.
Our project team at IBD were very strong on the day-to-day work, initially led by Rob Evans* , and then Francesca Williams who were both excellent at the administration and processing of the project as well as using the software. With a few small issues, a mid-project weekend ‘catch-up’, and of course some moderation process tailoring, this project went as smoothly as both IBD and Calibrand had hoped for.
In any digitalisation project in the examinations sector, winning the hearts and minds of the markers, often traditional and conservative, is one of the key indicators of success; moving to a digital solution in a matter of weeks with only a few weeks’ notice could have proved a leap too far at this stage.
Angus Steven, Chairman of the IBD Board of Examiners, said:
Calibrand were recommended to us as a company who would support us with our move to examination On Screen Marking. The system we installed has proved to be a great success with our worldwide examination teams. They are now able to mark scripts without the need to transport papers around the world. This has improved data security, ease of access to papers for examiners and the facility to mark on screen against agreed marking schedules. It has also given the Examination Management team access to trend data to highlight areas for improvement in both question setting and Examiner standardisation. We will further develop the system this year with Calibrand to enhance its functionality. I also found the Calibrand team to be very flexible and responsive to our demands as first year users.
Shane McNamara, one of IBD’s markers, said:
In my opinion the system is great and works exactly as an online scanned exam system should. No papers to lug around, can log on and do bits and pieces, it keeps tab on what progress has been made on which questions.
So, all in all, a really positive implementation done in a very short period of time, and setting IBD up well for the future. So good in fact, that I understand Calibrand intends to use it as part of their presentation at the https://www.beyond-multiple-choice.com/ conference in Washington D.C. in November.
I think we’ve made great progress with more still to come, in a very short time we’ve gone from paper to a digital solution with clear benefits for the security, storage, marking and transporting of scripts as well as better record-keeping, audit data and, of course, some clear cost benefits. It puts us in a good position going forward to keep expanding and extending the reach of our exams.
** Rob Evans left IBD mid-project to take up a role as Head Distiller for a distillery in Vancouver, Canada. (Some people! #dreamjob)