Systems are not the only key factor when providing assessments. At Calibrand we understand this and can assist you with your whole assessment design process & strategy.
Assessments designed to measure competence are most susceptible to the maxim that – the quality of what comes out is equal to the quality of what goes in or to put it less positively – rubbish in rubbish out. In a similar way as marriage, it is not something to be taken lightly.
The content of any assessment has to be:
• Fair – those who pass must be capable of performing the task which are the subject of the assessment and conversely, lack of success must indicate a meaningful deficit in competence,
• Valid – it must measure what it is supposed to measure
• Reliable – it must measure what it measures in a consistent manner
Initial consultations – to assist in determining the outputs required from assessments and the most effective method of achieving these.
This might include some or all of the following,
• project planning,
• assessing what type of assessment will be the most appropriate,
• determining what management information requirements there are,
• consideration of delivery mechanisms and aiding in implementation,
Assessment structure development – to ensure that the assessment is developed in such a way that it will measure what it is supposed to measure.
This might include subjects such as:
• job task analysis
• what knowledge, understanding and skills need to be assessed
• how many questions
• if operating with a bank of questions, how large the bank should be,
• how long the assessments will take,
• how many sections will there be,
• will there be a pass mark and, if yes, what will it be?
Pilot testing – prior to the launch of assessments, ensure the validity and reliability, in terms of content, timing and pass/fail decisions are all correct.
Exactly how best to develop the content for a competence assessment test is, in our experience, different for each client. It is a function of, how many people need to be assessed, when will the assessments will be carried out, where they will be carried out, what needs to happen immediately post assessment, where the results need to go to, what is done with the results, how many people are to be assessed and what resources are available for the development tasks.
We're able to develop the assessments on behalf of clients and/or provide training and facilitation services for subject matter experts. Support is available in the following ways:
• Question writing events – depending on needs, level of expertise available and the outputs required, these can last between one and three days. If required, a facilitator will firstly train subject matter experts in the basics of writing good multiple choice questions and then proceed to facilitate the development of the questions. If the question writing will take place a later date then only the training workshops will take place.
• Creating questions – as an alternative to the above and where appropriate, Calibrand can take responsibility for developing questions. This would include managing the question writing process and ensuring sufficient questions are produced in line with any standards, before submission to the editing committee.
• Editing committee meetings – to protect the integrity, robustness and professionalism of an assessment, once questions have been developed, they will need to be subjected to a rigorous but efficient editing process, The facilitator will ensure that this process is based on good editing principles and avoids the type of scenario where editors may believe their job is to find things, which in their opinion, are wrong.
• Assessment grading meetings – how to determine where any pass/fail score will be set or even if there is to be one, is perhaps one of the most demanding tasks facing those who are responsible for competence assessments. This, of course, is particularly important for high stakes assessments where a person’s whole future could be determined by how well they perform.
Once in place, assessments need to be reviewed regularly, to take account of:
• changes in the external environment, perhaps budgetary, legislative, regulatory or technological
• changes in the internal environment, perhaps procedural, course content or product
• the use of the questions for other purposes
• the quality of questions as revealed by ongoing statistical analysis of their validity and reliability
• the quality of the assessments particularly their validity and reliability but also an analysis of the performance of particular learning methods or particular geographical areas or particular parts of the company.
In order to facilitate the above Calibrand provides training, facilitation or fully outsourced services in the following areas of ongoing management.
• Post assessment analysis – how to analyse the statistical evidence available following assessments to determine the validity and reliability of the full assessment and the individual questions.
• Further development of questions – to increase the number of questions in the bank, to increase the validity of those already available and to ensure that the assessment bank is up to date.
• Marking of assessments – in the event that clients do not have sufficient throughput of candidates to warrant the purchase of marking facilities themselves.
• Reporting and validation – provision of independent reports to support the validity and reliability of assessments used by clients.
• Benchmarking – where assessments are already in place, benchmarking against current best practice and provision of detailed reports.
We hope you will find many benefits in forming a business relationship with Calibrand. Some of those enjoyed by our present clients are listed below.
Cost effective – because of Calibrand’s experience gained through focussing on needs shared by all our clients, we are able to complete projects more quickly and less expensively than may otherwise be the case.
Expertise available – because of our expertise, across many different industries and countries, you can be sure that your assessments will be developed in a way which will meet your objectives, the questions will test what needs to be tested and the method of testing will be most suited to your organisation’s culture and procedures.
Ease of outsourcing – as Calibrand focuses purely on the area of assessment design and development, our clients can easily see exactly what they are outsourcing and how much it is costing.
Flexibility – Due to the varied internal needs of clients, our services range from simple advice on how to get the most from assessments, through training and benchmarking to fully outsourced management and development of question banks and assessments.
Long term relationships – Calibrand's basic business philosophy is to form long term relationships with our clients. In practice this will be achieved through a dedicated Technical Account Manager who we hope will be considered by the client company as one of their own managers.
We hope this brief introduction to Calibrand's competence assessment solutions will give you a broad idea of the added value which we can bring to your company.