Competency based medical education (CBME) involves outcomes driven education and assessment. CBME needs strong multifaceted assessment tools. Simulation is often used for assessment of trainee’s competencies considering its many attractive features. Assessments through simulation are standardized, performance-based and able to address higher levels of cognition and multiple dimensions of competence. Simulation is also necessary to assess high importance, low frequency skills. An assessment tool is needed to help faculty assess the performance in simulation. Noel at al found that faculty fails to detect 68% of errors when observing a trainee and the use of a tool increases error detection significantly. However, assessment tools and judgments that are made as a consequence of those assessments are important and should be compatible with assessment strength (validity). In a recent systematic review of simulation based assessment Cook et al found that from 217 eligible studies only 6 provided a unified five source validity framework and call for more robust studies with good validity evidence.
The workshop focuses on how to develop a simulation-based assessment tool by using consensus-building methods and preparing the necessary data for validity evidence of the tool based on Messick’s five-point, unified construct validity framework. To support the use and judgment made on test scores the following evidence is collected: content, response process, internal structure, relationship to other variables and consequences of the assessment.
In the last part of the workshop, participants will judge the validity of some the already published simulation-based tools in small groups.