Finding the Ideal Support for Assessment

For over a decade the Faculty of Medicine’s Educational Assessment Unit (EAU) used the same tool to manage questions, deliver exams and evaluate students: the International Database for Enhanced Assessments and Learning or “IDEAL”. While the tool met the needs of the Faculty of Medicine in the early 2000’s, it wouldn’t stand up to the years of technological advancement and unprecedented medical program growth ahead of it.

Over time, the list of issues grew. Connectivity became a problem as more computer-based exams began to be delivered at regional sites. Users increasingly ran into unexpected, obscure error messages seemingly at random. Technicians needed to be present during computer-based exam delivery just in case the program crashed and the learner’s work needed to be recovered. Computers in multi-purpose labs across the province used to deliver exams had to be restricted to Internet Explorer 9 because the questions simply wouldn’t display on any other browser. EAU staff started using Word documents to manage paper-based exam creation in an effort to avoid using the tool.

Everyone agreed, IDEAL just wasn’t ideal.

Charting the course

The EAU and MedIT set sights on a goal: to replace IDEAL with a system that was administratively and technically easier to manage.

After a comprehensive environmental scan two tools were shortlisted. EAU staff tested them against IDEAL for functionality, usability and technical fit. ExamSoft came out as the top performer and the team got to work planning the transition.

From software procurement, to data migration, to tool configuration, to implementation, the EAU and MedIT worked as a tight team through the whole process. Everyone involved was satisfied with how clear and collaborative the project was, with MedIT offering planning and technical expertise, and the EAU offering their subject matter expertise and long-term vision.

Navigating the waters

However, the project wasn’t without its challenges. In the tool evaluation process they ran into a policy snag: ExamSoft is a “Software-as-a-Service” tool, meaning the tool and all the information it contains is hosted on the company’s servers in the United States. Some of this would be a student’s personal information. The Freedom of Information and Privacy Act (FIPPA) is very strict about how public bodies manage personal information outside of Canada. After some research and discussion, the team found that they could address this by asking students to provide their consent via a form administered through MEDICOL. But what if a student declines? Knowing they couldn’t develop a separate evaluation process for a handful of students, the team created a method to anonymize data input into the system. That way they could link the exam data to the student while still adhering to privacy laws.

Another challenge was the sheer amount of information to migrate from one system to the other. Each MD Undergraduate Program year had thousands of questions to be migrated. Unfortunately, this wouldn’t be as simple as downloading from the old system and uploading to the new. Members of the EAU team had to export the questions from IDEAL, categorize them, and then input them to a computer script written by MedIT’s Brandon Nam that enabled ExamSoft to recognize the questions. The EAU team is still going through the process of validating each question to ensure the system contains accurate question information before it is used in an exam.

The journey continues

The EAU has started to create paper-based exams using ExamSoft. Dion Zhai, Senior Assessment Coordinator, was pleased to find how easy it was to pull questions for exam creation. He and the rest of the EAU team are excited for ExamSoft’s future potential in medical student assessments.

In the near future the team will start to pilot computer-based exams using the system. The four-year vision is not only to have all exams computer-based, but to have the students take exams using their own devices.

The tool also has a lot to offer the team in terms of analytics. They can input paper-based exam results back into the system and track student results as they progress through the program. They can also use the data to evaluate the exam questions themselves. Previously a manual task, ExamSoft will calculate a question’s difficulty index and see how that question performs over multiple exams.

Despite the excitement about ExamSoft’s future potential, the EAU team admits that adopting the new system still required a culture shift for their team. They had to think about the manual workarounds they had developed over the years and be intentional about creating and following new procedures going forward. They also had to find new ways to work together effectively using the new system.

For Niyati Kasana, Senior Assessment Coordinator, these challenges present themselves as exciting learning opportunities. She is looking forward to the improved access to knowledge sharing and professional development that is enabled by being part of the ExamSoft education community.