When Aga Khan’s Medical College in East Africa was looking to streamline and improve their exam administration processes, they turned to Speedwell to help.

Aga Khan University was founded in Pakistan in 1983 with a mission to improve the quality of life in the developing world and beyond through world-class teaching, research and health-care delivery. Today, AKU is a global university, with 6 campuses over 3 continents.

The AKU Medical College in East Africa has campuses in both Kenya and Tanzania and is part of the faculty of health sciences. The College closely collaborates with the Aga Khan University Hospital, the AKU School of Nursing and Midwifery and the Aga Khan Health Services network in Kenya, Uganda and Tanzania.

Mr Kennedy Njenga, the Manager for Academic Programmes at the Medical College is responsible for overseeing the College’s activities in the region. The main programme at the College is the Masters in Medicine, this is a 4-year specialist course, stretching across 13 different medical specialisms all of which are at masters level in medicine.

Exam administration challenges

Historically, for the team at the College, creating and delivering exams for the programme was a time consuming and cumbersome process. Due to the many specialist areas covered, lots of people are involved, plus there’s the geographic split to contend with. Individuals are responsible for the exams and exam questions in their specialism area but are also all faculty doctors who have busy clinics. Previously setting exams would be a huge challenge with it being almost impossible to get everyone in one place at the same time to complete the process.

With no formal, unified approach, it was common for exams and questions to be stored on individual laptops and other portable storage devices. In instances of staff movement, it would often be difficult to establish where questions were stored. Password protection could lead to further problems when passwords were forgotten or lost. Then, there were security considerations – the concern that exams or questions could potentially be compromised through lack of password protection or by being stored insecurely in a PC in the office or on a USB drive. For the team, efficiency in the exam management process and minimising security risks was a priority.

On top of this, there was a large body of old items. With no centralised repository, it was difficult for the team to establish which item had been used, when it was used and how many times. This information was simply not available. For the exam moderation process, files would be issued, with secure passwords sent separately. A lot of back and forth would follow and sometimes issues with operating system compatibility would arise creating further headaches for the team.

Additionally, the team were conscious of the amount of paper and ink used each year to produce the papers needed to administer the exams. Not to mention the extensive manpower resource required to produce, bind and pack them, before finding suitable space to store them securely. Then there was the added laborious, time-consuming task of manually marking thousands of scripts following an examination exercise.

The whole process was unwieldy and a lot of work, the consensus being that there was huge potential for improvement and so Kennedy and his Academic Office team began their search for an alternative solution.

The Speedwell difference

The Medical College had previously had an exam management system which had not met their needs. So, this time around, the stakes were higher. It was imperative the solution was more robust and better developed. Kennedy explains the main criteria the new system needed to meet; “Firstly, we were looking for a repository, somewhere to bank and have easy access to questions. Secondly, it needed to be a collaborative tool for exam development. Something that could transcend space, so that everyone involved could work on the exams without having to be in the same room at the same time and the system would automatically track the progress. Then finally, security was obviously a big requirement. We needed to feel confident that there were security measures in place both in exam development and exam administration.”

After searching online and seeking referrals, five companies of interest were shortlisted, including Speedwell. Each company gave a demonstration and the College assessed each on merit against a point scoring system. Kennedy explains that Speedwell scored very highly and really stood out. Not only did the eSystem meet the required criteria, but our credentials in the medical education sector set us apart: “It was clear from your client list that you were very mature in the medical education sector. Ultimately this gave us the confidence that Speedwell was the right choice. We knew that you would have the knowledge and expertise to deliver a system that would work for us.”

Another important factor was that the successful solution was ready to go. They did not want a system that required tweaks to match their needs or a lengthy roll-out plan. They needed a system that was fully developed and would be ready for quick installation. Kennedy said:

“Speedwell was the only company that could deliver a complete solution to our requirements, and be ready for quick deployment. We were up and running with eSystem within 7 days of placing the order”

Once the team had completed the comprehensive training session they were ready to start using the eSystem to improve their exam processes. Additionally, the ready support provided by Speedwell’s technical team gave them a lot of confidence.

Exam administration transformed

The Medical College has been using eSystem’s question banking facility for several years and they couldn’t be happier with the improvements they’ve seen;

“We have seen a significant time-saving in our processes. Using eSystem’s question bank we can set new exams in less than a month. Previously this would have taken us around 3 months. This is clearly a tangible win” says Kennedy.

He goes on to explain that having one system where all tasks can be completed has been hugely beneficial. Examiners no longer store items locally, they add them directly into the bank, and everything is now in one place and easily accessible. A marked improvement from the scattered word files of the past.

Exam moderation has also been streamlined, with drafts being created within the eSystem and user permissions being utilised to ensure that only authorised users have access. There is no longer the need to bring everyone around a table. Examiners can access the drafts any time and importantly at a time that fits around their clinical responsibilities. The audit trail feature ensures there’s transparency over any changes made.

With so many specialisms making up the programme, ensuring that all areas were covered in the exam was always a big task. Previously the team would have to manually sort through the relevant questions and pick which ones to use.  It was hugely time-consuming. Using the eSystem’s question banking facility has brought the benefits of exam blueprinting to the team as Kennedy describes:

“The speed with which we can create an exam now is quite something. With eSystem’s blueprinting facility, it’s so easy, with the touch of a button – voila, the exam is created!”

And as they have been gradually adding in exam delivery for selected exams, Kennedy, cannot state enough how much time eSystem has saved them; “Setting exams, moderation discussions, standard-setting – all this can be done within the eSystem. Post-exam we just click on the report and get the results, no more in-person marking or reviewing. There is a high degree of efficiency and accuracy.”

Examiners are grateful that they don’t have to find time away from their clinical responsibilities, to mark and moderate papers. Now they can just focus on teaching and treating patients. As with any change, however, there was a degree of internal resistance. The College found that it was important to take the time to comprehensively trial the system, build confidence and bring everyone on board.

Commenting further on initial reticence from users, Kennedy confirms that he felt it was critical to provide all users with detailed training. After initially being trained by Speedwell, they developed an internal training programme; “Once people start to use it, they see how intuitive and easy it is. Then the benefits also become clear, there is no further resistance” Having achieved such huge successes with the eSystem, thoughts turned to how they could use it to gain further efficiencies.

Delivering exams during a pandemic

In early 2020, COVID-19 hit and its impact started to be felt across the globe. Like many other institutions, the Medical College’s plans were disrupted. They were about to run mandatory paper mock exams for residents. However, some had been impacted by Coronavirus and were required to quarantine at home.

Fortunately, prior to the pandemic, discussions had taken place about the possibility of using eSystem’s online exam delivery externally, for the admissions exams. Although they had been using exam delivery successfully internally, Kennedy recalls “The big question for us was whether people outside   of the institution could access the system.” After a reassuring call with Speedwell’s support team, he was pleased to discover that this was possible and straightforward to set up.

Due to this advance planning for the admissions exams, they were in a strong position when COVID-19 hit and so, in a bid to minimise disruption to the mock-exams, the decision was made to switch to an online format. Not only would this allow the exams to run and all students to partake, but the Medical College also saw it as the perfect opportunity to trial external online exams ahead of the higher-stakes admissions exams.

Kennedy explains; “We granted access to a VPN so that the candidates were able to connect to the secure exam server at the university and take the exams” he continues, “Everything went smoothly and that really gave the College the impetus and confidence to push forward and use the eSystem for the admissions exams.”

Having the peace of mind that their exam programme could continue despite the disruption of COVID-19 has allowed the College to focus resources on moving the teaching and learning online during the pandemic.

Improving exam delivery

As planned in early September, the College began recruitment for their next cohort. The format, a two-stage exam, an MCQ and an article critique – a short answer question exam. Candidates must meet the standard pass mark set for the MCQ exam to be invited to take the second exam. Previously, the College would invite everyone on campus and conduct both exams on paper.

For the current cohort, they had applicants from 15 different countries totalling 333 candidates. Kennedy explains that what is normally a time consuming and logistical challenge, became much simpler;

“Amazingly, with Speedwell’s eSystem, we were able to seamlessly run exams across 15 different countries, for over 300 candidates – all at the same time. A great feat! And saved everyone a great deal of trouble!”

Prior to running the exams, there was understandable apprehension over what would happen in the event of internet failure, but the team were reassured by the safeguards in place with the eSystem. Kennedy explains what happened when inevitably these instances did occur;

“Candidates have experienced screen freezes, internet failure and even lap-top malfunctions during exams, but thanks to eSystem’s back-ups we have  not experienced any major problems. All candidates were able to re-join and complete the exam with  little disruption”

As a result, technical issues during an exam have not caused any problems for the College or candidates.

At the end of the first exam, marking needed to be completed. With the paper-based exam, the marking process was testing and stressful, to say the least. Examiners would need to mark the MCQ exam to determine who qualified for the second stage. In reality, this meant marking a 100 question MCQ paper for over 330 students and then releasing the results in under an hour for shortlisting. Kennedy was pleased to report that this year was very different;

“With eSystem, the stress was taken away. With online marking, the results were available immediately, at the touch of a button – this is a huge highlight for us”

Not only has the Medical College been able to improve their exam processes, but they’ve also completed their student recruitment in unprecedented times:

“Speedwell has really enabled us to run these exams – we absolutely couldn’t have done it any other way, especially in COVID times”

Furthermore, it means that the Medical College is on course to deliver exams in November as normal. They have not deviated from their planned calendar, which given the global disruption, is a huge achievement:  “I keep wondering what we would have done had we not been using Speedwell’s eSystem before the pandemic. I don’t think we would have been able to run all of our exams – it would have been too much to learn and too much work to do in such a short space of time” Kennedy explains.

Enhanced Candidate Experience

Although the benefits of using Speedwell’s eSystem for the exam administration team is clear, it has been a positive experience for candidates too. Being able to take their exams in their home location is an enormous benefit. With the paper exam, it was not unheard of for candidates to travel for days across several different countries, to be present at the College to take the exam.

For many, it was a huge personal expense, both financial and in time away from their clinical duties. Not to mention that after days of travelling they may not be on their best form for taking an exam!

This has changed with the move to online exams and has been very well received by candidates. Perhaps, going forward, it makes the programme even more accessible to applicants from far and wide.

The Future

Summing up the experience of using the eSystem Kennedy says:

“I cannot underscore enough that the eSystem has saved us so much.  In terms of the environmental impact – saving on paper, in terms of time – it’s injected a high degree of efficiency into our exam processes. And of course, it’s saved money, both for the College and our prospective students – everybody is happy”

With such huge benefits, the Medical College is keen to roll out the eSystem to another one of their programmes. In addition, the School of Nursing and Midwifery are currently conducting tests with a view to adopting it too.

Ultimately, there’s no going back, using the question bank is now mandatory and with growing confidence in exam delivery, the Medical College is looking forward to greater efficiencies as the eSystem is further utilised.