Exams are a fact of life – rarely pleasant at the time, but often an essential requirement if you want to get on, especially in a discipline such as medicine. Up until recently, most physicians could comfort themselves in the knowledge that, once they had successfully navigated Parts I & II of MRCP(UK), they wouldn’t need to face another exam, before taking up an appointment as a consultant (unless they chose to do so). But then things changed…

In 2008, the Federation of Royal Colleges of Physicians (UK), in conjunction with the relevant specialist societies (in our case, the Society for Endocrinology and the Association of British Clinical Diabetologists) launched the Specialty Certificate Examinations (SCEs), leading to the award of MRCP(UK) (Specialty).¹

Interestingly, the MRCP(UK) website promotes the SCE programme as an opportunity to ‘broaden your horizons’, although how many trainees view the exam in that light while preparing to sit it could be debated!

However, it is undoubtedly the case that, over the last decade, the SCE has established itself as one of the most respected international postgraduate qualifications. It is in effect a ‘quality stamp’, providing objective evidence that a specialist trainee has sufficient knowledge of their discipline to practise safely and competently as a consultant.

Perhaps the strongest endorsement of the SCE programme has come from colleagues in other European countries, who are seeking to collaborate with their UK counterparts and build on the expertise and quality of the SCEs. The European Specialty Examination in Gastroenterology and Hepatology is already up and running, and Endocrinology and Diabetes is not far behind. In 2018, the European Society of Endocrinology, in collaboration with the European Union of Medical Specialists (UEMS), will hold the first pan-European Examination in Clinical Endocrinology, Diabetes and Metabolism.

SCE FORMAT

The SCE in Endocrinology and Diabetes runs once a year, typically in early summer. It is a single exam, consisting of two papers, each containing 100 questions. Candidates are allowed 3 hours per paper with a 1-hour break in between. The examination is delivered electronically at designated test centres.

Each question (or ‘item’) is presented in a standard format (see Example question), with the majority of items based around a brief clinical scenario (the ‘stem’), followed by the question (the ‘lead-in’), and then a list of five possible answers (the ‘options’).

Importantly, these options are the ‘best-of-five’; i.e. each of the options should be plausible for the clinical context described, but one should be clearly better than the others. This is very different to the historic multiple-choice question papers that many of us sat in the past, when the candidate was often presented with a list of true/false options – several of which were often completely implausible!

Importantly, most of the 200 items seek to test the application of knowledge rather than simple factual recall, thereby attaining the second level of George Miller’s classic framework for clinical assessment (Figure), now commonly referred to as ‘Miller’s triangle’.² In essence, the candidate who successfully navigates the SCE will have demonstrated that they know how to apply the core knowledge relating to their discipline.

Inevitably, critics of such an approach may say that, by settling for an applied knowledge test, the examiners cannot be sure that competence in the SCE inevitably translates into successful/safe clinical practice. However, such criticism fails to acknowledge the broader context in which the SCE exists. No trainee is granted a Certificate of Completion of Training (CCT) simply by virtue of passing the SCE. Instead, the SCE is now a compulsory component of the CCT, but satisfactory progression through the ARCP (Annual Review of Competence Progression) process remains the vehicle through which trainees are required to demonstrate their attainment of the higher levels of Miller’s triangle (Figure). The SCE seeks to ensure that the underpinning knowledge and its application are secure.

CREATION AND DELIVERY OF THE SCE

As a senior examiner in both the undergraduate and postgraduate arenas, I not infrequently hear candidates complaining that a particular question in an exam wasn’t fair, or that the correct answer wasn’t included in the five options. My response is usually to smile, politely reassure the concerned students/trainees that the latter is very unlikely, and raise the possibility that perhaps they haven’t quite remembered the question properly!

I can do this based on an understanding of just how much time and effort have gone into the creation of each question. Indeed, understanding this process is vital to reassuring candidates that the exam they are about to sit is a high quality assessment, devoid of errors or ambiguity.

The Question Writing Group

Trainees are often surprised to learn just how much time and rigour are involved in perfecting individual questions and in creating an exam containing 200 high quality items. The process begins with individual clinicians being asked to draft questions, which are then subjected to scrutiny by their peers during a 2-day face-to-face meeting.

Such meetings are not for the faint-hearted! It can be quite uncomfortable, watching a question that you are proud of being slowly dismantled. However, if the question survives it almost always emerges a better/more robust item.

In the decade since 2008, the SCE question bank has expanded significantly and many of the items now carry performance data from previous diets, which helps inform the creation of future exams.

 

The Exam Board

Each SCE exam is mapped against the SCE Endocrinology and Diabetes blueprint³ which covers the breadth of the Specialty Training Curriculum as published by the Joint Royal Colleges of Physicians Training Board (JRCPTB).⁴

Potential questions for inclusion are reviewed by the Exam Board (comprising at least 10 practising clinicians) at a 2-day meeting. During this review, questions are either accepted in their current format, revised through minor modifications or rejected/returned to the Question Writing Group for more significant revision.

The Standard Setting Group

Once the papers have been finalised, the baton is passed to the Standard Setting Group (SSG) whose job it is to determine the pass mark for the exam. The SSG includes examiners who have been involved at each step of the process, but also includes ‘new faces’ who can bring a fresh perspective/ critical eye. Ultimately, the final pass mark for the exam is reflective of the judgements of the SSG examiners, linked to the actual performance of the candidates in the exam.

Post-exam analyses

A rigorous psychometric process is followed after each exam, to ensure that the performance of each item and of the exam as a whole is at the expected level, and to allow decisions to be made about poorly performing items.

Inevitably, this takes time, which can be frustrating for candidates who expect a computer-based test to be turned around in a matter of days, if not hours. However, it is vital that these post-examination analyses are completed properly to ensure that the reputation of the SCE as a whole is maintained.

 

CONCLUSIONS

In 2009, when the first SCE in Endocrinology and Diabetes ran, none of us could be quite sure whether the project would be a success and deliver on its stated aims and objectives. As we approach the 10th anniversary, it appears that the SCE is in good health.

Most trainees now accept that achieving the SCE is an important part of establishing their credentials as a bona fide specialist in the field. Indeed, some are keen to point out that they have a badge which many of their seniors don’t!

The fact that significant numbers of European trainees are now keen to sit a ‘Europe-wide SCE’ based on the UK model/exam speaks loudest for the success of the project – it seems UK Endocrinology and Diabetes has created a badge worth having!

Mark Gurnell, Exam Board and Standard Setting Group, Endocrinology and Diabetes SCE

PS Dr Gurnell is happy to provide the correct answer to the example question for anyone who isn’t sure…

REFERENCES

1. MRCP(UK) 2018 Specialty Certificate Examinations www.mrcpuk.org/mrcpuk-examinations/specialty-certificate-examinations.
2. Miller GE 1990 Academic Medicine 65 (9 Supplement) S63–S67.
3. MRCP(UK) 2014 SCE in Endocrinology & Diabetes Blueprint www.mrcpuk.org/sites/default/files/documents/sce-endocrinology-and-diabetes-blueprint_0.pdf.
4. Joint Royal Colleges of Physicians Training Board 2017 Specialty Training Curriculum for Endocrinology & Diabetes Mellitus www.jrcptb.org.uk/sites/default/files/2010%20Endo%20Diabetes%20curriculum%20%28amendments%202017%29_1.pdf