Phase II - Peer Review
Currently, approximately 240 pharmacists per year are selected for Phase II of the Practice Review (the Peer Review), which is a clinical knowledge and practice-based assessment lasting approximately six hours. Peer Reviews are held in Toronto at the College four times a year.
The Peer Review consists of:
- Orientation session (30 minutes)
- Learning Portfolio sharing session (60 minutes)
- Clinical Knowledge Assessment (CKA) consisting of 15 cases each followed by four multiple choice questions (115 minutes)
- Standardized Patient Interviews (SPI) during which interactions with trained standardized patients in 5 case scenarios will be assessed by peer assessors (75 minutes)
- A general feedback session at the end of the review process
The peer assessors who volunteer as evaluators during the Peer Review are pharmacists that represent various practice settings. These individuals have been specially trained by a consultant to assess members in a standardized fashion.
Peer Review Components
(1) Clinical Knowledge Assessment (CKA)
The “Clinical Knowledge Assessment” (CKA) component of the Peer Review is a case based, open-book multiple choice question examination. It represents relevant, common situations encountered in everyday pharmacy practice. The CKA cases have been written and reviewed by practicing pharmacists to ensure that they are relevant and reflect the type of cases that are seen regularly.
The following references will be provided to the candidates during CKA of Peer Review:
- Compendium of Pharmaceuticals and Specialties (CPS)
- Therapeutic Choices
- Patient Self-Care
It may be tempting to look up the answer to every question. However, time is not allotted for this, and candidates must rely in part, on their professional knowledge in order to demonstrate a sufficiently broad base to effectively provide direct patient care. The Clinical Knowledge Assessment examines the candidates’ ability to apply knowledge to clinical situations, using references when needed to supplement their own knowledge base.
An interactive Clinical Knowledge Exam Tool is available. The rationale behind the correct answer of each question on this Clinical Knowledge Exam Tool is readily available.
Click here for the Clinical Knowledge Exam Tool
Standardized Patient Interviews (SPI)
The “Standardized Patient Interviews” (SPI) case scenarios have also been written and reviewed by practicing pharmacists to ensure that they are relevant to general pharmacy practice.
The following references are provided to the candidates during SPI portion of Peer Review:
- Compendium of Pharmaceuticals and Specialties (CPS)
- Therapeutic Choices
- Patient Self-Care
- USPDI Vol. 2 – Advice for the Patient
In each of the 5 SPI cases, a standardized patient will present a prescription to the candidate. The candidate will be expected to provide usual and customary pharmacy care services. A trained peer assessor (one for each case scenario), who is a practicing pharmacist, will observe and assess the candidate’s interaction with each of the 5 standardized patients (in the 5 cases respectively) on the following components.
(2) Gathering Information
The “Gathering Information” component of the standardized patient interviews should flow smoothly in an organized, logical and systematic fashion, with the candidate leading the process confidently while encouraging the patient to ask questions and give input. When patients raise an issue of concern which signals a “red flag” in the candidate’s professional opinion, it is important to pursue the issue further, being as thorough as possible and making sure that all necessary information about the potential drug-related problem is gathered.
(3) Patient Management and Follow Up
“Patient Management and Follow Up” strategies consist of addressing the most relevant drug-related problems and communicating clearly to the patient in lay terms. This includes discussing contributing factors such as lifestyle issues and devising a plan together to resolve or prevent similar situations from occurring. Patient education is the key when providing practical recommendations to help them make better health care choices. The conclusion of this component of the interview should include reviewing important educational points and providing the patient with an opportunity to ask questions to clarify any outstanding issues. Follow up may be quite informal, for instance, a simple invitation for the patient to call with any concerns, or sometimes it may be more important for the pharmacist (candidate) to contact the patient within an appropriate time frame.
(4) Communication Skills
Candidates are expected to demonstrate their ability to communicate through assessment in five domains:
- verbal expression
- non-verbal expression
- empathy
- organization of the interview/coherence
- clinical knowledge
When seeking resources to develop patient interview skills, it is best to combine print resources with live, interactive learning opportunities, as these skills require practice to develop.
Exemption from Selection to Phase II
In June of 2001, Council agreed to the following exemptions related to selection into Phase II of the practice review.
- To exclude members who have recently passed the Pharmacy Examining Board of Canada Examination including the OSCE component from the pool for Phase 2 selection for a period of 5 years from the date of the examination.
- To exclude new graduates in the pool for Phase 2 selections for a period of 5 years from the date of graduation.
- To exclude members who have successfully completed Phase II of the Practice Review from the pool for Phase II selection for a period of 5 years. Should they be chosen for Phase II of the Practice Review in the next 5 years, they will be assessed with the clinical knowledge assessment in a convenient location.
Deferral Requests
Every attempt is made to be as flexible as possible when scheduling the Peer Reviews, and deferrals have been granted in certain instances. Such instances have included illness of a member or one of their family members or conflicting work or travel plans. Please be in communication with the College immediately if you have a conflict.
Duties of the QA Committee With Regard to Participation in the Peer Review
The Quality Assurance Committee has the discretion to transfer a member from Part A to Part B of the Register by the authority granted under Section 47(3) of the College's Quality Assurance regulation, which reads:
If a pharmacist listed in Part A fails to undergo a required practice review, the Committee may transfer the pharmacist to Part B after giving him or her a reasonable opportunity to make written submissions.
This section would apply to any member who fails to participate in either Phase I or Phase II of the QA practice review process after being randomly selected to do so. The QA Committee would only act under the authority of this section in instances where repeated communications from the College to a member have gone unanswered. Section 47(3) gives the Quality Assurance Committee the power to transfer a member from Part A to Part B in such instances after giving him or her reasonable opportunity to make written submissions to the QA Committee. Members must be given 14 days in which to make written representation or to request an appearance before the QA Committee in order to make an oral submission to the QA Committee citing reasons why they should not be transferred into Part B.
Reimbursement
Members residing in Ontario who are randomly selected for Phase II, the Peer Review will be reimbursed for expenses that they incur in traveling to the College upon submission of all receipts according to the College's Peer Review Reimbursement Policies and Guidelines for Travel and Accommodation, which is included in the appointment package.
Members practicing or residing in other Canadian jurisdictions who are randomly selected for the Peer Review will be reimbursed up to a maximum amount established by the QA Committee (currently CAD $1,200).
The College will not reimburse expenses of members practicing or residing outside Canada.
Reporting of Results
Within eight weeks the consultant presents the individual results from a Peer Review administration to the Quality Assurance Committee. The QA Committee considers individual results by candidate number only, and as such, is unaware of the identities of individual candidates. In this way, all candidates are considered in the same fashion and treated equally.
Once the direction of the QA Committee has been received, College staff will send results letters to all candidates who have met or exceeded the standards in all four areas of assessment, and to those candidates who may have fallen below one or more of the standards but where follow-up is not indicated. These letters are sent approximately six to eight weeks after each Phase II administration. Where it is the view of the Quality Assurance Committee that remediation and/or reassessment is required of a candidate, the candidate will be contacted by written communication stating the view of the QA Committee and offering the candidate the opportunity to meet with a Peer Support Group (see Peer Support Group below).
Candidate Performance Report
Attached to the results letter, each candidate receives a performance report. (A sample performance report is available upon request.) This report includes a detailed explanation about how to interpret and understand the information enclosed. It includes:
- A performance summary table that outlines their score in each result area along with the Minimum Performance Level (MPL) and their status (Met Standard or Fell Below Standard).
- A table of results from the Clinical Knowledge Assessment broken down by case and a guide outlining the clinical focus of each case along with a comparison against other candidates from the same administration.
- A table outlining their performance in the Gathering Information Domain in the Standardized Patient Interviews with a comparison against other candidates from the same administration.
- A table outlining their performance in the Options, Management and Follow up Domain in the Standardized Patient Interviews with a comparison against other candidates from the same administration.
- A table outlining their performance in the five Communication domains scored with a comparison against other candidates from the same administration.
Decisions Regarding Peer Review Results & Remediation
Each candidate is considered on an individual basis with respect to his or her Peer Review results. Members have the right to make written or oral submission to the Quality Assurance Committee following the receipt of their report and Committee decision. Historically, the Quality Assurance Committee has given College staff the following direction upon its decisions.
- Candidates meeting or exceeding standards in all four components
Successful candidates receive a letter which congratulates them on successful completion of the Peer Review process and invites them to contact a designated staff person should they have any questions or concerns. These individuals are also invited to participate in the Peer Review development process or be trained as an assessor.
- Candidates falling below the standard in one component
Candidates failing to meet the standard in one component of the Peer Review (either Gathering Information or Patient Management and Follow up) will be informed that the Quality Assurance Committee encourages them to review the list of educational resources included with their results letters and participate in any programs that may assist them in enhancing their professional knowledge and skills, especially in the noted areas. In some cases, the Quality Assurance Committee may recommend or require a member to prepare a written education action plan and ask for subsequent evidence or confirmation that the action plan has been successfully completed. These candidates have not been required to undergo reassessment.
Candidates who fall below the standard in the assessment area of Clinical Knowledge or Communication Skills are generally asked to submit a written education action plan, undertake their planned remediation, and be reassessed in the clinical knowledge component or the standardized patient interview component of the Peer Review.
- Candidates falling below the standard in two or more components
These candidates will be required by the Quality Assurance Committee to submit an education action plan to the College within four to six weeks upon the receipt of their results. The action plan should outline the remedial education activities that they intend to pursue for the purpose of enhancing their professional knowledge and skills in the areas that have been identified in their Peer Review. These candidates are also offered the opportunity to meet either in person or by teleconference with a Peer Support Group (see Peer Support Group below) to discuss their plans for remediation and subsequent reassessment. Generally, candidates schedule their reassessments within a year, but this time frame is flexible depending on the candidate's comfort level and degree of readiness.
One of the supporting resources for remedial candidates is access to a Peer Support Group. This group comprises of two education minded pharmacists and a staff resource. Generally each remedial candidate is provided a 45-minute session with the Peer Support Group.
At these meetings, support is provided to help the remedial candidate develop an education action plan, set a target date for reassessment, and discuss any issues which may be preventing the individual from participating in remedial activities and ultimately achieving success in the Peer Review assessment.
The Peer Support Group is generally offered three times per year at the College.
Role of the Peer Support Group
To ensure the candidate understands the importance of continual professional development as a responsibility of being a health care provider
To ensure the candidate understands the seriousness of the situation from a pharmacist/colleague perspective
To provide support and encouragement regarding the candidate’s ability to successfully remediate
To assist the candidate in articulating his/her learning needs and guide them towards appropriate learning opportunities
It has been determined that it is a conflict of interest for a Committee Member to be an Assessor. This will avoid conflict situations when candidate results are considered and decisions subsequently made.
The Committee agreed that members of the Quality Assurance Committee should be encouraged to participate in other non-assessment components of the Peer Reviews such as the educational session on the Learning Portfolios, and act as invigilators for the written test of clinical knowledge.
It was further agreed that members of the Quality Assurance Committee could continue to take part in the Quality Assurance process as question and case writers or as standard setters.
The Practice Review - Four Pharmacists Describe Their Experiences
Pharmacy Connection, Sept/Oct 2002
How will you feel when you get a letter from the College asking you to submit your learning portfolio and come in for the [Peer] Review?
In this article, Pharmacy Connection's former Associate Editor Alison DeLory, reports on the experiences of four candidates (from various backgrounds) who have been through the [Peer] Review. Candidate's names have been changed to protect their privacy. (Please note that this article has been excerpted from the original which was published in Pharmacy Connection Sept/Oct 2002. For more information, please contact the Continuing Competency Programs).
While these four pharmacists each practice in different settings and have varying levels of experience, they all admit that they felt very apprehensive when selected for the [Peer] Review. They prepared in various ways and came to the College somewhat unsure of what to expect. Yet after having gone through it, all four said the [Peer] Review was less intimidating than they had expected, and their opinions of the process, and how they regarded their own skills, were raised by the experience.
We hope that in our sharing of their stories you have gained some insight into the process and better understand how to approach the [Peer] Review should you be called in. We also hope that you remember that if you are called, you will be among colleagues and friends experiencing similar emotions.
So look for future items on the QA process to be published in Pharmacy Connection as well as on our website. The College wants you to perform your best and will continue to collect feedback, monitor the program results, and devise new ways to help all members maintain quality in their practices.
Brian, Over 30 Years Practice, Passed First Time
Brian, 54, works as a freelance pharmacist in a few independent pharmacies in Eastern Ontario. Previously he worked in hospital pharmacy and owned a community pharmacy.
Until a year ago, Brian also served as a regional CE coordinator for the College. He was busy coordinating and pursuing learning opportunities but found it hard to keep his notes organized until he downloaded the sample professional profile and learning portfolio from the OCP website.
He says the College, especially Manager of Continuing Education Programs Bernie Des Roches, helped him prepare for the [Peer] Review and did a good job explaining the process. "I knew it was just a matter of seeing whether we were on target. I think people misunderstand the process. I felt comfortable with what would happen. I was curious to learn if I had any weaknesses, according to the College, that I needed to work on."
Brian watched a video that the College supplied about the [Peer] Review to help him prepare. He also did the Clinical Knowledge Assessment on the website and scored fairly well on the sample multiple choice questions. "That put my mind at ease." He wanted to be stronger in certain areas so did some reading on asthma and diabetes feeling this would give him an edge.
"The hardest part was trying to stay calm and focused. To do what I do at work," says Brian. His [Peer] Review day began with cookies, coffee and a discussion of portfolios. "I had a chance to sit back and relax which helped a great deal."
He was nervous about the role-playing and wished that the College assessors could have watched him interacting with real patients in one of the pharmacies where he works instead. (All [Peer] Reviews take place at the College for administrative purposes and so that they can be as consistent and fair as possible.) "In your pharmacy you have more distractions but you're comfortable. You get into a pattern," he says. But in the end he says all the mock patients had presented themselves with "very everyday, ordinary things."
Brian scored well on the written exam, and emphasizes, "There are no trick questions." But he says he found the clinical questions in the [Peer] Review more difficult than the website samples.
He enjoyed the feedback session and listening to the comments of his fellow candidates at the day's end. He believes the profession is right to check for minimum standards and says if you can't meet them, you should have a program set up for you. Overall, he says as he left the College that day, he felt good. "I was glad it was over and surprised that, although not easy, it was fair."
Jon, Over 25 Years Practice, Passed First Time
Jon, 63, says he felt "nervous as hell and angry" when learning he'd have to undergo the [Peer] Review. "It's like you've been on your own for 25 or 30 years. Now 'we (the College) are going to tell you whether you know what you're doing.' It's a lack of respect."
Jon has been practicing since 1963 in industry, community and hospital settings. These days he works at both a hospital and rehabilitation facility in Central Ontario.
Home study, says Jon, was never his forte. He had at one time been active in his local pharmacists' association, but admits to neglecting continuing education during the busy years when he owned his own retail operation. Since he joined the hospital in 1996, he has been going to its monthly lunch-and-learn sessions and simply saving notices about the events for his portfolio. "I find the documentation to be frustrating and time consuming," Jon admits.
He filed two self assessments: one each from the hospital and community pharmacist perspectives. "The differences were striking," says Jon. "Depth of clinical knowledge is higher among hospital pharmacists . . . but they're considerably weaker in communicating with patients."
To prepare, Jon reviewed the sample case studies in the Clinical Knowledge Assessment on the College website (www.ocpinfo.com), finding them "very realistic." On the day of the [Peer] Review, he says it was hard to resist the temptation to look up everything using the reference books provided for the written exam, but there wasn't time for second-guessing and besides, his initial reaction was almost always right.
He found the mock patient scenarios easy, saying that as a former retail pharmacist he did that kind of work everyday. In fact, he says interacting with patients in a real pharmacy is more difficult because of distractions like the phone and other patients. Though candidates are given five to 12 minutes per encounter, Jon says "I was done in three minutes."
Jon found the information sharing session on how to maintain the learning portfolio "a complete waste of time," and says he had little to contribute to the feedback session at the end of the day because he was so mentally exhausted. Yet even still, he says the day was better than he had predicted it would be. "It's not an insult to our intelligence or integrity. It's reasonable to expect a [Peer] Review sometime in our lifetime." Jon would also like to reassure pharmacists, especially older members, saying "We know more than we think." He concludes, "It's a good system, I guess."
Alice, Over 20 Years Practice, Passed Second Time
Alice, 51, has more experience than most with the [Peer] Review – she's been through it twice. The first time, she scored well in the written exam but her communication and interviewing skills during the mock patient scenarios were weak.
"I was nervous. I went through the first scenario very quickly. I didn't think to ask lots of questions. Things started off badly," says Alice.
Alice has been working as a pharmacist since 1975. It had been many years since she had written an exam and she didn't look forward to being judged. To prepare for her first [Peer] Review she went through her pharmacology books and looked through notes she had collected from various seminars she'd attended over the years. She reviewed the sample cases on the College website and read Pharmacy Connection articles about the [Peer] Review. She said that helped her know what to expect. "There were no surprises.
Despite her preparation, she admits she did not perform well when push came to shove. "Because I was not in a pharmacy, my mind was not in that practice setting," she says.
When she received her results by mail she learned that she'd have to meet with a peer support group at the College. She says the group gave her useful advice to help ensure she would be successful during reassessment and Bernie Des Roches provided a list of courses that would assist her. When Alice explained that a close family member had recently died, she says the peer support group was understanding and gave her about a year before scheduling her second [Peer] Review.
She used that year to continue reviewing her pharmacology textbooks and take courses (provided by her employer) to strengthen her skills in asking patients open-ended questions. She also ordered a home-study continuing education course on patient counselling.
Alice says she felt more confident on her second attempt. She says she was very thorough with the mock patients in the oral portion, probing more thoroughly and providing more detailed advice than she had during her first attempt at the [Peer] Review. Alice also wore her lab coat, which she says helped. "It made me feel like I was at work.
Alice's results improved and she believes her experiences with the two [Peer] Reviews probably made her a better pharmacist. "I'd reviewed everything because I didn't know what they'd ask." Her main advice to others facing the [Peer] Review would be to "remain calm, and pretend you're at work."
Joyce, 9 Years Practice, Passed First Time
Joyce has been working as a pharmacist since graduating in 1993. At 31 she is younger than the average [Peer] Review candidate. "I was in one of the first years (in pharmacy school) to have lots of practice in role playing. That helped a great deal with the mock patient scenarios."
Joyce has been a staff pharmacist and manager with a franchise pharmacy, and now works primarily in operations at the company's headquarters. She maintains a Part A registration by working part-time in dispensaries as needed.
When she learned she'd been selected for the [Peer] Review, Joyce enrolled in a consultant pharmacist accreditation program offered by her employer. This, she says, helped her review major topic areas like cardiovascular health, diabetes and asthma. She also went through her journals and textbooks and brushed up on her knowledge of new drugs.
Joyce worried that because she only works directly with patients on an occasional basis, she would encounter patient scenarios that she's not familiar with. But that wasn't her experience. "All the situations were very typical, and common. They were things I have exposure to all the time."
She also worried that because she is a slow reader she'd have trouble with the written portion. Joyce answered all the multiple choice questions without using the reference books to save time, and when she found she'd finished early she returned to the answers she'd doubted and verified information using the available materials. Candidates are allowed to bring two additional books into the written portion, and Joyce says she agonized over what to choose. "At first I wanted to bring in lots. But in reality I didn't need anything extra. What was provided was enough," she says.
Joyce admits to being "freaked out" at times, but says the facilitators worked hard to make everyone comfortable. The only really stressful moment for Joyce came during the mock patients scenarios. Candidates are allowed one practice scenario to begin, for which they aren't marked, and Joyce felt she flubbed it. But when it was time to begin with scenarios that counted, Joyce was faced with the same evaluator that had seen her falter. It also bothered her that this evaluator sat facing her. In other scenarios evaluators sat behind her and she says it was much easier with them out of sight. Though she worked hard to control her nerves, she remembers one unsettling incident. "I was holding a piece of paper in front of a patient and it was shaking!"
When it was over, Joyce says she felt sure she had passed but was still nagged by doubts about her performance. She was "pleasantly surprised" when she received her results showing that she'd done very well. Her advice to other pharmacists facing the [Peer] Review would be to familiarize themselves by referring to Therapeutic Choices, and mainly to relax. "If you don't have problems in everyday practice you won't have problems in the [Peer] Review."