Standard Setting is a process used to define an acceptable level of performance for an exam and to establish a passing score for the exam. For the Written Component, the passing score serves to classify candidates into two performance levels: pass vs. fail. Candidates who score at or above the passing score (i.e., pass) are judged to possess the minimum level of proficiency required, while those who score below the passing score (i.e., fail) are deemed not to possess the minimum level of proficiency required.
In accordance with industry testing standards, CAPR reviews the passing standard periodically to ensure it remains appropriate and reflects current practice.[1] We will be conducting our next standard setting study in May 2024.
Recruitment
CAPR is seeking practicing physiotherapists who have a full license (no conditions, not working under supervision) to serve as members of CAPR’s 2024 Standard Setting panel for the Written Component. Panel members may be from existing CAPR committees, past examiners or written station markers of the Clinical Component or may be in independent practice with no prior involvement with CAPR.
We are looking to form a diverse panel with representation from each province and from different professional practice contexts such as hospital, community-based, private practice, etc. Panel members must be familiar with physiotherapy entry-to-practice competencies and entry-level physiotherapy practice. No prior experience with standard setting is required as training will be provided.
What does Standard Setting entail?
Standard setting involves convening a panel of subject matter experts who are familiar with the requirements of entry-to-practice physiotherapists and who consider the knowledge, skills, and abilities expected of minimally-competent candidates when reviewing the exam questions in order to establish the passing score.
We use the modified Angoff method to establish the passing score for the Written Component. This method of standard setting is well documented and is one of the most widely used procedures for establishing performance standards on credentialing exams.[2]
During the standard setting, panel members are trained to conceptualize the minimally-competent candidate by developing and using performance level descriptors (PLDs) that describe the knowledge, skills, and abilities that different examinee groups (e.g., less-than-minimally-competent, minimally-competent, exceeds minimal competence) have in common. Then, using the PLDs and the entry-to-practice physiotherapy competency profile, panel members individually express their judgments as to how a hypothetical group of minimally-competent candidates would be expected to perform on each exam question (i.e., what percentage of minimally-competent candidates will answer each question correctly?). The question-level judgments are aggregated to determine a recommended passing score for the Written Component.
What does a panel member have to do at Standard Setting?
Participation in the standard setting requires full panel attendance at two virtual meetings (held via Zoom over two days) as well as independent work (to be completed before and in between meetings). Panel members will receive detailed instructions and training on the standard setting process and on rating the difficulty of each exam question. Ratings are done in two rounds with a group discussion in between each round. The Standard Setting will be facilitated by Kelly Piasentin, Lead Psychometrician, CAPR.
What is the time commitment?
Participation requires approximately two days of commitment and requires panel members to be available for both virtual meetings (attendance for only one of the two meetings will result in not being able to use that panelist’s ratings).
Interested panel members are asked to set aside the following dates and times:
| Independent work (~4 hours): May 23-30, 2024 | You will be provided access to the 200-item exam to complete online just as a candidate would (i.e., without the answer key). This will give you an understanding of the type and scope of the questions, the difficulty level of each question, and of the candidate experience. Candidates are given 4 hours to complete the Written Component. You will not have your time limited and may enter and leave the exam to complete the questions in shorter blocks of time if you prefer. |
| Virtual Meeting 1 (3.5 hours): Friday May 31, 2024 11:00 AM – 2:30 PM Eastern Independent work (~3 hours): Friday May 31, 2024 2:30 PM – 8:00 PM Eastern | Attend a virtual meeting with your fellow panel members to complete Standard Setting Training. Sign off Zoom and independently complete Round 1 of Standard Setting and enter your ratings into an online survey. Ratings must be submitted by 8 pm Eastern so that CAPR staff can use the ratings to prepare for the next day. |
| Virtual Meeting 2 (5 hours): Saturday June 1, 2024 11:00 AM – 4:00 PM Eastern | Attend a virtual meeting with your fellow panel members to discuss Round 1 ratings and complete Round 2 of Standard Setting. Recommend a final passing score for the May 2024 Written Component. Complete an evaluation survey of the standard setting experience by end of the day. |
Is this a paid position?
We provide an honorarium to all panel members in the amount of $500.00.
Interested in becoming a member of the Standard Setting Panel?
Please submit your interest to CAPR by April 7, 2024 if you are interested in participating as a Standard Setting panel member by completing the Application Survey at this link:
https://www.surveymonkey.com/r/2024_stdsetting_application
If you have any questions, please contact Kelly Piasentin, Lead Psychometrician at kelly.piasentin@alliancept.org
[1] National Commission for Certifying Agencies (2021). Standards for the Accreditation of Certification Programs. Washington, D.C.: Institute for Credentialing Excellence.
[2] Institute for Credentialing Excellence (2009). The ICE handbook. Washington, DC: Institute for Credentialing Excellence.
BACK
