Exploring Remote Proctoring of High Stakes Assessments

Exploring Remote Proctoring of High Stakes Assessments

by | Apr 21, 2020 | Customer Solutions

Exploring Remote Proctoring of High Stakes Assessments

We recently participated in Prometric’s demonstration of their remote proctoring system called ProProctor™.  Remote proctoring has been gaining traction as an acceptable, secure delivery method for a variety of high stakes exams. More recently, remote proctoring has become acceptable for some credentialing examinations that were previously administered at testing centers, which are now closed due to the COVID-19 crisis. Remote proctoring allows examinees to conveniently schedule and take an examination safely from their homes or other off-site locations and avoid the logistical hassles and inconveniences of having to appear at a testing center.

The concern for educators and administrators responsible for high stakes assessments are exam security and exam integrity. Ensuring that the results are fair, accurate and reliable over time is most important. Frankly, cheating is always a concern. Typically, these examinations require test-takers to be at a testing location, such as a computer lab, classroom or other location, where they can be monitored as a group. These involve using controlled access computers to ensure there is no cheating or other activities that may compromise the examination. The technologies and protocols now available with remote proctoring, including ProProctor™, offer a suitable alternative to the traditional computer lab, test center or classroom-delivered assessments.

In our ProProctor™ demo, we watched how much effort and work goes into ensuring the integrity of an examination from a remote setting. The better remote proctoring systems, like ProProctor™, do involve having human proctors who monitor examinations remotely. Human proctors are typically responsible for monitoring six to eight examinees at one time. These systems use a variety of technologies, processes, and protocols to ensure that exam security and integrity is protected. Our impression was that remote proctoring protects the examination process almost as well as it would be at a testing center, and probably better than a classroom or computer lab setting where the ratio of proctors to students is often not favorable.

As we learned from watching a demo of Prometric’s system, a big part of the remote proctoring process involves the initial technology set up, authentication of the exam candidate’s identity, and verification that the candidate’s examination environment strictly adheres to a stringent set of protocols. We are talking about the assessment of a candidate’s computer, capturing a candidate’s facial image for later monitoring, a thorough review of the candidate’s testing location, authentication via a government-issued ID, and including having candidates roll up their sleeves and show their ears to make sure they do not have hidden devices! During the examination, there is constant monitoring, both from human proctors as well as AI-type technologies that examine candidate movement and behavior during the assessment. Our impression was that very few candidates could or would attempt to game the system.  It is that good.

ProProctor™ will be used for early administrations of the Physician Assistant College Admissions Test (PA-CAT) as a safe alternative to testing centers. We believe Prometric’s remote proctoring system offers a robust and safe alternative for PA-CAT candidates. Testing for the PA-CAT begins May 1st.

Thank You Drexel University!

Thank You Drexel University!

by | Jan 6, 2020 | PA-CAT Activities

Thank You Drexel University!

While developing Exam Master’s PA-CAT website, we thought it would be great to use images of actual PA students on the site. Drexel University is local to the Exam Master headquarters and was a participating school in the research phase of the PA-CAT. We were delighted to be invited to take pictures of their students and facilities.  

We walked through the halls of the College of Nursing and Health Professions building, stopping in classrooms and simulation rooms to photograph the learning environment. There were mannequins with bright eyes and tattoo sleeves laying in hospital beds in every room. Our photographer snapped pictures of simulation activity prompts, small examination rooms, and hospital-like hallways.

During our visit to Drexel, we met students who had just entered their didactic year of Physician Assistant school. When we mentioned the PA-CAT to the students, many of them acknowledged that they had taken the test during their first week of the program. The students had positive feedback about the development of an admissions test specifically for physician assistant programs and were excited to be a part of the research phase.

We were able to watch an educator demonstrate a level of patience and care while showing her students how to perform a patient evaluation of fine motor skills that was exceptional. Four students allowed us to capture the process of them evaluating each other under the close watch of their instructor. It was evident from our time at Drexel that the educators take great pride in the way that they prepare the future PA’s in their program. 

Exam Master would like to thank Drexel University for allowing us to visit their PA program to take pictures for our PA-CAT website. We understand that instruction time is valuable to both educators and students, and they were gracious enough to allow us to capture photos of that time. 

Take a look at some of the great pictures we took while at Drexel University!

First PA-CAT Webinar, A Big Success

First PA-CAT Webinar, A Big Success

by | Dec 13, 2019 | PA Educators

First PA-CAT Webinar, A Big Success

Over the past few months, Exam Master has been increasingly engaged in conversations with PA educators regarding the 2020 launch of the Physician Assistant College Admissions Test (PA-CAT). With registration for the PA-CAT opening next month, we recognized the usefulness of bringing the PA education community together to discuss important updates and the commercial launch of the test. We decided to partner with the principal investigators of the PA-CAT project, Scott L. Massey, PhD, PA-C from Central Michigan University and Johnna Yealy, PhD, PA-C, from the University of Tampa to help plan our first informational webinar.

In preparation for the PA-CAT webinar, Dr. Massey and Dr. Yealy worked with Exam Master to decide what information would be most valuable to a program deciding to implement the PA-CAT into their admissions process. We knew it would be important to provide impactful and engaging information to participants of the webinar, such as the usefulness of the PA-CAT, the opening of PA-CAT registration this January, and the test becoming available for students to take in May.

 

The PA-CAT Webinar

On December 6th, we hosted the PA-CAT Webinar for PA educators, and it was a great success. The webinar was led by Dr. Massey and Dr. Yealy, and moderated by Exam Master’s President and CEO, Matt Bader. PA educators from programs across the country gathered together to learn about things like the launch of the PA-CAT, the details of the test, and how it will benefit their admissions process in the future.

Dr. Massey and Dr. Yealy provided a dynamic presentation that provided detailed information about the PA-CAT. The review of information was very well received by the participants of the webinar. The presentation covered the official score reporting process, assessment considerations, and the content of the test.

During the discussion, Dr. Massey and Dr. Yealy provided programs with three critical reasons to use the tool:

  1. PA-CAT scores can be used to help identify candidates suitable for interviews.
  2. PA-CAT scores can be used to help select among similarly qualified interview candidates
  3. PA-CAT scores can be used to identify candidates who meet overall admissions criteria but could benefit from a pre-matriculation program.

The PA-CAT webinar participants received tips on how to strategically plan for implementation. The tips for strategic planning included considering when the admissions cycle begins, providing notice to prospective candidates prior to the beginning of the admissions cycle, and more. It is also recommended that while trying to secure approval to implement the PA-CAT educators should meet with admissions to provide information, provide evidence regarding implementation, and determine if the graduate school mandates the GRE.

 

Q&A Session

At the end of the presentation, we were able to engage in Q&A session with participants of the webinar. It was our endeavor to answer as many questions in as much detail as possible. The questions asked were similar to the questions we have heard during many conversations with PA educators over the past few years. Check out a few of them here.

Q: What have you done to mitigate bias in the exam?

Scott L. Massey – We compare the populations to see if there’s any differences in the students’ outcomes, but it’s going to take time to be able to determine that. Through our psychometrician, for example, they’re trying to look at individual questions to see if there’s any significant difference between different ethnic groups in terms of the performance. [So] there’s no exact answer for that, it’s going to take time; but the efforts are there to try to achieve that. 

 

Q: In the event that we gain approval from leadership to use the PA-CAT in replacement of the GRE, would there be any fee or cost for the program, or would the cost be only on the candidate?

Matt Bader – So the way the program is set up is the fee for the exam is paid by the candidates. That’s fairly standard practice. There’s no particular cost to the program. Now since Exam Master has relationships with many PA programs, we are looking at some supplementary kinds of things that we can do around the PA-CAT like the pre-matriculation program, for example. So, if you had your candidates and you select from a group of candidates and you get that scoring feedback, we will have a program for any of those that may need additional help. Apart from that there is no cost to the program at all.

 

Q: If an applicant takes the PA-CAT and the program does not participate, can the applicant still have the results sent to the program?

Matt – Students, as we know, apply to multiple programs. Some of the programs they may be applying to don’t require the PA-CAT, some may recommend the PA-CAT, [and] some may not have an indication at all. Our understanding is [that] so long as the student wants to make that information available to any program, that they can do so. So, I don’t see any reason why that cannot happen.

Jim – For CASPA, the student will select the schools he/she wants to send the score reports to. Also, that score report will be attached to the student’s overall record so if a school is not participating with the PA-CAT, they still would receive that score report on that individual student and they would just disregard the score. 

 

The PA-CAT 2020

Exam Master is dedicated to supporting all PA programs during the process of implementing the PA-CAT into their admissions process. We are very excited for the launch of the commercial version of the PA-CAT in 2020. There are just a few weeks until the PA-CAT registration window opens in January. The test will then be available at over 5,000 Prometric testing centers in the United States in May 2020.  It is important to provide as much information to PA programs as possible. If you have any questions or would like to schedule a conversation about the PA-CAT, please schedule here.

Click here to watch the PA-CAT Webinar with Scott L. Massey, PhD, PA-C and Johnna Yealy, PhD, PA-C.

PA-CAT: The First Step to a Career as a Physician Assistant

PA-CAT: The First Step to a Career as a Physician Assistant

by | Dec 9, 2019 | PA-CAT Activities

PA-CAT: The First Step to a Career as a Physician Assistant

Over the past five years, there has been a significant increase in the total number of applicants to PA schools. While this is partly a result of the tremendous increase in the number of accredited PA programs in the U.S., it means that a typical PA program must manage hundreds of applications. Many applicants are highly qualified, making the choice of filling the small incoming class for a typical PA program very difficult. Acceptance rates at PA programs hover around 33%, even lower for the most competitive programs.

Undergraduate GPA, undergraduate science GPA, prerequisite coursework, and prior work history are among the criteria that PA programs use to make their admissions decisions. Since there has never been an admissions assessment specifically designed for the PA curriculum, many PA educators have had to also rely on the GRE to help with admissions decisions. With difficulty in comparing GPAs from different undergraduate schools, variability in grading practices (e.g. grade inflation), and with a standardized exam that does not focus on the science subjects needed for success in PA school, PA educators have struggled to make the best admissions decisions from a large group of highly qualified applicants.

After conducting extensive research, including conversations with dozens of PA educators, it became clear to us at Exam Master that an admissions exam truly targeted to the PA profession was needed and wanted. Given the demands of the PA curriculum, PA educators felt it was paramount that applicants be assessed on their academic preparedness to ensure they could handle the rigor that awaited them —applicant familiarity with prerequisite science subjects upon admission would be fundamental to their success during the critical didactic year. 

Led by Johnna Yealy, PhD, PA-C and Scott Massey, PhD, PA-C, the research and development phase of the PA-CAT featured field testing with over 35 participating PA programs, involving over 1600 students and interview candidates. The 180-item PA-CAT research exam was developed to assess student comprehension of common basic science subjects required as prerequisites by most PA programs. Results from initial testing were tracked and correlated with subsequent student performance.

PA-CAT field testing benefitted from significant community involvement and feedback. The exam was challenging, but anecdotal feedback from students suggested that the material covered was relevant to what they learned in college. PA educator Sandra Keavey, PA-C reported  a “strong correlation between scores on the PA-CAT and academic performance.” The strongest students in her program’s current cohort performed well on the PA-CAT. Keavey sees this PA-CAT as a tool helpful for screening potential applicants and identifying incoming students who may be at risk for falling behind during their didactic training.

The commercial version of the PA-CAT is available for registration in January 2020, with the exam window opening in May of 2020. This exam features 240 items and covers the following subjects: 

  • Anatomy
  • Physiology
  • General Biology
  • Biochemistry
  • General and Organic Chemistry
  • Microbiology
  • Behavioral Sciences
  • Genetics
  • Statistics

Used as part of a holistic admissions process, the PA-CAT has been developed to help PA programs better predict which applicants are more likely to be successful in their demanding programs. There will be rich opportunities over the coming years to refine the exam, perfect it, and ensure that it meets the needs of the PA community and all of its stakeholders.

If you would like more information about the PA-CAT, visit us or contact us directly.

Why Develop a PA Admissions Exam

Why Develop a PA Admissions Exam

by | Dec 2, 2019 | PA Educators

Why Develop a PA Admissions Exam

PA educators have expressed significant interest in having an admissions exam specific to the PA community to make better-informed admissions decisions. Most PA programs rely on criteria such as prior GPA, science GPA, GRE scores, and interviews to help determine who should be admitted into their program. Often these admissions criteria alone are not sufficiently predictive of potential student success in a PA program. The Physician Assistant College Admissions Test (PA-CAT) was designed specifically as a tool to assist PA programs in determining which applicants are academically prepared for PA school. 

 

The Importance of Academic Preparedness

Over the last five years, applications to PA school have risen dramatically as the number of accredited PA programs in the United States has increased. PA Programs face tremendous challenges in identifying the most qualified applicants during the admissions process. We have spoken with many PA educators who have expressed significant interest in having an admissions exam to assist them in identifying which applicants are most likely to succeed in their program. PA educators recognize that their curriculum is rigorous, and in an effort to reduce student attrition they would like to admit students who are academically prepared to handle the demanding course load. With each PA program receiving hundreds of applications per admissions cycle, having a tool to better discern student readiness is beneficial to the community. 

 

What Subjects Should Be Covered on an Admissions Exam?

After extensive research and conversations with PA educators, Exam Master was able to identify a common core of subjects typically aligned with PA program curriculum. Educators believe that testing applicant knowledge in the most relevant subjects would be a valuable tool for making better admissions decisions. During the research phase, the pilot PA-CAT focused on assessing student knowledge in prerequisite subjects including Anatomy, Physiology, Biology, and Chemistry. As the development of the test continued, adjustments were made to the exam to improve item performance as well as to focus on assessing the most relevant concepts within each subject. The commercial version of the PA-CAT covers the following prerequisite subjects: Anatomy, Physiology, General Biology, Biochemistry, General and Organic Chemistry, Microbiology, Behavioral Sciences, Genetics, and Statistics. Questions on the test cover a variety of cognitive domains to provide a deeper assessment. 

 

Research Recap

The first phase of the PA-CAT research project consisted of field-testing the 108- item beta version of the test at dozens of PA programs across the United States. While the commercial version of the PA-CAT will be taken by applicants hoping to be accepted into PA school, the beta version was administered primarily to new PA students just prior or shortly after matriculation. In some cases interview candidates were also assessed prior to acceptance. Results from initial testing were tracked and correlated with subsequent student performance.

From 2018 to 2019, 36 PA programs administered the PA-CAT to over 1700 new students PA programs who participated in the field-testing phase of the project have reported a correlation between performance on the PA-CAT and performance in their program. One program reported that of the students who have taken the PA-CAT, the students who had the highest scores on the test are performing best in the program. 

As the PA-CAT now becomes available commercially there will be many opportunities to study the role and efficacy of use of the PA-CAT as a standardized admissions exam. As many PA programs incorporate use of the PA-CAT into their admissions process in 2020 and beyond rich opportunities for data collection and analysis will exist. During this initial launch period, a number of PA schools will be invited to participate in collaborative research to determine whether the PA-CAT is a useful indicator of academic preparedness for PA school. Based on research results during the development phase of the PA-CAT, we expect the positive correlation between the commercial version of the PA-CAT scores and student performance in PA school.

 

Looking Ahead

The commercial PA-CAT is a 240-item assessment that covers the 9 pre-requisite mentioned above. Registration for the PA-CAT will open in January 2020. Applicants will be able to take the PA-CAT at Prometric test centers throughout North America starting in May 2020. Upon registering for the PA-CAT, each student will receive a study guide and a practice test to prepare for test day. 

The true value of the PA-CAT will be based on how well it provides an objective assessment of applicant readiness to enter PA school. Thanks to broad community support and participation, PA-CAT development benefitted from extensive field testing. Over time, we expect its value will be demonstrated among a wide variety of programs and applicants.