Minimizing Cheating on Professional Certification Exams, with Guest Pete Van Dyke, Certification Security Program Manager at Amazon Web Services
On episode 014 of The Score, we look at cheating from a different angle than we have before. Our guest is Pete Van Dyke, the Certification Security Program Manager at Amazon Web Services, the office responsible for minimizing cheating among people taking professional certification exams.
Host Kathryn Baron (@TchersPet) and Mr. Van Dyke discuss the issues and challenges involved in preventing cheating on professional certification exams, as well as the cost of cheating when it occurs.High points of the conversation follow. Listen to the full episode Apple, Spotify, and The Score Website. Note: Removal of filler words and minor edits have been made for clarity.
Kathryn Baron (01:57): Would you describe what you and your office do?
Pete Van Dyke (02:00): We divide our time among three different activities. One is looking at people that steal our exam content and post that online or charge money for that online. Those are known as brain dump websites. You’ll probably hear me talk about that a couple more times today. The second thing that we do is we look at what are known as proxy testers. So, individuals or organizations that take exams for candidates charge them a fee for that, and then through remote control of the computer screens take an exam for them. And then the third thing that our team works on are individuals who misbehave during their exams. So, whether that’s accessing a cell phone or hidden notes or having a third-party present…
Kathryn Baron (07:16): If I’m taking one of these exams, what can I expect before I’m cleared to actually begin the test?
Pete Van Dyke (07:22): Well, we present our exams in two different formats. One is at an in-person test center. We have literally thousands of in-person test centers all across the globe. If you were to take an in-person exam, you would schedule that. You would go in and there’s a live proctor who would observe you as you take your exam. But once COVID hit, the second modality for us, which is online proctored exams became very popular. With an online proctor exam, you don’t have to go to a test center. You can take that right in the confines of your own home, and you don’t have to interact with people live. What happens for online proctoring is that there is an online proctor located somewhere else in the world who is observing up to 16 or 18 people taking in an exam at one time, and they make sure that they’re not misbehaving.
If you were to take an online proctored exam, there’s an entire formal check-in process. We verify that the government issued ID is the same person as the person taking the test. You don’t want someone who looks like me taking the test under the name of someone who looks like you, Kathryn. There’s a very detailed room scan by video to make sure that there aren’t any learning materials, that there aren’t any secondary computers or electronic devices, any note-taking materials, pens, paper, etc. in the area.
And then there’s also a systems check. The test delivery provider looks at that and sees what kind of programs are running in the background to make sure that there’s nothing that would allow a candidate to record the testing experience and then steal content from the actual exam.
Kathryn Baron (09:07): What have people done to try to trick the security measures? Are there any anecdotes that stand out for you?
Pete Van Dyke (09:15): It’s really limited only by creativity. For online proctored exams, because you don’t have a human being in the same room, people attempt to cheat that system in lots of different ways. They may try to record the session, either audio record or video record. They may surreptitiously have notes and access notes during the exam. It’s not unusual for someone to try and have a third person, a third-party individual in the room with them to help with the exam and indicate which questions have which answers. And we’ve seen evidence in the past of people using things like recording devices built into eyeglass frames or even using earbud type communicators so that someone can communicate with them what the correct answer is for items.
Perhaps one of the more interesting things that we’ve had is when someone takes an exam with a proxy tester. The proxy tester loads software on their machine that allows them to remote control, take control of the desktop as they’re using it. So, during that hour or hour-and-a-half that they’re taking the exam, the candidate pretends to be taking the exam while someone thousands of miles away is actually taking the exam for them.
One of the funnier instances that we’ve had of exam misbehavior, we had a candidate that actually fell asleep during his exam. His head was leaned over, and he was snoring very loudly for about a 10-to-15-minute period. Yet, his exam continued to move forward because the exam proxy tester didn’t realize that the candidate was sleeping, and he was just moving forward as had been planned.
Kathryn Baron (10:59): What are some of the less obvious red flags that the proctor will look for?
Pete Van Dyke (11:20): When you take this exam, you can see the webcam capture as it’s running to make sure that your face is completely visible, and your shoulders are visible. But if you were to place that just slightly outside, if you continuously look, say down into the right or down into the left, that would be an indicator that there might be something there that that candidate is using to cheat on the exam. Leaning partially off-screen would be the same type of violation or exiting the exam completely.
We don’t allow for breaks on our exam, even bathroom breaks. So obviously getting up and leaving for a minute or two and then coming back is a sign that there’s at least the very strong possibility that the candidate was accessing information that they weren’t allowed to have during the exam.
Kathryn Baron (13:53): Have security measures increased in recent years due to an increase in the products that enable cheating and the increase in online exams due to COVID?
Pete Van Dyke (14:53): As the pandemic continued, about a year-and-a-half, two years into it, we saw about 85% to 90% of our exams being taken via online proctoring and only 10% being taken in test centers. So that created a whole new environment for us. Obviously, if you don’t have someone standing in front of you, it’s easier to misbehave, it’s easier to try and hide things, and it’s easier to have another person in the room that’s hidden from camera view. So, we had to adapt to all of that. And the proxy testers are very, very sophisticated.
Kathryn Baron (15:40): How organized are they, the proxy and all the other companies? Is it difficult to find them and maybe put them out of business or anything like that?
Pete Van Dyke (15:50): Well, the challenge for us, we’re a US-based company and a lot of the brain dump websites and a lot of the proxy testing organizations operate in countries outside of the United States. So, it then becomes very lengthy, very expensive and very difficult to pursue any type of legal action against these individuals in countries that may not even support that type of a lawsuit. So, it’s very challenging.
The proxy tester networks themselves work a lot like a multi-level marketing campaign. They advertise all over the web. If you’re on Facebook and a Facebook group about certification exams, it’s not unusual to see multiple posts a day with people offering to take exams for you. We’ve seen this on LinkedIn. We’ve seen it all over the place, even on Etsy, believe it or not, and eBay.
You have one level of their organization that is recruiting potential customers. You have another level that works with them and negotiates pricing and details, and then you have a very sophisticated technical side of their organization that actually makes the proxy test happen by taking over candidates’ computer screens and taking an exam for them.
Industry-wide, we estimate that this is a multi-$100 million-a-year business. It’s not unusual for a proxy tester to charge as much as $1,200 above and beyond the cost of an exam for someone to have an exam taken for them.
Kathryn Baron (17:19): You were talking about stealing the test questions earlier, and how do people do that? Is it that the people who are the proxies, they can take a screenshot of things because they’re not quite on the exam legitimately? Or how would that work?
Pete Van Dyke (17:36): Historically, prior to COVID and prior to the explosion of online proctored exams, there were really two ways that brain dump websites harvested exam content. One was to literally snag candidates that just finished taking an exam and say, “What do you remember from the exam? What are the questions that you remembered?” Another way was to work with a test center that was in cahoots with the proxy testing. The test center would allow someone to take photographs or to record a session where they took an exam and then sell that content to a brain dump website that would then publish it or sell it to others for a fee.
Kathryn Baron (19:07): What are some of the potential consequences to us, to the people who use different services in terms of our safety or the legitimacy of something that a person who cheated on an exam to get a job is responsible for?
Pete Van Dyke (19:32): Let’s imagine you come to me. I’m a proxy tester, and you want me to pass a Google certification exam for you. I charge you $500 plus the $300 it costs to take the test. I make arrangements and the first step of that is you have to give me control of your computer. I load software that surreptitiously allows me to control your machine during the exam. Through that process, I can load anything I want. It’s possible to load malware or spyware, all sorts of tracking information. Within the industry, we’ve seen evidence of this happening, of people that thought they were just going to find a way around taking a certification exam that ended up having banking information and personal information like social security numbers and all of that stripped from their machines etc.
And then if that were to happen to you, who do you report that to and who do you complain about? “Oh, well, yes. I was cheating on this exam, and I worked with this person who was from some country thousands of miles away, and I gave him complete access to my machine so that he could pass an exam for me. But in the process, he also stole some of my banking information.” I don’t see that part of the process happening very frequently.
Kathryn Baron (20:50): What would be the potential impact in terms of maybe public safety, that type of thing?
Pete Van Dyke (20:56): In the IT field, it’s not quite the risk, but we also see these types of behaviors and people finding ways around taking the exam for things like nursing and doctor certification.
If you were to imagine someone going through a two-year or a four-year nursing program, they get to the point where they’re going to take a certification exam that certifies them as a nurse, and rather than demonstrating fairly that they have the knowledge and expertise necessary to be certified, they pay somebody else to take their exam. It’s pretty frightening to imagine that, because now that person can go ahead and get hired and the area that they were weakest on, maybe the ones that someone’s life depends on down the line.
Kathryn Baron (21:44): What about in the sector that you test for?
Pete Van Dyke (21:47): There are many areas of the world, many regions of the globe where a certification itself is enough to get a job, and if someone is able to land a job because of a false certification, whatever it is that they work on could be affected.
Kathryn Baron (23:59): What is it that keeps you in this job that you find most interesting about it?
Pete Van Dyke (24:13): It takes about 9 to 12 months to fully develop a brand-new certification exam. Hundreds of people are involved. There are subject matter experts at all different levels that are responsible for determining what that certification exam should look like, what kind of questions should be contained that evaluate each and every one of those questions to make sure that they’re fair, that they’re valid, and they’re legally defensible.
Tens of thousands of dollars, hundreds of hours are put into these exams, and then we discovered that people are stealing this content sometimes just a few weeks after it’s been published.
Responses