Home > Blogs > VMware Education & Certification

Introducing the Next Generation of Testing at VMware

Written by Guest Author Lourdes Soler 

Over the past year, the VMware certification team has been hard at work making enhancements to our development processes, exams, and delivery infrastructure. Since many of these changes are behind the scenes and not immediately apparent, we are going to begin telling you about what these changes are and how they will improve your certification experience.

For example, what if I told you that you didn’t have to wait for weeks to receive your beta exam results and subsequent certification status update, but could receive your score immediately?

How It Works

With the traditional beta exam process, all test-level and individual question-level analyses occurred after the beta period ended, which meant candidates had to wait until after the final production exam launched before receiving their beta exam results.

With our new process, under the direction of a trained measurement specialist (more formally referred to as a psychometrician), Subject Matter Experts participate in a rigorous process (called the standard setting process) before the exam release to ultimately select the best set of questions for inclusion in the exam and determine an appropriate pass score. This means that test candidates can walk away with their exam results at the end of their testing event.

The Benefits

Other than the immediate scoring, benefits include:

  • Less time burden: With traditional beta exams, candidates must sit through and answer the entire pool of questions, even taking time to answer questions that do not make it into the final exam. With our new process, candidates are presented with a subset of questions that were considered valid during the standard setting process, providing a shorter exam time compared to past beta exams.
  • No blackout period for analysis: With traditional betas, analysis begins after the beta period closes and a blackout period is instituted, during which candidates are unable to register for or take the exam. The blackout period was needed to review and analyze question and exam data and participant comments to determine which items will appear on the final production exam. Depending on the amount of data to be analyzed and the time needed to construct the production exam, this blackout period could take from 6 to 10 weeks. Because all of this analysis will now be done before exam launch, there will no longer be a blackout period.
  • No retake restriction: Traditional beta exams can only be taken once by each candidate. With production exams, candidates can retake exams under the standard retake policy.
  • No risk for recertification: With traditional betas, if taken for recertification purposes, scoring may not be completed before an individual’s recertification deadline and certification status could be in jeopardy. Candidates would need to plan accordingly and allow enough lead-time to retake the exam after they receive their results, in case they did not pass, or pursue an alternate route to recertification. With the immediate scoring factor, this is no longer an issue.
  • No “set-it-and-forget-it” mode: In the case of traditional beta exams, there is one health check done during the post-beta analyses. Now, since every effort is taken ahead of time to eliminate questions that are ambiguous, too easy, too difficult, or just bad, there is increased confidence in the quality of the content. Even so, VMware will still conduct periodic exam health checks throughout the life of the exam and make improvements as needed.

Keep an eye out for future blog posts describing further program enhancements and improvements.



Introducing the Next Generation of Testing at VMware

Written by Guest Author Lourdes Soler 

Over the past year, the VMware certification team has been hard at work making enhancements to our development processes, exams, and delivery infrastructure. Since many of these changes are behind the scenes and not immediately apparent, we are going to begin telling you about what these changes are and how they will improve your certification experience.

For example, what if I told you that you didn’t have to wait for weeks to receive your beta exam results and subsequent certification status update, but could receive your score immediately?

How It Works

With the traditional beta exam process, all test-level and individual question-level analyses occurred after the beta period ended, which meant candidates had to wait until after the final production exam launched before receiving their beta exam results.

With our new process, under the direction of a trained measurement specialist (more formally referred to as a psychometrician), Subject Matter Experts participate in a rigorous process (called the standard setting process) before the exam release to ultimately select the best set of questions for inclusion in the exam and determine an appropriate pass score. This means that test candidates can walk away with their exam results at the end of their testing event.

The Benefits

Other than the immediate scoring, benefits include:

  • Less time burden: With traditional beta exams, candidates must sit through and answer the entire pool of questions, even taking time to answer questions that do not make it into the final exam. With our new process, candidates are presented with a subset of questions that were considered valid during the standard setting process, providing a shorter exam time compared to past beta exams.
  • No blackout period for analysis: With traditional betas, analysis begins after the beta period closes and a blackout period is instituted, during which candidates are unable to register for or take the exam. The blackout period was needed to review and analyze question and exam data and participant comments to determine which items will appear on the final production exam. Depending on the amount of data to be analyzed and the time needed to construct the production exam, this blackout period could take from 6 to 10 weeks. Because all of this analysis will now be done before exam launch, there will no longer be a blackout period.
  • No retake restriction: Traditional beta exams can only be taken once by each candidate. With production exams, candidates can retake exams under the standard retake policy.
  • No risk for recertification: With traditional betas, if taken for recertification purposes, scoring may not be completed before an individual’s recertification deadline and certification status could be in jeopardy. Candidates would need to plan accordingly and allow enough lead-time to retake the exam after they receive their results, in case they did not pass, or pursue an alternate route to recertification. With the immediate scoring factor, this is no longer an issue.
  • No “set-it-and-forget-it” mode: In the case of traditional beta exams, there is one health check done during the post-beta analyses. Now, since every effort is taken ahead of time to eliminate questions that are ambiguous, too easy, too difficult, or just bad, there is increased confidence in the quality of the content. Even so, VMware will still conduct periodic exam health checks throughout the life of the exam and make improvements as needed.

Keep an eye out for future blog posts describing further program enhancements and improvements.

12 thoughts on “Introducing the Next Generation of Testing at VMware

  1. Pingback: Introducing the Next Generation of Testing at VMware – Tovmug

  2. Roman

    HI Karl,

    Thank you for sharing this information. It is definitely a progress comparing to the options we have at the moment.

    Just to confirm, what is the current blackout period for 2V1-602 vSphere 6.5 Foundations Beta Exam and VCP6-DCV (6.5) Beta Exam (2V0–622)? I believe many people are eager to know the results.

    Thanks,

    Reply
    1. Karl Childs

      Hi Roman, watch for the announcement next week with dates and information for those exams.

      Thanks,
      Karl

      Reply
  3. Pawel

    Hello,

    I am glad to hear that and hope there will be such progress.
    However for now I am a little bit disappointed especially with VCAPs exams – I mean in particular VCAP6-CMA Design.
    I tried it twice and in both attepts there is a Design question which is totally wrong. There is a scenario description which consider some requirements but then when you go to description what you are suppose to do or connect there are completely different task and items to use. Furthermore in icons/elements to built the design there are also different blocks which are not related to requirements described in scenario and the scenario is simply copied from one of the designs before. Thats why I am strongly convinced that someone miscopied it when build the design question. I am also afraid that it is the main case of my failures because as far as I know design questions are the most important, than missing this one is simply the amount of lacking points. I gained 279 points so there was only 21 to pass..

    I have reported it to education and certification teams but they simply answered that the test cannot be verified because automat checks it.

    My next bad experience is with VCAP6-CMA Deploy exam – I took it once but the whole vRA environment was not working – I was not able even to log into vRA portal. Thats why I was able to do just a few question which were related to supporting infrastructure.

    I am looking forward to see this progress…

    Reply
    1. Pawel

      Unfortunatelly, I am not talking about beta exams above. These have been GA since almost one year as far as I know and they still do not work as they should.

      Reply
    2. Karl Childs

      Thanks Pawel. I remember checking your exams, and we were able to verify your score. Yes, they are automatically scored, but even when looking at individual item differences (which we can do although it typically doesn’t change scores), the individual item wouldn’t change your overall grade.

      I am also sorry about the bad experience on the deploy exam. That is also something we are currently working on improving and are making some strides forward. Look for additional blogs in the weeks and months ahead for information on our improvements there as well.

      Thanks,
      Karl

      Reply
  4. Chris

    I’m not totally clear on this here. Does this mean that beta exams are being phased out and exams will be released as production straight out of the gate? Or is there still a beta period?

    Reply
    1. Karl Childs

      Hi Chris, we still conduct a beta process, but it will be different in that there won’t be an exam published as beta, like we have in the past. You will see the exam available as production, which allows you to receive your score immediately and removes the black out period. We essentially are able to conduct all the beta steps and analyses prior to first release, so you will only see the final, higher-quality exam.

      Thanks,
      Karl

      Reply
      1. Chris

        That’s a nice improvement to the process but I had been hoping to take a crack at the vcap7-dtms for the cheaper beta price!

        Reply
  5. Rahman

    Hello there,

    1. When are the production exams available for 6.5? I have done ICM course on 6.5 in Feb and still waiting to write the exams.

    2. I got VCA and VCP vouchers (v620 & v621) issued in March (last month). Will they be valid for registering 6.5 exams?

    Please confirm., Its been a long wait for me.

    Regards.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*