Here at Qualified, we've recently introduced some new features to our already highly customizable Question & Answer challenges, and we thought it might be a great time to reintroduce Q&A challenges and highlight some of the great ways you can use them to assess candidates and students.
Q&A vs Code Challenges
We have two primary ways you can test candidates using Qualified Assessments.
Our primary challenge type is a code challenge, which is designed to represent a work sample and test a developers ability to perform in a realistic scenario, using code to solve problems.
We introduced Q&A challenges as a way to expand on your ability to learn about candidates. Q&A challenges feature multiple-choice and free-text questions, and allow your team to dig deeper into the developer's knowledge, learn more about them and their personality, and gather basic profile information.
Using both challenges together will expand your ability to gather valuable evidence and make efficient and effective decisions.
Learn more about the different types of challenges and their uses in our content developer guides.
Q&A Challenge Basics
The foundation of Q&A challenges is a series of questions. We have two base types for questions, but each of these base types can be customized to fine-tune the answer to get a signal-rich answer.
Multiple-Choice Questions
Multiple-choice questions are the easiest to build, and allow for instant, automated scoring. With our multiple choice format, you can create questions with a varied amount of correct answers, and even allow for some answers to be rated higher than others, or provide partial credit.
TIP 1: Use Timed Questions to Prevent Researching
One of our more powerful features is to enable per-question time limits, which forces the candidate to read and answer each question one-at-a-time, and also limits the amount of time they have to choose answers. Learn more about how to use time-limits in this article.
TIP 2: Use Custom Weighting to Allow Partial Credit
You can apply custom weights to each answer, allowing for a more refined scoring process. You can use this to help determine job fit (by allowing some scoring for less-desirable skills), create knowledge-based questions that have "partially-correct" answers, and more. Learn more about how to use weighted choices in our docs here.
Free-Text Questions
Free-text questions allow candidates to type in an answer freely. Free-text can be trickier to score, but eliminate the possibility of simply guessing an answer. You can customize the appearance of a free-text question in several ways, and still validate the answer in an automated fashion.
TIP 3: Select the Right Answer Format
You can style the free-text question in many ways, from single short-answers to very-long-form essay answers by using a highlighted, language-specific code editor. Make sure you select the right format to help your candidate provide the kind of answer you expect. Learn more about free-text answer formats here.
TIP 4: Validate Free-Text Answers Automatically
One of the more time-consuming aspects to free-text questions with traditional quizzes is manually reviewing and validating each answer. We wanted to alleviate this, and added in a powerful validation matching toolkit. For each question, you can define a series of matchers that check to make sure certain phrases are either present, or ensure they are not used. You can even use regular expressions for complex validation. Learn how to apply validation matchers in our docs.
Advanced Q&A Features
Beyond the question formats, there are several options you can use to change how your challenge works.
Challenge Time Limits
The entire challenge can have a time limit assigned, which prevents a candidate from researching every answer. Time limits can also help candidates keep track of tim, so they don't get too side-tracked with just one challenge.
Randomize Question Order
A relatively new feature, this allows you to shuffle the questions into a different order for each candidate. Once the candidate starts the challenge, their order is locked-in, so they will have a consistent experience even if they have to leave and return to the page.
When reviewing the candidate's answers, you will see the questions in the original order, but you have the option to see them in the order presented to the candidate. This could be helpful in case the order of the questions has affected the candidate's answers.
Question Points & Scoring
Each question defaults to being worth a single point. If you want to increase the value of a question relative to other questions, you can change the number of points it's worth.
You can also disable partial credit for questions with multiple answers or validators. This means that the question is now all-or-nothing, no matter how many points it is worth.
Q&A Challenge Wrap-Up
Hopefully you learned something new about Q&A challenges in this article. If you are already using Q&A challenges, you may want to take some of the information in this article and revisit your challenges to see if you can improve the evidence you are gathering.
If you are new to creating challenges, our content creator docs provide detailed, guided lessons into how to develop your own assessments. Inside you'll learn how to analyze your needs, build a plan, and build an assessment from existing challenges and custom challenges.