Challenge Program

by Asya Isakova, Evgeny Dmitriev 5 mins read

The challenge program is a perfect tool for discovering hidden talent by autonomously vetting potential employees and interns at the start of the hiring process. The program consists of issues that can be solved without any access to company resources and require the participants to manage their own work. Such projects give interested individuals a chance to prove themselves, learn a bit about the company and products, and produce useful results in the process. In addition, it allows companies to suggest a better fit for candidates based on their challenge results.

Watch the video where Maya describes an example of a challenge program developed by BlockShop.

Bounty Issues

Bounty issues are the main building blocks of a challenge program and should:

  • target driven people in a way that demonstrates their abilities, creativity, and work mentality (this is crucial to identifying talented issue solvers early).
  • be explicit and consist of clear tasks with defined metrics of successful completion while also providing the candidate with opportunities to go above and beyond if possible.
  • produce useful results for the company and reduce our workload in some capacity.
  • not take much time from the team to accomplish or vet/use the finished results.
  • not require an understanding of the company’s internal systems or special access/permissions to complete.

Issues Structure

Each issue requires the following parts:

  • Description – explains what the task is and how it might be used/integrated.
  • Requirements/functionality – details the specifications of the task, how results should be formatted/represented, which aspects to prioritize, qualitative or quantitative definitions of completion, and technical deliverables.
  • Specific necessary resources – tools, libraries, or files that will be needed to complete the issue.

How to Build Your Challenge

Step 1. Define the task. The challenge should be based on a real job. If it is a new position, start with job analysis: the knowledge, skills, and abilities necessary for the role. Otherwise, run through the current work tasks of your team. Check your team’s open and closed issues to collect ideas for the challenge. From these tasks, cut out everything that is too complicated and cannot be completed in several hours of work. You might also want to simplify one of the current issues for the challenge or break it down into smaller pieces that would be workable for the candidates. The more specific and unique your challenge is, the better the selection of results it will bring. If you struggle to simplify the tasks, have a look at the complex challenge tasks design strategies.

Most often, there are 3 types of candidates:

  1. Those who already have experience with the technologies in your challenge;
  2. Those who have partial knowledge and are ready to learn quickly;
  3. Those who are encountering this type of task for the first time but are ready to learn whatever is needed to get the result.

A good challenge task usually has different solutions for people with different backgrounds. The challenge also plays the role of an additional motivator: if a person is interested in solving a challenge, they will most probably be interested in working on other day-to-day tasks in your team.

Step 2. Come up with the evaluation criteria. Design the criteria against which you will assess submissions. The better you work with the challenge task itself and its evaluation criteria, the less time you will waste evaluating faulty applications. Developing a standard rating scale will also help, especially if you have several raters to evaluate the results.

Among the most reliable criteria are the following:

  • Correct answer: this is a fail/pass question. For example, a code should run without returning any errors. A report should clearly answer the task question.
  • Knowledge and skills: whether a candidate has a solid background or googled the night away to solve a challenge, the result demonstrates the level of expertise in a certain matter.
  • Reasoning: follow the logic of a candidate’s response. Critical thinking brings extra points.
  • Creativity: it is not only about fresh ideas but also about the way to apply them.
  • Task engagement: look at the solution options, data sources, and technologies used – how much effort was required to find and use these for the challenge?

The challenge evaluation should not take lots of resources from you or your team, and the more objective facts the challenge results generate, the better you can understand the work approach of the candidate. For example, “unit test coverage,” “algorithmic complexity,” and “corner cases handled” are facts. Based on these, it is easy to compare different solutions, come up with thoughtful interview questions, and eventually make reasonable hiring decisions.

Complex challenge tasks

Sometimes, you can’t come up with a simple task that can be solved in a reasonable amount of time. Other times, it’s difficult to describe the deliverable in details and anticipate all possible questions the challenge participants might have. Here are some ideas that can help you in those situations.

Provide a submission example

No matter how well you describe the task and how long your FAQ list is, people will have questions. Most of these questions, however, can easily be addressed by proving an example of a successful submission. Not only it will help people understand what is enough, but it will also highlight those who went above and beyond of what is expected of them. Check out this issue’s description to see how examples can make a difference for people working on a challenge task.

Improve instead of building from scratch

Let’s say you want to find someone capable of designing systems, but such a task takes time, requires well-defined technical requirements, and usually involves a lot of back-and-forward with the customer. Evaluating submissions can also be difficult, since everyone has their own tool sets and technology preferences, requiring even a better expert to evaluate the submissions. To address these problems, try asking the challenge participants to improve the existing design instead. Most likely, you already know the weak spots that need to be addressed and can quickly dismiss submissions that overlook them. Here’s a good example of such a challenge.