Tuesday, January 19, 2016

​ Assessing Aptitude

​I have sat through too many interviews where the candidates claimed to have plenty of experience and they were even able to rattle off good answers to all of the technical questions, but then, when I would ask them to actually write a short program to solve a simple problem, they were totally unable to do so. Sometimes the contrast was downright shocking. Situations like this make two things readily apparent. First, requiring candidates to write actual source code as part of the interview process is crucial. Second, standard technical questions are not always effective.

The reason that such a stark contrast can exist is because most technical questions test for knowledge rather than aptitude. With enough study and memorization, candidates can give convincingly good answers to knowledge-based questions. For instance, if you ask a question like "What is the difference between second and third normal form?" a candidate may be able to give you an accurate academic answer, even if they have no practical understanding of database normalization. That doesn't mean that such technical questions are useless. Gauging a candidate's depth of knowledge in various subjects is an important part of a technical interview. However, knowledge without aptitude does not a good programmer make.

While a hands-on programming test is a great way to test for abilities rather than knowledge, doing so is rather cumbersome, so giving more than one or two of such tests usually isn't practical. Besides that, ability and aptitude are not always the same thing; just because someone is good at something doesn't mean that they will have an easy time learning new things. So the question is, are there ways to craft your technical interview questions in a way that they probe for ability and aptitude as well as for knowledge?

It takes more time and care, but yes, it is often possible to convert technical questions from testing trivia to testing ability. You can do so by focusing on judgment rather than definitions. For instance, rather than asking for the definition of third-normal form, you could ask for examples of situations in which denormalizing a database would be appropriate. Even if the candidate has memorized the definition of database normalization, they may have trouble making good judgments when applying that knowledge. However, while testing judgment rather than trivia is beneficial, that still largely only addresses ability, not aptitude. One method that I have found particularly useful for assessing both ability and aptitude is to ask candidates to analyze prepared code samples. Here's how it works:

First, you must prepare some good code samples that you can use. They should be somewhat advanced and a little difficult to understand. Each code sample should contain a solvable but unexpected bug. Each one should be a fully compilable application, but they don't need to be long; none of mine are longer than one page. You'll need to print two separate pages for each sample. One page should contain the source code and the other should show the actual output of the code when it is run.

An Example Sample (or is it a sample example?)

To give you an idea of what I'm talking about, here is an example of a C# sample that I might use:

using System;
using System.Collections.Generic;
using System.Linq;

namespace MyConsoleApplication
    class Program
        static void Main(string[] args)
            int x = 0;
            string[] y = new string[]
                { "One", "Two", "Three" };
            IEnumerable<int> z = y.Select(i =>
                { x += i.Length; return i.Length; });
            foreach(int i in z)

And here is what the output looks like:

If you are not at all familiar with C#, that's fine. You don't need to understand the details of this sample in order to grasp the purpose of it. As you can see, it isn't very long or overly complicated. It's not making use of any obscure tools. It only uses core language features that any C# programmer should at least be able to comprehend. Some of the features that it is using, however, are a bit more advanced, so they are things with which many novice programmers will be less familiar, namely:

The intent of the code is to perform the following steps:

  1. Create an array of strings populated with the words "One", "Two", and "Three"
  2. Calculate the total number of characters in each of the words (3, 3, and 5)
  3. Calculate the sum total of all the characters in all the words (3 + 3 + 5 = 9)
  4. Output the sum total on the first line
  5. Output the total for each word on the following lines
The bug in the code is where it outputs the number zero, on the first line, for the sum total, rather than the number nine. The reason it does so is because, in C#, the LINQ Select method uses deferred execution. Deferred execution is, yet again, a somewhat advanced topic, but something which even a novice should be able to understand once it's explained to them.

Using the Code Samples

Now that you've taken the time to prepare your code samples, you are ready to use them. During the technical interview, pass one of them to the candidate and ask them to explain what the code does, line-by-line. If they understand every line, that's great; they actually know their stuff. If not, that's all the better; now you get to assess their aptitude. Explain to them what they don't understand. Tell them how it works and what it means. If they have a real aptitude for programming, they should be able to learn and comprehend the new concepts, provided that you explain them well.

Once they claim to understand the code, ask them what the output would be. If they really did understand your explanation, then they ought to be able to ascertain what the code is supposed to be doing. But then this is where the bug comes into play! As in the example above, the actual output should be something different than what they would expect at first glance. If they catch the bug and describe the buggy output correctly, then that's great. If not, then that's even better; now you can assess their problem solving skills.

Show them the actual output and then ask them if they can find the bug or, at the very least, the most likely problem areas. If they figure it out, great! If not, that's all the more opportunity for you to assess their aptitude. Explain the bug and why it happens. Answer any questions they may have. Try to explain it to them as best you can. Once they claim to understand the bug, ask them how they would fix it. If they really understand your explanations, they will be able to answer how to fix it, even if the concepts were previously unfamiliar to them.

I also like to take the opportunity, at this point, to ask them if there is anything else about the code that they would change. Sometimes they may surprise you with some interesting suggestions that give you a window into their coding style. I even intentionally write some of the samples poorly, using some bad coding practices, just to see if they catch them. For instance, in the above example, I intentionally used poor variable names and an overly confusing algorithm. It would have been simpler and less bug-prone if I just used LINQ's Sum method to get the sum total.

When you are done with the first code sample, give them another one. Once you go through more than five of these samples with them, it can get a bit tedious and redundant, but don't be afraid to give them as many as you find interesting. Don't try to go it alone. Ask one of your senior-level developers to help you prepare the code samples and even have them conduct that part of the interview, if necessary.

Not only have I found this style of questioning to be one of the most revealing parts of the interview process, I have also found that by crafting the code samples properly, I can avoid asking a slew of technical questions. For instance, if I'm using a code sample that makes use of lambda expressions, then I don't really need to ask them any technical questions about lambdas. By the time we get done going through the code sample, their level of expertise on that topic will have already been made clear.

So, that's my formula for assessing aptitude. I ask technical questions that involve judgment calls rather than just trivia. I ask candidates to explain and fix buggy code samples (teaching them new concepts in the process). And finally, I ask them to write a real program to accomplish a specified task. After following those steps, I always have a pretty good idea, not just of their level of experience, but also of their natural abilities, their judgment, their problem solving skills, as well as their capacity to learn.

No comments:

Post a Comment