internship - Is there a correlation between passing fizzbuzz on paper and (good|effective|productive) programmers?

I understand the motivation to weed out non-programmers from programmers. It seems like a lot of blogs either conclude or insinuate that candidates who fail fizzbuzz questions don’t make good developers. Of course, the term “failing” encompasses a number of scenarios:

  1. Fumbled logic i.e not testing if i %15 == 0 before the other two conditions.
  2. The loop not counting from 0 through 100 i.e a classical for (int i = 0; i < 100; i++)
  3. Using the wrong operator i.e using / instead of % in one of the conditions
  4. No solution at all (Candidate refuses to write any code)

Of course, the fourth scenario is a big NO but if you were an interviewer and a candidate submits code on paper under scenarios 1, 2 or 3 and you don’t offer them an opportunity to walk you through the code to understand how they arrived at a particular solution or redeem themselves from a silly mistake they can easily rectify, would you conclude that they failed the test?

Would you fail someone who has correctly formulated a solution to a problem situation but fails to write on paper code that would give the desired output on the first run? Does it sound unproductive if the code doesn’t give the desired output on the first run, and is it a realistic expectation at all to always produce code that gives a desired output on the first run when creating software?

Would you hire a developer with a good reference of 2 years of experience working in XYZ technology who submits code under either 1,2 or 3 scenarios? Of course, you would hire a developer with the same credentials who passed the test, but would you base the entire hiring process on whether someone gets the fizzbuzz challenge right or wrong on paper? And while we are at it, would you conclude that the former is better than the latter?

Does the fizzbuzz challenge factor in the fact if people are capable of making stupid mistakes even in the comfortable environment of a text editor they could fare worse on a whiteboard or sheet of paper? And If you are judging someone on the basis on those mistakes instead of the overall thought process, testing, coding style, readability, debugging, understanding requirements, etc. how effective is the fizzbuzz challenge in determining a good hire?

Or perhaps, the fizzbuzz challenge could as well be a misunderstood test after all candidates make mistakes on a dry erase board under interview pressure and sometimes get it right or wrong depending their mind set. In that respect would you agree to the fact that it’s just a test to see who has a clue vs who couldn’t program their way out of a wet paper bag?

Is there evidence showing that candidates who can write the fizzbuzz challenges turn out to be productive programmers and those who don’t are not productive at programming whatsoever? Evidence provided thus far has shown that a small number of candidates can whip out fizzbuzz under five minutes and the rest cannot (Are we talking about a solution that produces desired output on the first run or a decent solution that shows competence?). IMHO it does seem that fizzbuzz checks to see if people can reason through a problem and break it into components. The skills are essential for effective programming, but this doesn’t verify the connection between good programmers and fizzbuzz performance on paper?

I'm not looking for opinions, just facts. If you don't have numbers to back up your reasons keep your opinions to yourself, please. Thank you.

Edit

tl;dr - Is there any kind of test such that if administered during an interview process can give an interviewer some perspective into how (good|effective|productive) a candidate is for a developer role?

3 Answers

  1. Franklin- Reply

    2019-11-14

    Of course, you would hire a developer with the same credentials who passed the test

    Not necessarily. As you noted before this, the fizzbuzz test is a quick way to weed out totally clueless candidates quickly to save everyone's time. So the answer to

    would you base the entire hiring process on whether someone gets the fizzbuzz challenge right or wrong on paper?

    is No. It is necessary, but not sufficient in itself, to get hired.

    As for your detailed questions, I personally am not strict with the criteria. I would expect a decent programmer to pass this test easily. However, some people may be nervous and make stupid mistakes like the above in a live interview. So I can imagine scenarios where a candidate misses some use cases or fails to write 100% correct (pseudo)code - like your examples 1-3 above -, still (s)he demonstrates her general problem solving skills and applies logic correctly. In this case, with a guiding hint, she should be able to find and fix the problem and get a 100% pass on the fizzbuzz challenge. However, if she can't complete it even after receiving a few hints, that's likely a fail then, although not automatically - there is always room for individual judgement and further discussion if needed.

    Is there evidence showing that candidates who can write the fizzbuzz challenges turn out to be productive programmers and those who don’t are not productive at programming whatsoever?

    I am not aware of any studies on this topic, but IMHO it would be difficult to arrange such in a rigorous enough way, and it may not be worth the effort overall. The original Fizzbuzz test has been published well over a decade ago so by now it is widely known, thus candidates can prepare for it specifically, including even learning the solution by heart. So you would need a different variation in real life, maybe even at each interview. Which in turn makes it difficult to get any standardized results.

    Not to mention that the root cause of this whole controversy is the difficulty of even defining what being a good / productive programmer means. Without a good enough definition, how can you even try measuring any correlation to anything?

    At any rate, so far I haven't heard any compelling arguments on how it would be possible for someone to be a decent programmer but at the same time unable to solve Fizzbuzz. The whole point of the test is that this task - while being a real, concrete programming challenge including fundamental concepts like variables, loops or conditions / branches - is significantly simpler, quicker and easier than any programming task a professional developer is likely to ever encounter in a real job. So I can't imagine how someone who fails Fizzbuzz would be able to solve real life programming problems at an adequate level. Then again, this may show my lack of imagination though, so I am open to counter-arguments :-)

    Update

    Is there any kind of test such that if administered during an interview process can give an interviewer some perspective into how (good|effective|productive) a candidate is for a developer role?

    If you are thinking about some standardized test which can be evaluated automatically, the answer is no, and there never will be. Even though a lot of hiring managers seem to keep looking for it in new and newer flavours, that doesn't make it any more realistic. Which is not to say that such tests are totally useless, just that they should be applied carefully and in the right context.

    As most of the answers here point out, even during a really trivial test like Fizzbuzz, there are lots of subtle considerations which require human experience, practice and judgement to decide correctly. And being a good developer takes a lot more than passing the Fizzbuzz test. Sheer coding is only a small, albeit important part of the SDLC.

    Not to mention that it's not enough if the candidate is a good developer in general: (s)he must be a good developer in the actual team which is looking for a new peer. The same guy can be a perfect fit in one team and a complete failure in another.

    A worthy, albeit subjective :-) read on the topic is Joel Spolsky's Guerrilla Guide to Interviewing. It has some say related to tests too:

    The second worst kind of interviewer is the Quiz Show Interviewer. This is the kind of person who thinks that smart means “knows a lot of facts.” They just ask a bunch of trivia questions about programming and give points for correct answers.

    [...] software teams want to hire people with aptitude, not a particular skill set. Any skill set that people can bring to the job will be technologically obsolete in a couple of years, anyway, so it’s better to hire people that are going to be able to learn any new technology rather than people who happen to know how to make JDBC talk to a MySQL database right this minute.

  2. Fred- Reply

    2019-11-14

    Fizz buzz is a good test, at least in my experience. The point of the test is that it gives the following :

    • The exercise is trivial so shouldn't take long in the interview to do
    • It tests some basic coding: loop, mod, if etc
    • It doesn't follow any design patterns or real world tasks, so unlikely to have been encountered unless you've done the test before.

    I'm not looking for a perfect answer, but the ability to talk it through. If you know the solution I still expect you to be able to discuss it, and maybe then start code reviewing it to see what you say.

    If you can't make a decent attempt it raises alarm bells which I can then drill down into. You'd be surprised how many 10 yr+ veterans cannot do it, and it really shows up issues.

  3. Gabriel- Reply

    2019-11-14

    I would hire anyone who did 1, 2 or 3 if they otherwise seem qualified. Four and its equivalents are what you're looking for. Fizzbuzz won't help you determine if someone's a good programmer; it will let you determine who is a programmer at all. There are people who apply for programming jobs, and may even have a bachelor's degree in computer science, who simply do not understand a for-loop. I do not mean that they make off-by-one errors; more like they do not understand the concept of iteration. These are the people you are trying to weed out.

    With all the rest, you go on and do your regular interview process.

    To put it in terms of your title question:

    Is there a correlation between passing fizzbuzz on paper and (good|effective|productive) programmers?

    No, there is a correlation between making a reasonable attempt at fizzbuzz and programmers.

Leave a Reply

Your email address will not be published. Required fields are marked *

You can use these HTML tags and attributes <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>