Building on top of my notes in Fake Jobs I recently have seen a couple of interesting discussions around:

• AI is the reason interviews are harder now by Prakhar Gupta

  For past few weeks, I've been prepping for interviews. One thing that caught my eye was the difficulty of questions have dramatically increased.

  In 2021, you were expected to solve a question in DP/recursion/trees etc in 45 minutes. which is tough by itself.

  But in 2024, companies are giving leetcode hards and asking to solve them in less than 30 minutes.

• How hard is it to cheat in technical interviews with ChatGPT? We ran an experiment.

  Predictably, the verbatim group performed the best, passing 73% of their interviews. Interviewees reported that they got the perfect solution from ChatGPT.
  Questions that are lifted directly from LeetCode were no problem at all.

Now if you think the existing process of grinding "LeetCode" style questions was good you can agree with the idea from the first one:

Solution
Do the interviews in person. There's no other way.

Or the second one:

Companies: Change the questions you are asking immediately!
The obvious conclusion from these results is that companies need to start asking custom questions immediately, or they are at serious risk of candidates cheating during interviews (and ultimately not getting useful signal from their interviews)!

My take on this #

I was never a fan of "LeetCode" questions. They are a waste of time for everyone involved.
The worst thing is that they don't even measure your work performance, just how much did someone grind for the interview. And I don't even need much evidence as no one does them to get better at the job and you stop doing "LeetCode" immediately after signing up with a company. So if you learn those or cheat by using an LLM it won't matter for your job performance. Programmers day job is nothing like LeetCode so there is no incentive to actually learn it.

So what are the alternatives?

The Science Behind Making Software Engineering Interviews Truly Predictive of Job Performance

The purpose of the interview is to figure out if you can do the job. So why instead of solving puzzles you try to simulate real work the candidate should be doing?

“Work sample tests are hands-on simulations of part or all of the job that must be performed by applicants,”

Some people may be surprised, but interviewing people with work they are supposed to do (work samples), predicts how they will do their job. Who would guess?!

All of which brings us to work samples tests, the single most predictive measure of a job candidate’s future on-the-job performance.

Cheaters #

Another good aspect of using real work instead of puzzles that LLS are trained on is that you can avoid worrying about cheaters. And it's way more important than most people think. I've seen some crazy ideas for "solving" cheating in the interviews and most of them are crazy. It will put off most candidates and leave you with only the most desperate.

I recommend looking at my older post Designing a human process around pathological cases leads to processes that are themselves pathological.

Feedback or Comments?

← Home

Want to learn more?

Sign up to get a digest of my articles and interesting links via email every month.