Impact of LLMs on Interviewing in 2024

Is cheating during the programmer interview required "best practice" now?

Building on top of my notes in Fake Jobs I recently have seen a couple of interesting discussions around:

Now if you think the existing process of grinding "LeetCode" style questions was good you can agree with the idea from the first one:

Do the interviews in person. There's no other way.

Or the second one:

Companies: Change the questions you are asking immediately!
The obvious conclusion from these results is that companies need to start asking custom questions immediately, or they are at serious risk of candidates cheating during interviews (and ultimately not getting useful signal from their interviews)!

My take on this #

I was never a fan of "LeetCode" questions. They are a waste of time for everyone involved.
The worst thing is that they don't even measure your work performance, just how much did someone grind for the interview. And I don't even need much evidence as no one does them to get better at the job and you stop doing "LeetCode" immediately after signing up with a company. So if you learn those or cheat by using an LLM it won't matter for your job performance. Programmers day job is nothing like LeetCode so there is no incentive to actually learn it.

So what are the alternatives?

The Science Behind Making Software Engineering Interviews Truly Predictive of Job Performance

The purpose of the interview is to figure out if you can do the job. So why instead of solving puzzles you try to simulate real work the candidate should be doing?

“Work sample tests are hands-on simulations of part or all of the job that must be performed by applicants,”

Some people may be surprised, but interviewing people with work they are supposed to do (work samples), predicts how they will do their job. Who would guess?!

All of which brings us to work samples tests, the single most predictive measure of a job candidate’s future on-the-job performance.

Cheaters #

Another good aspect of using real work instead of puzzles that LLS are trained on is that you can avoid worrying about cheaters. And it's way more important than most people think. I've seen some crazy ideas for "solving" cheating in the interviews and most of them are crazy. It will put off most candidates and leave you with only the most desperate.

I recommend looking at my older post Designing a human process around pathological cases leads to processes that are themselves pathological.

Share on Hacker News
Share on LinkedIn

← Home

Want to learn more?

Sign up to get a digest of my articles and interesting links via email every month.

* indicates required

Please select all the ways you would like to hear from Krzysztof Kula:

You can unsubscribe at any time by clicking the link in the footer of my emails.

I use Mailchimp as a marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.