Conducting Interviews and Interviewing in the Age of AI Tools

Thanks for coming. This is a talk about interviewing in the age of AI. This topic has come up in every one of the discussion rooms today, so it's probably top of mind for a lot of us, either on the hiring side of this or the interviewing side.

So, the agenda is a bit of an intro here. We'll talk about AI, like it's the elephant in the room, a bit of history on interviewing for software engineering, and then skills, like things to think about for interviewing in the future.

This is me doing my best Combustion Man from Avatar. I don't know if you know the series, but it's a great one. And a little bit of background on me, I'm currently the VP of Engineering for Vercel. Vercel makes a deployment and developer tools platform that I hope many of you are using. We make an AI SDK to make the building of AI applications easier and it's very easy to build these things on Vercel, so worth a check out. I have done steak dinner interviews with the CEO, because that's how it was done early in my career, to puzzlers, algorithms, data structures, kinds of interviews once I was thinking about going to Silicon Valley. It was a complete mind shift for me. It was very different. I have failed some of those interviews miserably. I've passed some rather unexpectedly, so just know that it's a process. You get better by doing it. But this is what we're going to talk about today.

So AI. It's not the elephant in the room. Obviously we spent all day in the discussion rooms talking about AI in some form or another, and it's the topic of every Silicon Valley CEO today. Everybody building a company is thinking about what to do with it. That's why I picked the taper, and if you don't know the taper, the taper is a very cute looking animal. It's more of a cross between a horse and a rhinoceros than an elephant. And that's the thing. But I picked the taper because I think AI in the context of interviewing remains a fairly taboo kind of dangerous topic. And so I have some questions to get some baseline.

So this is the one that I'm very curious about. Does your company currently allow candidates to use AI for technical interviews? So I'm curious. Okay. And actually, maybe another form of this is, does your company have a policy with regard to the use of AI during technical interviews? Has this even been established? Okay. It's very few. I think that's why the taper is a good analogy. Nobody kind of wants to touch this one.

So okay. Do you use AI for work on a pretty regular basis? Okay. And now given the choice, you're about to do a technical interview. Would you want to use AI? Yeah, most of you. I certainly would. I use it every time I write code. I would not want to change what I'm using. So again, this gets into the opinion section. So the questions we're going to touch on today, so how are tools like the Copilot changing the landscape for coding interviews? A pretty interesting example of this, this guy's the co-founder and CTO of a company called Hatchways. Hatchways helps, they're like interviewing as a service company. And that's actually a very insightful article about AI during the interview process, but what's so fun about it is he picked the clickbait title, right? That using AI is cheating. And I think this is why it's a taboo topic, because we haven't decided, is it cheating or is it the next best thing in the world? So there's the question, should we embrace or resist AI during the interview process? And are the like leet code, traditional coding challenges, are those things becoming obsolete right now? So we're going to talk a little bit about coding interviews, a bit of history, going back to the 60s, you know, this is the reality. If you wanted to program computers, you might learn Fortran, COBOL in school, but you didn't have one at home. And so the interview processes were not typically done on computers, they were pretty theoretical.

They were, you know, can you break down problems, problem-solving questions and really about raw intellect and understanding. And, you know, that's just how it was done. And if you were smart, you could get the job. And you move into the 80s, and now computers are proliferating a bit in the 80s and 90s, and so specific programming languages become part of the interview loop, right? So do you understand syntax for C++? Can you write object-oriented Java correctly? Can you explain how prototype inheritance works, etc? And debugging. Debugging now is a little bit more part of the process. It's faster. So they're testing whether you can do it. But it's also the era of Hackers the Movie. And so what's pretty interesting about this, you know, this is like Zero Crash was this guy's name. All of a sudden, you know, Revenge of the Nerd style, programming got to be more cool and edgy and culture. And the idea that you might be using programming for malicious endeavors, you know, extract one penny for every transaction is a thing. And so the interview process for software engineering starts to incorporate behavioral interviews and understanding your past experience. And so this is pretty new in the interviewing process.

So you get into, you know, now into the 2000s and 2010s. Really practical skills, like can you sysadmin a machine? Do you understand, you know, some of the free software that's come out? Do you understand TCP IP, you know, core networking, database knowledge, systems administration. But also HTML, CSS and JavaScript. Because products being put onto the web are delivered to customers through this technology. And so how well do you understand it started to become part of the interview.

Famously, also, whiteboard interviews come out during this time where you're given a problem and in real time now, you're coding on a whiteboard in code and they're testing your syntax. Like, I've done a whiteboard interview where, you know, you're dinged for missing a semicolon, right? Like, you know, we didn't have repels to do this very easily, but we did do this on whiteboards. Very nerve wracking. If you've done a whiteboard interview and you haven't practiced it, it feels very combative potentially. But, you know, again, this is also the era of the, you know, some of the puzzlers. This was famously Google's puzzler put up on billboards in Silicon Valley, which if you did the math and did the theory and the research behind this, you'd end up at HTTP 714, you know, to a website which took you to another puzzler which you had to solve. And so basically, again, starting to proliferate the idea that multi-stage analytical problems were part of the job.

So you know, you get into the present. Right? You get into the full stack spectrum. You're expected to understand operating systems, web frameworks, programming languages, multiple ones, databases, web servers, clients, how clients interface with these backends, you know, JavaScript, CSS, and HTML, and native applications.

And you probably, if a company you work at is building consumer products, they have both of these things running in production, talking to backend services constantly. And so you have to understand networking, RPCs, how you write the code to do that. So it gets pretty intense.

And so back to this question on coding challenges becoming obsolete. This is in February. Gaspar is one of my favorite engineers from Vercel, and I had drafted a full stack coding challenge that we ran as part of the interview loop. It's a 90-minute interview. Okay? And it's a, we give you a problem. You can build a thing on your machine. You can use any tools you want to. And, you know, go to town. Build a thing.

And Gaspar writes me, he's like, oh, I've done it in two minutes using GPT and coding tools and assistants. The world has changed. And he's right. It's very interesting. Recently, this article came out. It was on Hacker News, but it was a guy who interviewed at Stack Overflow for the second time. And he was really mad. He got to very well in the interviews.

He got to his last interview with Joel Spolsky, which I would love to, although maybe I wouldn't love it. I get to my last interview with Joel Spolsky. But he gets to his last interview, and he gets a question. It's kind of like this famous one from Facebook. It's to take a decimal number, convert it to base negative two. And so he wrote on his blog, he's like, okay, this question, which is possible and doable in 40 minutes, is more really about the problem-solving process than anything. But it's the dumbest fucking question I've ever had, and I don't care what anyone else says.

So he had a bad experience. He was mad. It's an off-by-one style question, right? Rounding errors, rig-type precision. So I thought it would be fun to plug this into Vercel's AI SDK and see what kind of answers we get. Because this is kind of a Leap Code style question. Let's see if it just starts. I think I pushed the button.

Okay, look. So this is actually two models side by side. And this is me pushing the question in. So gpt40 and Gemini flash. All right, this 40-minute question is about to be done in 20 seconds. Pretty amazing. But check this part out. The output from each one is different. GPT is off by an entire power here. So what does that tell you? It tells you that actually if you don't know that it's off, you might take the code from the first one and move on to the next task. You broke it down. You picked this foundational layer. Because now we're going to give longer problems, assuming you might use GPT for this.

I do think interviewers will pick interviews now to kind of play off of this. Where if you pick a model and run part of the interview through it, you'll get the wrong answer. And if you can't correctly see that this is the wrong answer, you'll fail the interview. So it's both a real benefit and a total risk now to think about this. And in fact, this is just sort of the analysis. I plug this into GPT to ask it, you know, okay, you got it wrong by a factor of 10. What was the problem? And it was a rounding error. So again, rounding errors have always been part of these kinds of interviews and they will continue to be subject to error with AI.

All right. So what are the things, what are the skills in the future? What are companies going to value? What are you going to need to practice and bone up on? How should you think about this? I think technical proficiency with AI tools undoubtedly becomes part of the interview loop.

What will that really look like? So I think, you know, given a set of tools, you have the inner loop. This actually is the language coming from a paper that came out last week from Google on how they've been instrumenting and building AI for their software engineering team. And it's been years in the making. But these are the areas where they're doing testing, right? Like if we give suggestions in the IDE during code reviews, you know, during code search, they get great data feedback on whether it's working, right? Like you either pick that the code review bit was good and you accept it, you're in your IDE, you complete a line of code or you're in code search and you get the results you want from the AI assistant text. So AI in the inner loop is pretty common already today. But really the paper starts to talk about how to get AI into this outer loop of your development process. And as a manager, this is the part that I'm pretty excited about now, in terms of bug management, planning, code design, upgrades of libraries, things that traditionally used to be least fun parts of software maintenance, can we actually leverage AI in some of those? But also the softer pieces. I was thinking about this and talking about it in the discussion room earlier. You know, if English, say, isn't your first language, but you work with a company where that is the case, should you run your RFC through Grammarly? Absolutely, right? This is not a non-question for me. People's perceptions and their brains make all kinds of unconscious bias decisions about other people. And you're combating that every day at work. You're combating perception to build trust and to find the right things to have impact on. So I think this will play in. If you can use a tool to make your natural language appear better, you should be doing it.

This is a slide that comes from Jared Palmer's talk a few days ago at the Lead Dev Conference in London. And his presentation here was really about the AI native flywheel. This is about the development of products that are built on top of AIs, and how you should do this. If you start with the evals here, these are the prompts, right? This is where you build your testing to tell you whether the AI tools you're using are giving you the answers you expect. Because they're not unit tests, right? These are non-deterministic systems that are feeding your product at the end of the day by generating data. Influencing your models and your strategies, which then you build into the product. And now, that product doesn't matter if you don't get distribution. Obviously there's great tooling for that in the AI space as well. And then you get feedback. Oh, okay. This kind of tool, the model didn't work well when somebody tried a pathological use case. But it turns out, you know, that a thousand people came to your site and tried something roughly that a model has classified as like that use case. Now you tune your evals. And now you keep building. Understanding how to build with AI is going to be essential. And understanding how to build these pieces of the puzzle, it's quite different than just, For example, let's say I do TDD and I write code.

And I validate with analytics. I think it's getting much richer. And so this is worth knowing. And I think this will show up in interviews. So does that mean lead code is dead? You know, I don't think so entirely. Lead code has an interesting place in terms of, you know, setting a baseline for whether people can, you know, should get to the second stage of an interview, perhaps. But I also think it isn't the skill that companies really will care about. They're caring about your ability to build great products, your intuition, your ability to collaborate with other people, et cetera. So I do think, you know, lead code is probably less important right now than it's ever been. So we're back to the future. I think if you go back to those things in the early interviews that I was talking about, I think a lot of those things still matter because they don't go out of style.

Things like communication. You know, being able to talk through an interview. This is the most important part. Even of the whiteboard interviews I've done, usually, you know, if you end up on the other side of those interviews, what people talk about during the debrief is not the specific syntax of the code being written, but how the developer communicated their strategy, how they laid it out ahead of time, and then how they turned their thinking into code. That translation. That communication remains extremely valuable. So obviously, you know, knowing foundations, symbolic logic, how this turns into data structures and algorithms, that remains valuable. But importantly, I put this natural language piece here. So human-computer interaction moving towards natural language as the input to generate code, perhaps, or to generate systems, means that our ability to leverage natural language is ever more important.

Understanding how your language affects models. So this is the prompt engineering piece. Don't write it off. Because you're basically figuring out how to use another kind of programming language. Extremely valuable. And I think it will become part of tests. If I'm interviewing someone to come build things with models, I will test whether or not their ability to generate prompts is good. And how do they know if they're generating good prompts? Again, back to that AI native flywheel. So again, problem solving. Universal.

The ability to take a really hard problem that scares the crap out of you in a short interview where you're under the gun and not just blank out. Can you break it down into pieces? Maybe you can't solve all the pieces. Maybe you can solve half of them yourself during the interview, but the other half is fairly rote and you could do it with AI. Or you can identify which pieces are the right ones to shift to using an AI system. I think those will become part of the interview. One thing you're trying to figure out is what is someone's ability to spend time and be effective? If you can identify the things that should best be done by an AI versus you, that's extremely useful. And that means you know because you've tried.

This is also changing the education scene. This article kind of talks through a series of universities in the United States about the history of how they've taught programming. Traditionally, syntax like Java. When I was in school, you only learned Java. Python reluctantly ended up in the teaching rubrics for a while. But now they're starting to teach debugging and testing because these things all influence, again, back to that flywheel. This is how products are built. And this is what companies are hiring for. This is how you get jobs. So this is changing the curriculum, and I'm certain the ways to work with these models to work with AI in school will become parts of the curriculum. If you're going to school and it's not in the curriculum, go to a different school, right? It's probably a good choice.

So again, all of these things, interviews, a lot of the end of the day you get to the debrief, right? And they're talking about a candidate's growth mindset and their ability to learn. We don't know what AI produces six months from now. But if you find people who love learning and who are good at learning, who given primitives can use them, this remains and will always be incredibly valuable during the interview process. And it's a thing that I would want to look for when I'm interviewing people. So good design. This one is a fundamental and I think applies very much to the interviewing process. If you're being asked to build a thing end to end, you're being calibrated on your taste for good design. And what does that mean? Go back in time. This is the age of the unicorn. When I got to Google in 2007, they wanted designers who could code. Unicorns in the industry is what they thought of it as. Now, you know, designers who can code, engineers who can product, engineers who can design.

Engineers, it's a very different phase. But you need to learn these things and they will be in the interview process because we're looking for goats now instead of unicorns. Thank you.

What do you think about EU AI Act? Will it affect the interview process a lot by prohibiting scoring candidates by AI, especially for mass hiring? I am not the best person to ask about mass hiring. So I don't think I should even give an opinion on this to be honest. We, you know, I work I tend to work I've worked at large companies before this time and scoring was always done by people and then we did analysis of it. One of the fun maybe my funnest fact about scoring from a series of data work that was done at Google, Google had a one through four scoring system for candidates after the interview. And one thing they found was someone who got a one during the interview process and maybe two fours and a three and got hired did better and had more impact over their career at Google than people who got like three fours and a three. So I think what you find is again this like adaptability and people who cause more sorts of variance in their scores may result in better to work with. But mass hiring we do bespoke hiring at Vercel. So I am not the expert there. Thanks.

Is there any way to recognize whether a person is using AI or not during an interview? They are. They are. I think I totally believe again we're back to this question how many of you use AI during your job or life on a day to day basis. The answer is yes. So if you think that the interview is some you know laboratory of you know independent life that is a thing you could do as an interviewer and build interview loops like that. I mean you can tell if they're not using AI because you stuck them in a room and taken away their computer. I just think that that's not a very good test of do you want to work with this person. So you can obviously but I think you're much better off assuming like in my interview example with Gaspard that people could get to a slightly working prototype in two minutes instead of 20. But it doesn't change how I'm going to evaluate the outcome of the interview. It means okay cool they got something working in two minutes. Now where do they spend their time? Where do they show me the craft? Where do they show me they know how to do something that will impress me or blow my mind that remains true from every interview I've ever done. And I think that now it's just OK cool the basics are no longer as impressive. But the fundamentals and how you leverage them and the problems you can identify those remain incredibly important.

Our next question about juniors and beginners would it get harder to land the first job as a junior when simple tasks are now done with AI? A lot of fears regarding this. Yeah there's a lot of fear here. You know I don't hire juniors to do simple tasks. So I think as a junior looking for your first job this is the most important time for you to pick the best first thing.

So I think as a junior looking for your first job this is the most important time for you to pick the best first thing. If you get out if if this took you out of a job as a junior you don't want to work there. You want to go somewhere where you can learn the most. You have the least to lose at this phase in your career. So again I don't think this should equate to you losing opportunities. If you lose an opportunity for this reason you are applying at the wrong company.

In an interview they asked me if and how I use AI encoding but they didn't choose me in the end. What intention do you think this question may have? I think this goes back to identify you know if you break a problem down into five steps and you know how did you do that? Okay. Which parts did you use AI for? At that point I think an interviewer has a perception of whether you've made the right choices in that you know did you pick the right pieces to give to an AI and why? And did you pick the right pieces to do yourself and why? And in my example of the relatively rote conversion example if you picked AI to do it and you didn't know how to debug it you would have failed the interview, right? So I think this is part of knowing where to deploy and where not to deploy AI. Knowing what it's good for, what it's not so good for today. This is going to change though, right? Like that mistake won't happen and it's a prompt failure, right? I asked the question without providing the instinct that okay when you round that it's a very important part of the question so when you get better at prompting, better at using AI, you know how to use it correctly. Again coming from your experience of doing it. So if they're not teaching it in school, which I'm not sure they are yet, spending your time learning how to do it to prepare for interviews seems incredibly worthwhile. Like everywhere you need to try to do it several times and then you will handle it.

How do you integrate LLM usage when writing and debugging code for projects under NDA? I think in this case if you're in this situation, building your own models and working on local systems makes a lot of sense. That seems wise. But even NDA, your system, your computer may not even be legit in this case. You might get a VM somewhere else. So in that case, learning how to spin up an assistant or a system, a model you can ask questions to in a VM seems like from scratch would be a valuable thing to learn. Are there any thought leadership articles you can recommend on this topic of changing your interview process? I think the Hatchways blog is at least decent. There's not a lot. In my research for this talk, I didn't uncover a whole lot of articles about this. Again, I think it's a little taboo still and people aren't willing to spend quite as much time talking about it or publishing about it. But I think we'll see what happens in the industry. People's experiences, they're going to write blog posts about it like this guy. I think we'll start to see it. Okay. So maybe you like some of the questions. I have a lot of them here. What's your biggest green flag in an interview? That's a good one.

I go back to, you know, the first thing that happens is can someone, do they take the time, do they take a breath, and do they communicate before they start writing code? Usually, you know, someone who asks questions to make sure they understand the question that I'm asking is key, right? The repeat thing, you know, this is like classic psychology. If you're trying to debug a thing that doesn't work with someone else, usually you try to do reflective listening, right? You repeat back what you think the problem is. So that's a good first step and is usually the first step of anyone seasoned in interviewing, they start there. Then they lay out how they think they'll solve the problem, and then you agree that that's a good strategy, or you tackle parts of the strategy, and then you write code. And then I'm looking at, okay, do you have the vocabulary and fluency to go from natural language to code? Yeah. But starting with questions and validation is very much a green flag for me.

Yeah. I think we have time for last question here. How can we educate companies that are still very much against AI? The top one. You don't have to. They're going to die. Right? I mean, it's... They won't be around. You don't need to educate them. Find another company. There's another one I'll take, which is what should you focus on learning as juniors in the age of AI? And I said this during the panel discussion. Build something for one of your friends, your family, someone you care about. It could be you, but I actually think it's nice if you pick somebody you really care about and try to build something for them, because you're going to go show it to them and see if it solves their problem, and in this process, use the AI native flywheel to build something end-to-end, a product on the web that they can interact with. And then they can tell you if it's good or not. And then you can make it better, and see if you can make it better and improve the flywheel. And if you do that for someone you care about, it's the best incentive as a developer. Right? If you see them struggle with the thing you built for them, you're going to feel terrible, and you're going to go fix it. If you see them succeed, you're going to feel great, and they're going to feel great. And that is what got most of us, I think, into the industry, is saving people time and making people feel better, and accomplish things. So that's the strategy.

Yeah. That's amazing. So thank you very much, Lindsey. So let's applaud. Thank you all. Thanks. Thanks.