Interviewing in the Post-LLM World

Okay, awesome. Thanks. Let's go. Let's kick it off. You'd be surprised to see how many companies are still struggling to adapt their interviewing processes into this new post-LLM world, right? Where the difference between humans and machines are getting closer and closer and it's difficult to tell them apart from each other. It feels a little bit like we're stuck in the 2020s, right? And having had such a big change in the industry, it feels like we've lost something. We've lost the way of doing interviews as we're used to for the last I don't know how many years. And that's why I think a lot of these companies are going through the five stages of grief, right? And here I have a couple of examples so that you can identify where your company is in this journey, let's say.

So, you know, the first stage is always denial, right? And in the denial phase, if you see a company in this phase, you'll often see them, you know, they'll try to ban AI from their interviewing processes or, you know, they might remove the take-home assignment because they know that AI will be doing it instead of the human itself, right? Or they're just obsessed with having everything live. They'll invite you to the office and they'll observe everything you do on site, right? So that's the denial stage. Then comes, of course, the anger stage, right? Where interviews will suddenly become unrealistically more difficult. You'll start getting these trick questions spread all over the interview process so that they sort of catch you off guard, right? Or they might even, these tests suddenly become sort of purity tests where they're looking for a unicorn engineer that doesn't even exist. And then they're trying to ask you all these difficult questions just to see whether you're capable of answering them yourself or is it actually sort of AI who's going to get stuck at some point, right?

If you made it behind the anger stage, you're in the bargaining stage where you'll see that, you know, you'll come to an interview and they might tell you to, you know, they might give you the LLM of their choice and not yours. Or they might tell you, they might give you weird requirements that you don't really understand and so on. And eventually this will create a sort of unclear, you know, experience for the person being interviewed, then you will not understand what they're looking for and what they're not looking for. And, and eventually, you know, almost getting there. We're in the depression phase now where companies are sort of giving up on the way they do interviews. They don't really know what works. They don't really know what does work and what doesn't work, right? And they're trying to constantly change their interview cycle, which is making more difficult for the interviews as well as for the interviews to understand what is expected of them, right? And eventually they sort of introduce more and more stages, right? They might be new, a new stage being added like the AI check stage or a LLM usage stage or whatever. All these new stages being added, which makes the experience even worse. Eventually, if you finally, you know, come to the end and you've accepted that, that it's a different world now, you will find yourself that, you know, you finally have a clear process of what you're going to do, how you're going to ask questions and how many stages you will have and so on. You should definitely encourage LLM usage during all of those stages of the interviews. And hopefully along the journey, you'll be focusing on their, on their sort of reasoning skills instead of the output that has been given to you, not the code itself, but how do they achieve that code? Right? I'm Daniel. I'm a team leader at remote.com. I'm the coauthor of the engineering managers compass. I'm a blogger at incremental forgetting.tech and I love hiring. And I think partially that's because I've spent quite some time in a couple of scale-ups where I had to do in some cases four or five interviews a day for years, years long, right? And it added up, it added up. And I always tried to optimize it. I always tried to understand how I could do these more efficiently and, you know, and eventually end up with better results down the line. Right? So a lot of my friends ask me, why, why do you love hiring so much? Is it because you love talking to people? Is it because you love experimenting with interview questions and so on? And actually the, the answer is way simpler than that, right? It's because it's the most important thing there is in a tech company and actually any company, I would, I would argue.

Here's a quote from Laszlo Bock who's the ex SVBP of Google People Operations. And he says, you know, great people are a company's most important asset. Hiring is the most important thing you do. And I could have easily picked any other company that that is, you know, significantly large and impressive in this day and age. And I could easily find a very similar quote from any leader that you'd see. It is definitely the most important thing, right? But before we sort of go ahead and, you know, think about how we're going to change our interview process, what kind of questions we should ask and so on, we should actually take a step back and think about what we're looking for. What are the skills, the techniques? What are, what is it that we're actually after? And even when we know that, how do we make the question to actually get that, understand whether that skill is present or not? Right. And in order to do that, you know, let's take a fictive role and, you know, a role that potentially is sort of valid today, the AI Native Engineer role. The first thing I think you should do is sort of sit down and think about that job description, right? What is this person going to do on a day-to-day basis? Who will this person be working with on a day-to-day basis? What are the KPIs that they're expected to improve? And, you know, the level of support that they're going to get from the company. Are they backed up by the CEO? Are they expected to sort of flourish on their own? All of these things need to be, I think, decided up front, right? You need to really write these things down so that you have a rough understanding of what is it that you're looking for, because without that, you will not be able to find that thing, right? Like you need to tailor your questions around this thing. When we think about the sort of, you know, the type of things that we might be looking for, you know, on the left, we have like the what, right? Like it might be a specific skill that you're after or a specific technique, or, you know, maybe you're looking for a generalist or an extreme specialist, or maybe it's like specific past experiences that you expect this person to have and so on. There's a huge, huge, huge list of, you know, things that we might be after. And once you know that, there's also like, how do you get, you know, the answers to those questions? So, you know, whether this person has these things that you expect them to have, right? It might be a take-home assignment or a lead code thing or a brain teaser. I've even seen IQ tests being sent during tech interviews, right? Like there's a lot of, lot of different things you can do. But I think before you sort of jump into this crazy sea and try to figure out what you'll come up with, it's sort of smart to take a step back and think of a framework, a system in which you can function, because that will just make your job a little bit easier. And the framework for us today is CASAO. I have no clue how to pronounce it, but that's, that's the best I could do at it. CASAO stands for knowledge, skills, abilities, and other things. When we talk about knowledge, it's things that the candidate knows, right? A certain technology, a certain language, whatever it is that you're using in your company, that's the knowledge you're looking for. The second type are skills. Skills are the things that this person is able to do, right? They might be, I don't know, they could be, they could be a ROS specialist and they know how to do certain in a certain way. Maybe they've built CLIs their whole lives and they know how to build CLIs perfectly, right? Like that's sort of the skills that you might be looking for. Abilities are very similar to skills, but these are things that are a little bit more difficult to teach, like the ability to reason about a complex system, the ability to break down tasks into smaller tasks and so on. These are things that you can, of course, like little by little, teach a human, etc. But usually people are sort of, they've grown up to have these things, right? So abilities are a little bit more difficult to find versus knowledge or skills. Knowledge and skills you can teach anyone versus abilities, which is very difficult to teach. And lastly, we have the other, and other tends to be, you know, maybe you're looking for a specific motivation, or maybe you want this person to live in a certain country, right? Like all these other external extra things that you might expect this person to have, that all falls under the other category. So again, for the AI Native Engineer that we're looking for, let's try to do this exercise a little bit to see how it works, right? So we know what the job description is. Let's think a little bit about the knowledge we hope this person to have, right? We would want probably this person to understand how LLMs function a little bit. They should know their strengths, their weaknesses and all that, right? They should also probably have a bit of an understanding of sort of AI risk, AI governance.

How would they prevent prompt injection or which data they might potentially leak or not, right? These are very concrete and sort of black and white skills that we know that this person should possess already on day one. Then we have the skills of such a role, where we would definitely expect this person to be able to decompose complicated tasks. Given that now coding is cheap, we are hoping that people operate on a slightly higher level. So they will be getting larger tasks. They will be getting end-to-end projects and whatnot, and they should be able to split it up into smaller tasks. They should be able to write their own skills, invent their own prompts or whatever. Like all these things, these are expected skills from the role.

We might expect definitely for the candidates to have certain abilities. Curiosity is one. In this industry, in this day and age where everything keeps changing, if you don't have that innate curiosity in you, you will be left behind. You should always be wondering, always be learning. Similarly, the learning agility. When things change, are they able to adapt? Are they able to update their tools, their way of working, all of this stuff? This is what we would be looking for in such a role. Under other, we might want this person to have intellectual humility, as in, when you're depending on a tool to help give you answers of a complex system, you should know that you don't know everything that this machine gives to you. You might do mistakes. You might misinterpret things yourself as well.

Now that we know the job description, now that we've sort of looked into all the knowledge, skills and abilities and other stuff that we have, we should have a fairly good idea on what we're after. What are we trying to find in this candidate? Next step is to actually go ahead and revamp our interview process. The first one is a very old technique, to have the candidate go deep. Pick something you've worked on, something you've spent quite some time on, and explain it to me and go deep into it. The second technique I really like is reading code.

It is to have the candidate go deep. The famous question where you ask them to pick something they've worked on, something they've spent quite some time on, and explain it in detail. This process evaluates the candidates' abilities to explain complex systems thoroughly, proving their knowledge and communication skills. Even in the era of advanced technologies, going deep remains a valuable assessment technique.

Reading code is another effective method. Despite the increasing abstraction layers, individuals should still possess the ability to understand and work with code. Reviewing code focuses on finding mistakes, assessing quality, and scalability. By observing how candidates analyze and explain code, important insights into their technical capabilities and mindset can be gained.

Writing RFCs is a novel approach to evaluating candidates' system design skills. By having engineers draft plans, consider options, and present tradeoffs, this technique assesses their critical thinking and decision-making abilities. Candidates are tasked with creating an RFC for a complex scenario, demonstrating their understanding of tradeoffs and system design principles in a practical manner.

But still, you would rather want someone to have that skill of dealing with ambiguity, understanding something that's super large that existed there, and try to dive in and figure out what to change and how to change, right? So a question like help your teammates understand what this large code base does is a great way of getting to the bottom of that. Just give them an open source project or give them a large service that you might have in your company. Have them take it home, you know, they could use whatever technique they want. They could run their LLMs on it et cetera. Then you bring them on site and you tell them, explain it to me. And hopefully, they'll show you what they've understood and hopefully, they'll be presenting diagrams and they'll be able to show you sequence things and really, really make sure that, you know, information gets over to you. That's one of the greatest ways of sort of figuring out people who have these abilities.

Reviewing code is one of those. It's almost very, almost identical to reading code but it would have slight nuance to it because you're again, looking into not only skills but also abilities in the sense when you're reviewing code, you're focused on finding mistakes, finding what could go wrong and so on, right? So it's a slightly different mindset than trying to understand what the code is doing. Therefore, hopefully, you will sort of suddenly see what this person cares about, right? Do they care about quality? Do they care about scalability? All these things will suddenly jump out to you if you ask them to review code and you get their review.

Writing RFC, this is a new one that I have been playing around with. I haven't had enough time and sort of data to play around with this idea but, you know, again, because writing code is the cheap side now, it's about designing the systems. How do you come up with these, you know, complex systems? What do you build? How do you build? When do you build it and so on? And RFCs is a great technique to sort of, you know, have engineers make a rough plan, an idea, look at the different options and present you tradeoffs of what could work and what could not work and so on.

I don't understand this bit but I also don't need to understand it because here is a diagram that explains to me what it does, et cetera, right? Then you're not only seeing whether they can actually use all these tools to figure out what the code does and where it could be going wrong but they should also be able to explain it to you in words that, you know, you can take and understand and adapt and change, let's say.

Writing RFC, this is a new one that I have been playing around with. I haven't had enough time and sort of data to play around with this idea but, you know, again, because writing code is the cheap side now, it's about designing the systems. How do you come up with these, you know, complex systems? What do you build? How do you build? When do you build it and so on? And RFCs is a great technique to sort of, you know, have engineers make a rough plan, an idea, look at the different options and present you tradeoffs of what could work and what could not work and so on.

So again, you know, give them a complex task, send them home, you know, have them write an RFC and come back to you and have that discussion around that RFC, right? Did they consider the tradeoffs? Maybe the LLM has but have they understood the tradeoffs, right? So and then you can have that conversation in person. Systems design is another one I've used a couple of times. I still doubt whether this is a useful one but I kind of see sort of extreme positive results as well as extreme negative results in some cases where again, you do a similar exercise, you give them a complex task, you know, send them home, they could use whatever technique and system and, you know, they come back to you with a system design. And once you have the design and once you have that person in front of you, then you start playing around with it.

Why did you make this decision versus that decision? What would happen if I remove this box? How would your system react to it, right? So all these questions allowing you to see whether first of all they understand what's being outputted but also what would happen, you know, in case things change in that system and so on. Those are a couple of techniques and there's way more which you can find on my blog.

But I also want to give a couple of tips to help you out getting started because sometimes, you know, a technique on its own doesn't really mean much if you don't really have the sort of right way of executing it, let's say. So a couple of tips out of the bag is one is, you know, a lot of times I see people trying to cram in many questions into our session, right? Like it's and it's really not about the quantity that you're there that you can get the most out of an interview. It's really about the quality.

It's better to ask just one great question like to go deep question, then to have 10 questions, you know, that is very scattered and you never get to go very deep with it, right? So go for quality and not for quantity. Second one is don't do things live. Like there's no point in having a person, you know, join a call and you're interviewing them and you expect them to open up their LLM and they're typing in something and you're both waiting five minutes for like this.

You only get maybe four at best eight hours of quality time with a candidate from all the way from the start to the end of your interview process. And if you're wasting 20 minutes of that, you're wasting a huge chunk of that time just waiting for stuff to happen. Do all of that async, right? Just let them do that stuff at home. Then they come over and then you can have that conversation on the output that was generated back at home, right? This is way better. Don't do live. Encourage LLM usage, right? Like there might be cases where people assume that they shouldn't be using. No, just tell them like you can use any tool you want, any diagramming tool, LLM, whatever it is that you want, you're free to use it because you ultimately want to see what they are capable of with all the tools that exist today out there, right?

A day at the races is still one I feel quite strongly about, although I don't think it kind of fits the sort of remote hybrid world that we're in today. But it's so eye opening to have an engineer come over to the office, maybe spend a day or a half day with you. You get to have lunch with this person. You get to have a bit of a chat. You might sort of have a pair program with another engineer you have so you can see how the person clicks in the team and so on. It's extremely valuable, but of course, it's more difficult, right? You're taking almost a day from this person's life. And not everyone can do that, right? We all have jobs. We all have children or whatever, but if you can pay them or somehow give back something to this person so that they don't feel like they've spent a whole day and at the end, maybe they get rejected at the end of the interview process, right?

Extremists is one where I feel they can make it or break it, right? If you have a startup, it's very clear what you want to do. Get an extremist and you'll go super fast, super far. Whereas if you have like a mid-size, large-size company, a lot of different engineers, seniors, juniors, whatever, when you get an extremist, they will be very, very difficult to sort of manage and sort of fit into What I mean with extremists is, you know, those people that are believers of technologies or methodologies or whatever. And you've all seen them, right? You've all seen that guy who comes in and says, Ross can fix this. And why? Like, why, Ross? You don't want those extremists. They can be very, very difficult to maintain.

The way I kind of figure these out during interviews is to present them a dilemma, right? I often have this sort of scalar interview technique where I tell them, let's imagine there's a scale where on one side you have an architect who's very slow but very meticulous. And on the other side, you have a hacker who's very fast, but maybe not that careful about their decisions. Where are you and why? And then you'll see those extremists jump in and say, I'm a hacker because I'm fast. And then you'll be like, OK, but why? You know, it's all about the tradeoffs, right? It's all about understanding what the advantage of one versus the other is. And when you see an extremist, I think you should be very careful to consider to hire them or not. Scorecards. Ultimately, in an interview, it's a human thing, right? Like we sit down, we decide, we make a decision.

Ultimately, in an interview, it's a human thing, right? Like we sit down, we decide, we make a decision. Therefore bias will always be present. Whatever you do, bias will be present. I think scorecards is one of the best ways you can sort of remove bias a little bit further than what it actually is. By scorecard, I mean, you know, you've probably all filled those, after the interview, what do you call it, forms, let's say. Which might say a text box, explain to me what you liked. Another text box, explain to me what you didn't like. Well, that's just like opening up the field for interpretation, opening up the field for bias, right?

Ideally, you want to have sliders. Ideally, you want to have check boxes where you just say, could this person answer this? Yes or no. Could this, what was the level of the experience of this person? You just drag a slider. Therefore, you do not allow anyone to introduce any bias through their pretty wording or, you know, their sort of fancy way of selling this engineer that they've really liked interviewing in the process. Testing your stages. One, you'd be surprised by the number of companies that don't do it, right? We're changing stages. We're trying out different techniques and so on. And if you don't measure, if you don't test what you're doing, you will never know what works and what doesn't work, right?

Measure the time it takes to go through these things. Measure the time it takes to, you know, on board this person. How long did they stay in the company? Was it as successful as you expected them to be before you hired them and so on? All of these numbers matter. Once you have those, then you will see what is working and then you can adapt the process according to your needs, right? Being a little bit pseudo-scientific in this domain does help a lot. Does go a long way. And 10x, right? Don't forget now, everything is 10x. There's 10x more code, 10x more reviews, 10x more bugs and so on. So don't keep your, you know, your assignments, your code review assignments or tech review assignments, whatever. Expect the 10x. Or when you're asking them to review code, give them 10x more code, right? Because that's what they're going to have to deal with in this day and age.

And if they're not able to do that, then they're probably not up to the new challenges of this new world, right? So to recap, write down the job description, get your SEO, prepare your questions, test it internally, run experiments, right? Try to measure and understand and eventually measure the success of your interview process. And to close, I want to give a couple of acknowledgments for the people who've given me this idea and who sort of sparred with me on it and so on that helped me improve. This is the book if you're interested in learning more about, you know, the sort of tacit information on engineering management, which I'm trying to make a little bit more explicit in the book, together with my co-author Maxim. Here you go. And I will also provide a couple of links, which I'll also, you know, you'll get to see once I share my slides after the talk. And if you have more questions, you know, come and find me in the thing. I'm an approachable person. I don't bite. So just, you know, just come and say hi and we'll have a discussion. Thank you. We do have a set of questions up here. So if you haven't put your question in, do head on over to Slido, put them in, vote for the ones you like, that sort of thing. We will start here. So first question, most of these techniques require quite a bit of time investment from the applicant. So how do you propose we reward their time and maybe even the tokens? That's actually a good one. I think tokens, you could definitely help them out with, you know, providing an API token that you may be temporarily creating given so. And I think that's doable. Compensation, I think, can come in various forms and shapes, right? So you could definitely compensate for the time, or you could sort of help them contribute to something. So I've seen a couple of times, for example, that the assignment had a well-known open source project that was given as a challenge. At the end of the day, you know, even if you didn't get accepted, you had a commit that went into, I don't know, Mozilla Firefox or something like that. So you still went away with something, right? So I can imagine there can be like various other forms and shapes that you can compensate that with. But I mean, it's at the end of the day, a give and take game, right? Like you're there to hopefully get hired, and you're there to hopefully get the best out of this person. So yeah, I don't know. Hope that helps. All right. Next question. It starts with a compliment. Great talk. When you say you've measured these techniques and interview questions, how have you gone through empirical evaluation of them? That's a good one. That's actually also nowadays way easier, by the way.

But I'm well known to measure everything I do every day, you know, like, whether it be my engineering work or my interviewing work. So I'll make sure to time everything, measure successes, follow back, and so on. Like a lot of data I have around recruitment. And that's also, by the way, helps your recruitment teams, because they tend to not have those skills. They tend to not measure these things that detailed, right? And once you have those, there's a lot of techniques you can use to sort of like visualize, get a funnel, see the time speed, see the, what do you call it, the return on the time you've invested on this person and so on. And LLMs are actually great. Once you have that data, you can literally ask an LLM to sort of help you out figure what to improve, where to improve and so on. So that's good.

How are you dealing with the strict restrictions imposed by the EU AI Act on the use of AI and the recruitment and candidate selection process? That's a good one. That's a good one. That's one, I think, where again, sort of giving them that homework to go home and use whatever they have is, I think, a bit of a better approach than to sort of open up this case where you're giving them your LLM and maybe you're giving them part of your code base, et cetera. Like I think you really have to be careful in what is it that assignment that you're asking them to read or review, et cetera, which is why I often prefer an open source project that I take and give to the candidate to review instead of going into this path of figuring out what to do, how to do, and so on. So just avoid it, I think, is the right answer.

How can junior engineers prepare themselves to these techniques that you've been talking about? I think, I mean, often I find myself, so when I devise these type of questions or techniques or whatever, I will build several examples of such questions, and I'll take different projects, different sizes, and so on, and I'll run through those myself just to get a bit of a sense of it because you don't, you know, you should dogfood what you're doing, right? Like you're not going to give something that you don't know how it's actually working. But then often I'll get a couple of people from the company as well, so I'll get junior engineers and I might give them the assignment as well, and I'll see how they work and function, right? And if they're really struggling with it, we could add a couple of, you know, help clues or whatever in the readme to help guide the junior engineers during the interview process, or maybe it just works, right? And especially nowadays, like I think I've seen more surprises come through juniors than, you know, seniors because you don't expect a lot from them, whereas they come in and with all the tooling and all the knowledge they have nowadays, they can really surprise you with what they come up with and so on. So, yeah.

How can junior engineers prepare themselves to these techniques that you've been talking about? I think, I mean, often I find myself, so when I devise these type of questions or techniques or whatever, I will build several examples of such questions, and I'll take different projects, different sizes, and so on, and I'll run through those myself just to get a bit of a sense of it because you don't, you know, you should dogfood what you're doing, right? Like you're not going to give something that you don't know how it's actually working. But then often I'll get a couple of people from the company as well, so I'll get junior engineers and I might give them the assignment as well, and I'll see how they work and function, right? And if they're really struggling with it, we could add a couple of, you know, help clues or whatever in the readme to help guide the junior engineers during the interview process, or maybe it just works, right? And especially nowadays, like I think I've seen more surprises come through juniors than, you know, seniors because you don't expect a lot from them, whereas they come in and with all the tooling and all the knowledge they have nowadays, they can really surprise you with what they come up with and so on. So, yeah.

Do you have any insights on candidate experience with your LLM world interview and techniques? Are candidates positive? We always ask them at the end of the cycle how they felt about it and so on, and mostly we get positive results. There are definitely people who get, I guess, surprised by it because not everyone does these things, and sometimes they're a little surprising, right? So you do get, when a candidate is often upset, when we rejected them, so on, they might suddenly turn to that interview that they had, and they might be like, yeah, you made me review code and that was, like, we get that. But more often than not, people are actually happier with this stuff because they feel free. They can use whatever they want, and it's roughly what is expected from them on a day-to-day basis anyway. So I haven't had a lot of bad feedback from it. Yeah, I'll chime in on that a little bit, too. We do an LLM focused coding interview. It's a live coding interview, but when we tell candidates about it, it's a bifurcation. Some candidates are like, why? But most are like, wow, because they're doing that, that's how they're coding, right? And so to see, it's what you were saying about the stages of grief, right? Seeing the acceptance of, oh yeah, this is what we're gonna do, so let's interview based on it. A lot of candidates love it. Next question we have is around bias. This is a juicy one. Is bias that bad? Where ultimately the question is, would the team like to work with that person? So how is the chemistry? Bias, I guess, is not per se bad. It's a very difficult question. It's a juicy question. What you ultimately don't want is to, like when you're recruiting, you're trying to grow a company, you're trying to hire more and more people, right? Like you're just growing and you don't want to dilute the company's skills and knowledge and whatnot, right? Like you don't want to hire more and more mediocre, therefore the company just becomes a more mediocre company over time. You'd rather hire people that are better than you, that can teach you and grow you and so on. And it's often not the case that you get these things when you let people have that feeling slip in that they, where they say, oh, I really like this person, I could work with him. You could work with him, that's fine, but ultimately is that person going to bring anything into this company? That's the question you're trying to answer, right? But that being said, of course, like, don't hire brilliant jerks, right? That's not the case I'm making, but bias more often than not will lead you towards mediocrity instead of excellence, I would say. Yeah, I think interviewing on both sides is a real skill. It is. And not everyone who is interviewing candidates knows what to look for. And so having that scorecard really can help guide them in terms of, oh, I thought this was just looking at outcome, but really I'm looking for how do they adjust to controversy or being pushed this way or what have you. Yeah. Um, next question.

How do you know the metrics of your hiring, dev retention, onboarding speed, are not related to external factors like toxic team relations or company politics, etc? During hiring? I think to interpret, if I'm interpreting correctly, so you were mentioning you kept metrics throughout of how people were retaining, and they're sort of wondering how do you attribute it to the hiring process as opposed to like what team they landed on or other factors? That's a good one, actually. That's one I haven't thought about. I try to, I mean, ideally, I guess in a company, you are, your goal is to try to bring every team to roughly the same level, right? Like you don't want one team to suck and the other one to be the best. And so hopefully you're working your ass off to make sure that this, you know, the knowledge spreads across the company, and everything's roughly even and so on. So I kind of actually with my metrics, I just assume that's the case. So I'm not going to look at where this person has landed and whether or not that has had an effect on it, etc. Might be interesting to consider, but I personally haven't considered that. So I don't know.

All right. I think we are just about at time. So thank you. Thank you. Another hand for Dunja. Thanks so much.