Why Leetcode is Dead and Pair Programming for Interviews is the Way Forward

♪♪ Hello everyone. I am super excited to be giving this talk at Tech Lead Conf. Thank you for taking the time to join us. So, today, we're going to be taking a look together at what's wrong with online coding tests, why leetcode is dead, and how including a pair programming interview can revolutionize your entire interview process.

So, I want to start off by giving an overview of what we're going to be talking about today. Firstly, very quickly, why does growth matter? And why should you focus on interviews? Two, what are you looking for in the ideal candidate? What are the key characteristics, traits, skillsets that you should be targeting? Knowing that, thirdly, what are the skillsets that online coding platforms like leetcode, like HackerRank are actually testing? Do these match up well to the skills that you need to be looking for in a candidate? And lastly, we'll go in-depth on the whole pair programming approach.

So, a little bit about myself and where I work at. My name is Muhammad. I am the head of mobile at a company called Theodo, based in the UK. Theodo is actually a global consultancy with over 600 people spread across the UK, France, US, and Morocco. We help clients build digital solutions to some tough strategic challenges. Things like self-service and retail journeys, to e-commerce platforms for second-hand clothing, or building Challenger banks from scratch. So, if you're interested in learning a little bit more about Theodo or just wanting to have a chat, please feel free to shoot me a message on Twitter. I'm always down to grab some coffee either in person or have a chat virtually.

On a personal note, the reason I really wanted to give this talk was actually because of my own journey in joining Theodo. So, my decision to join the company was largely influenced by my own pair programming interview. It was a real enjoyable experience that pushed me over the line. It got me to say yes and actually join Theodo and the team that we have. So, the impact of it is very much felt in my own personal career. And I'm now one of the people who conducts the pair programming interviews. So, we've come kind of full circle. And it's really something that I enjoy. So, I'm very excited to share it with you all today. Let's jump into it.

What is the importance of hiring and growth? Not gonna spend too much time on this, but really if you look at it, you can split it up into three categories. One is team quality. This might seem very, very simple. But a high quality of hiring will lead to higher quality of teams. So, without having a well thought out and good quality interview process, you risk having false positives. So, maybe hiring the wrong candidates, someone who's not a fit to the team or doesn't have the skill sets that you were requiring.

Or you'll have false negatives. And that's not hiring the right candidates. Sometimes having a system that isn't effective will make you lose out on some really good candidates that could have potentially been a very valuable asset to your team and to your company.

Secondly, company reputation. I remember when I was a graduate and I was looking for jobs, one of the things that came up regularly amongst my peers was certain companies are difficult to apply to. Why take the time? They had a bit of a reputation for being a little bit difficult and not enjoyable processes to go through. And hence it was kind of known amongst the student community, you probably don't want to spend your time applying for them or it's really difficult. Hiring process can have really positive and really negative impacts for the reputation of the company. And throughout that whole interview process and throughout hiring, you can utilize it as an opportunity to get to know more engineers, learn about people's experiences, share your own experiences. And it's a really good opportunity to just get to know more people in that whole interview and hiring stage. So it's good for reputation and also a good opportunity for you to get to know more people.

And lastly, cost. So if you're starting to look at being an EM or a tech lead, one of the important things that you're going to have to be grappling with is cost. So that's anything from the cost of the pay for the candidate. More short term, there's the cost of a deaf team that are going to be interviewing the candidate. The cost of their time that they're spending to go through these interviews with your candidate, see if they're suitable or not. And then in the worst case scenario, if the candidate wasn't suitable and you needed to rehire after some point, the cost of needing to rehire obviously can be quite heavy as well. So with all of that in mind, we can have an overview of why hiring and growth is important and why we need to focus on this. What should we look for in a candidate? I've got some of the sun rays and the lone figure to give this kind of grandiose, holy vibe. So this is a perfect candidate. I'm sure at some point, we've all heard of this concept of the 10X developer. It's the developer who can deliver at 10 times the speed with one tenth of the lines of code. I'm glad to say that after extensive research, I have come to the conclusion that the 10X developer is a lie. It's a total facade. It doesn't exist. And throughout my research about the 10X developer, I actually landed on this really cool quote that I liked, which was, code is read four times more than it's written. So if you write easy to reason about code, then it pays off dramatically in the future. I think it really encompasses a few key factors about what is important when you are looking to hire a candidate. You want someone who is able to write good code, code that will be maintainable and understandable by others, by more senior members in the team, more junior members in the team, and code that is ultimately understandable and reasonable by others in that team. And that is what will make that 10X payoff in the long term, because that code is going to be reviewed and looked at so many times in the future and in the lifecycle of a code base.

Taking this concept of the 10X developer, more realistically, what is the job of a software developer? And I think it's very multifaceted. It transcends just coding. So a good software developer needs to be able to identify problems and break down problems from business requirements and business needs. They should be able to design a solution that solves the problem while working in a cross-functional team and really write that efficient solution within a good span of time. So they need to also be a bit pragmatic with the approach that they take. And if you're hiring for a higher level, so maybe a tech lead or an EM, there comes a coaching and a managerial aspect as well. So how well does the candidate teach more junior developers on the team, train them up, coach them? Can the candidate manage a project pipeline effectively with the PM? Can they keep code quality up across a multitude of projects? These are all factors that you're going to have to balance. So when you take a look at it on a grander scale, bigger picture, there are a number of skill sets that you need to actually look at when you're hiring a candidate.

So thinking about that, let's actually look at online coding tests. What skill sets are online coding tests assessing? I'm going to be a little bit bold here and say that, primarily, online coding tests are effectively only measuring a candidate's knowledge and algorithms and data structures. So I'm sure you've gone through your fair share of online coding tests. They are mostly following the same patterns. Maybe you have a linked list that you have to reverse, or you've got a binary search tree that you've got to traverse and find a certain note. Most of these problems are very similar, actually, in nature. And you really need to get familiar with the classic COMSCI algorithms and data structures course.

So it's very academic in nature, and you will have encountered them if you had a classic CS background, or you will see them at some stage. But it involves a lot of the similar patterns that keep coming up over and over again. And on top of that, online coding tests look at your ability to do them under immense pressure with a short span of time. So how quickly can you actually implement those algorithms and data structures to match the problem that you're given? So really, if you've studied, memorize the algorithms, data structures, and you understand the pattern behind the questions, you start to see them, and you will after a bit of practice, you'll practically ace these. If you don't believe me, there's literally a book for it. So this is staple in every graduate's shopping list. Cracking The Coding Interview has 189 programming questions and answers that are typical in these coding tests. It's a book with several hundred pages, and you go through it one by one, go through the exercises. It very much feels like you're studying for the SATs or some sort of standardized test, and you'll quickly start to sense these patterns. So very, very formulaic, not incredibly effective to assess the things that we talked about when we were looking at the perfect candidate. And the question that we need to think about is, how much of this is actually applicable to a day-to-day software development role? How often are you thinking about, how can I reverse a linked list? Or how often do you actually grapple with these problems where you need to come up with the most efficient solution possibly, the most efficient solution that is possible? It's not a common day-to-day occurrence. It's certainly not something that requires the majority of attention when we're looking to hire a candidate. So there's this imbalance. It's a small part of the role, yet it gets a massive amount of say in which candidates make it through and which ones fail. So there is an imbalance there between the hiring and the actual role.

Let's jump into pair programming. So to explain pair programming, I want to start with what pair programming isn't. And pair programming is not this. It's not a candidate going up to a whiteboard, starting to scribble down some notes, and you, as the interviewer, are sitting there very immensely and intensely looking at them. That is not pair programming. For a good pair programming exercise, you need a realistic problem and you also need a realistic coding environment. So let's start with realistic problem and we'll go through an example of this later on. Not going to focus too much on this right now, but you want to try to find a real-world-like problem.

And within that problem, you can try to encapsulate and encompass some logic, some knowledge that you would like to test the candidate for. It's really important to make the problem understandable and relatable. Something that they might actually code in a real day to day job. That includes some sort of complex logic as well as a part of it.

Now when we talk about a realistic coding environment, that means no whiteboards, no pen and paper, use a real code editor. There's so many great actual coding environments that will enable you to collaboratively work on code at the same time. And this is going to be the closest thing to the day to day role. This is going to be what they are going to be using every single day when they come into the job. So examples of this, CodeSandbox is a popular one. You can use VS codes, LiveShare extension as well, and collaboratively work on a single file or multiple files at the same time and you have one environment and you share it, it's a great tool to use. And it really makes sure that you give the candidate the closest thing to an actual coding environment that they would be using in their day to day job.

And when we talk about a realistic coding environment, it's not just the tools but it's also the style of interaction that they have. So a question to pose is, in the day-to-day role, what happens when a developer encounters a difficult problem? And I want to let this sit for a second. When a developer reaches a point in their application and they're thinking about it, they're reasoning about it, and they've reached a slightly difficult problem, what they'll do is they'll usually consult with the team, take a step back, talk to others, get their opinion and try to come up with an optimal solution together. That's where that whiteboard comes into play. With that in mind, it's so, so important that you're actually working together as part of this pair programming interview. You should be coding and speaking as much as they are. You really want it to be a pair programming session, and this will let you get a sense of what it is to incorporate them into your team, what is it like to work with them? If they were to join your team tomorrow, which is a great thing to know, as an interviewer, but it also has the flip side, which gives them the sense of how it is to work within your teams. So that, for a candidate, is invaluable. They really get a tester or a demo of what it would be like to join your company and to join your team.

So without further ado, let's jump into a demo, and hopefully this will all start to fall into place. So the example that we have here is a rather simple example. It's a tennis-scoring kata. And you can find a compilation of these real-world like problems across different GitHub repos. This one comes from a kata catalog that someone has spent a lot of time compiling, so props to them. And it's got a number of different problems with different languages for sort of templates for you to start testing people with. Let's go through the background of this game, and then we'll go through a short live demo. It won't be the full extent of an interview. And obviously there's different problems with different lengths, different difficulties. So you can really go in and choose whatever you think is more applicable to your specific interview process.

But in this case, just for this demo, we've chosen something that's a bit simple so that we can go through a little bit of it. And hopefully, it'll be easy to follow along with. So we're going to be implementing the scoring system for tennis. In a game of tennis, both players begin with a score of zero. And each successful rounds that they play, the points get added on in a sequence of zero, 15, 30, and then 40. And then once you're at that 40 stage, you will win the game if you score again, unless both players are at a score of 40, which is called a deuce. So that means that you're in some sort of deadlock. And the way that you win when you're in a deuce is that you have to score twice subsequently back to back. So if you were able to break the deadlock and score twice in a row, then you will end up winning the game. So basic tennis rules.

What we're trying to do is we're going to try to incorporate this and build the logic around this without focusing on UI, without focusing on data validation, just the bare logic of this. And we're going to try to see if we can do it in a short time span for this demo. So let's try to get into it. Before we do, I want you to focus on things like how much am I interacting with the candidate in the interview. When they get stuck, do they ask me for help? If they don't understand some sort of syntax, are they comfortable enough to ask me as the interviewer, how do I set a variable in this language? Do they take a step back before they try to take an approach and look at all the different paths, reason about it, think about the pros and cons? Or did they just go into the first route that they see? How receptive are they to feedback, if you give them hints? Do they start taking that and looking into that? Or do they just kind of ignore it and move on with what they were doing? And lastly, are they doing over engineered solutions or are they going for simple solutions, trying to be pragmatic with it? These are all kind of the things that you want to look at. And they're the things that will give you a good indication of how this person is in terms of their software development skills and just more generally how it is to work with them.

So we've got my good friend, Sadik here with us, who has graciously agreed to do this demo. Thank you Sadik for your time with us. No worries, a pleasure. Awesome, so we've got a VS Code live share environment set up. There's a tennis class, there's some tests, and we're going to be following the TDD programming style, we're going to create some more tests, get them passing, and then we're going to do a round of refactoring. We'll go through one round for this demo. Obviously you'd be doing a full fledge test if you were interviewing someone. So Sadik, should we move on to the next test? So we've got tests for handling a new game, a single pointer game, and a level game. What do you think we should be testing next? I guess this is handling pretty basic scoring. We could handle a win next. Cool. Let's do a test case for that. Cool. Fire in.

And to win, they'll be scoring four times, just to get all the way to the end past a 40. Past 40. Okay, cool. And what are we expecting the actual get score string results to be? Yep. Let's change that to… So this is the second player I've given a win to. So there are two wins. Let's do it. And I guess, yeah, quickest way would be returning it here somehow. Okay. So a win… Well, we need to represent it actually in code. So there's a couple of ways we could go about it here. One would be in scoring, like past 40, I know that's the highest score you can get. We could have a value though above it just to represent, since we're going through each index here, we could have a last element represent a win or we can have some sort of state in the class. That's good.

So taking a step back here for the audience, it's really good… Sadiq is taking a step back, looking at the different routes that he can take with this. It's a really good sign, he's speaking out loud. I can fully understand what he's thinking of. So I'm going to try to contribute a little bit to this interview now. So, yeah, that sounds good to me. I guess let's think about the pros and cons of each approach. If we were to use a flag, we need to do checks in multiple places within the score functionality. And then have a few if-statements to not increment the score as we're doing now. That could have some complexity. The other side is we represent a fake value in scoring, like you said. But then you're not sticking to the actual logic. It's not tennis scores. So I guess with those trade-offs in mind, what do you think we should do? Well, to get the quickest solution, just the quickest pass for this test, it probably is easier to go for a value. And then if we wanted to come back, we could obviously refactor it. Yeah, we could come back. So why don't you let me know what I should type, and I'll do the keyboard.

So let's choose a value, just anything over 40. Let's do 60 for... Oh, God. I don't like that. Okay, cool. We've got a tennis-efficient auto here, so, okay. We've added a value of 60 to this array, Nice. And then the GetScoreString, I guess we should check the score. Okay. If it equals 60 for each player, so if this dot, yep, first player score, second player score. If it's 60, then we want to return a string that says something like Player1Wins, right, following the pattern that you've set there. Otherwise we check, see, okay, has a second player won. And then in that case, we would also return Player2Wins. All the typos. Okay, cool.

So we've taken a new test, and originally it was failing. We made it pass. We've now reached a refactoring stage. So, Sadiq, is there anything either in the tennis class or the test that we've written that you would refactor or clean up? Yes, so I guess the first bit of code I can see that's repeating is inside the score function, where we're calculating the index of the next score. These are long lines. Maybe we could take it up, put it into function. Let's do it, go for it. Sure. So, any name suggestions? Well, so if we're semantically naming this, it's getting the next score's index on line 14 and 18. So, why don't we name it that? Sure, and we'll take in the player. We take the player? Or we can take the current score, whichever one. Sure, so we'll just take the score. Cool. And from that, we're putting this bit here. So, while Sadiq is getting that working, this is good for naming, for things that he wanted suggestions on, felt comfortable asking me, that's a good sign.

It shows that he's comfortable asking for help or asking for hints whenever he needs it. So, that's a good sign.

Cool, so, that function's there. We should probably return, yep, and then we can just call that instead of doing that long-winded scoring.find index on every single line.

So, here's the test. Cool. And then we can just copy this, do it again for the second player. Sounds good. Nice.

So, we've taken that. We've refactored it, let's save that and see if the test is still passing. Amazing.

One other thing as well, I think, on the test side, we have a fair bit of repetition here, like we're rolling, we're scoring multiple times for each one of them. We could potentially put it in a loop, especially for this one where it's happening four times. What do you think, Sadik?

Yeah, let's make a loop when it's four or more. I think it's getting a bit bulky. Let's do that. So, we'll create a loop here, I is less than four, we'll increment it, and then we'll iterate here and get rid of that. Cool.

So, we've got our tests passing. Again, we refactored both of our files. We're in a good state, we've done one loop of the red green refactor loop in TDD, you'd build up on this in a real exercise. But we're going to stop here just to keep the example short and simple.

It's a really interesting problem, I'd recommend you go and give this, this tennis problem a try, it gets really interesting once you have deuces and you're trying to deal with the advantages and whatnot.

Thank you so much, Sadiq, for taking the time with us.

No worries. Thanks.

So, let's wrap up. I want to leave you with some final thoughts. These are things that I typically do before and after the interviews, I think they really leave a good impression with the candidate.

Firstly, don't really jump straight in immediately. You don't want to have just a plain introduction and then running into the exercise. I'd recommend taking a little bit of time, try to get to know the candidate. There's so much value in getting to understand their story, their ambitions, what they're like to chat with, and don't discount how great that can be in easing the candidate and hopefully giving you a better environment to conduct the programming interview.

Secondly, take a little bit of time at the end, look at the solution that you've come up with together and see if you can find any improvements and see if you can get the candidate to critique their own work. It is a really good sign when a candidate is willing to criticize or point out bits and bobs which are missing in their solution. So I'd highly recommend taking a little bit of time at the end, a couple of minutes to just talk over it.

And lastly, ask for their feedback. It gives them such a good opening to give you improvement points, but also to talk about how their experience was going through. And speaking of experience, I want to just end on this note. We had a candidate a little while back apply to us. Unfortunately, they didn't get in, but I got this email a few days afterwards. And I just want to leave that up there for a second. It's messages like this that make this pair programming interview so cool in my eyes. It's such a nice feeling when you as an interviewer feel like you haven't wasted an hour of an interviewee's time, and hearing back from them to say that they feel like they've actually learned something or they had an enjoyable experience is something that really has made all of us worthwhile. And I hope that in the interviews that I do conduct in the future, I can leave a similar experience with others. Thank you so much for listening.

Yeah, it was like almost a battle till a few minutes ago, like few seconds ago, I checked it was 46 and 46. Now, the no option when the head with 56%. How do you feel about this?

To start with, I'm quite happy that most people have abandoned coding tests. It's cool to see and I hope that number just increases over time. The yeses are still I think about a third, so 32, 33%. I'm keen to see what people use it for as well. So the people that still use coding tests, like, what stage of the interview is it used for? Is it a fundamental part or is it part of a screening filter? But it's still a substantial amount, right? One out of every three jobs that you're applying to will probably have a coding test, which is, there's still a lot of time that's going to be spent on that.

Yeah, yeah. Time that's nowadays could have been way better invested in more practical, more like stuff. Because the energy in a higher process, and you mentioned in the talk, how long the cost is for interview for the higher process. Yeah. By the way, the other day was very trending, the meeting cost, you know, on Google, fake calendar and the cost of people. We should do that after hiring process at the end. How many people from our engineer side took the interview and how much that cost us? Yeah, yeah. Especially being in a, like, slight consultancy background as well, you know, it's a very like, it's very much a key factor there. So, definitely think about the cost of each individual person's time as well.

I'll start with a quick question. Someone was asking what was the coding exercise called again, tennis scoring kata or something in your talk? Yeah, so these type of problems are generally called katas. There's an entire GitHub repository with, you know, dozens of examples of them, you can find one that you suit you and your style. This specific one was tennis scoring kata. So if you search it up, you should find a GitHub repo to it. Good. Okay, that's being settled. The second question we got was about how you found the Pair Programming Method to scale with the candidates experience?

Yeah, that's a really interesting one as well. So, if you're hiring, let's say at a graduate level, what you're really looking for isn't something like experience with syntax or a lot of the things that you would expect with experience to come. What you're more assessing is that communication, that's the fundamental thing that you want, and how quickly they can pick up concepts, how much they're willing to learn, because that's that stage of the career. And that's what they're trying to pick up. If you're assessing, and doing a programming interview with someone who's wanting to come in at a tech lead level or a senior engineer level, then you're going to have to be looking at other things. The communication is important, but it's also how do they explain these topics to you? And you have to really be seeing if they can fit into that leadership mentor role. So maybe as the interviewer, you take a step back and you give them more remit to drive the interview.

And from that, you can have a sense, if you were a junior developer, working with the senior developer or with this tech lead, what would be the style of working with them be? So it's very tweakable. You can also choose more difficult problems if you're trying to hire a senior developer. There's so many of these CADAs with different requirements and different complexities. So it's very tuneable, I'd say.

Yeah. And as you mentioned, the communication, I've heard that at least some companies do these like a week trial or even more or less the person will be integrated in the team and literally has a job because that's the communication part you see. Asking question, debugging in a way, you know, even the process in the new company.

Yeah. The second question we got in was about the senior candidate, and it's like, I found with more senior candidates, the skill we're interested in may work on larger scales and not always lend themselves to looking at codes together. Examples will be architecture and organizational for green projects. What do you think, what would this format with staff class role feel for you?

Sure. So I've actually, I can give you an idea of how we handle this. So a part of, we have this pair of programming interview, we think it's still fundamental to hiring a tech lead or like an engineering manager. But on top of that, maybe we'll add an architectural design interview as well into it. And the idea is, well, we need to also assess how can they make architectural choices. How will they create systems and integrate them with different parts of an application? And I think you can add that on. But this pair of programming interview is still quite important, especially because I think we've learned throughout the day today is like a tech lead and an engineering manager's remit oftentimes is interacting with people, is communicating, is empathizing with the team, teaching them stuff. So I think it's still a fundamental part of it. If you wanted to tweak the pair of programming interview, one suggestion could be trying to take a step back and before the start of the interview, asking them overall about a more complex problem. Let's say you can have a problem, like how would you handle login in general and see how they can explain those more fundamental concepts as well. So that's another step that you can add before you actually start the pair of programming interview. Yeah, and you can tweak it. As you mentioned, it's very easy to tweak it to add any other levels or any other steps that you consider good for the team and even more, make it specific to each role, each team, and the level at the end. Yeah. Oh, that was really insightful, and I learned a lot. Again, I will go back and see how much time I did with test and the pair programming. We do that, how we can tweak it more to get the perfect fit, especially nowadays that, yeah, tech scene is moving a lot. Things change so fast, and we definitely want to have the perfect fit for our own candidates. Thanks so much for the insights today, Charitas.