Improving Developer Happiness with AI

I'm really excited to be here today. My name is Senna, and I'm a solutions engineer at GitHub, and I'm excited to talk about this because I know it's a hot topic on everyone's mind. I know that I really could have used some of these tools when I was first starting out in development. I remember my first interaction with React actually was in my first job, one of my first projects had to do with refactoring this really massive home-grown JavaScript library of a project into React. And my team and I were going to Ryan France's React trainings in a Yahoo building in San Jose. So it's pretty funny that seven to eight years later, I'm talking at React Summit in Amsterdam. So now I live here. But yeah.

I have a lot of things that I wanted to get through today. So a quick agenda. I'll just do an introduction to some of the different types of tools that I'm seeing people use when I talk to customers that use GitHub. GitHub Copilot is the one that I'm most familiar with through my work there. And also want to talk about the research we've done behind Copilot. And what does it even mean to be productive and what types of things are we solving here.

And then I'll go into some benefits and some challenges of using these tools. And I'll sprinkle in a few demos of some new Copilot features that we're working on. And then at the end, we'll have some time for Q&A.

So really quickly... What are AI paired programming tools? If you search AI paired programming in an IDE, like VS Code, you'll see a lot of these options come up. And just really curious, show of hands, how many of you are using one of these like AI paired programming tools at the top? Okay, cool. What about GitHub Copilot? Okay, cool. So a lot of y'all are familiar. I talk to people about this all the time. I'm excited about it. I know a lot of other people are as well. But there's also these nice chatGPT style experiences that we're seeing. Where you can have more of a conversation. Do some more discovery. Maybe push out some prototypes. So there are a lot of cool technologies by both Microsoft and Google and Open AI. So lots of tools in this space. And every day I feel like there's something new.

So yeah, Copilot is the one that I'm most familiar with. It's a really helpful auto-completion tool. So what you do is you prompt it with context and the context is basically where your cursor is in the file that you're working on as well as what other tabs you have open. So it's able to synthesize the best possible suggestion for you. Yeah. Based on what you're doing and what you're trying to accomplish.

And this works if you're using one of our supported IDEs at the bottom left. And, yeah, the title of my talk is about improving developer happiness with AI. So wanted to talk about some of the research that we've done that kind of like proved some of the hypotheses that we've had. And we've done a lot of research both to inform how we build the product as well as just like understand how people are using it.

And this research paper is pretty cool, because I believe it's the I think it's one of the only academic research papers about this type of AI paired programming tool. So one of our first hypotheses is that it is helping people feel more productive. And early user reports have confirmed that. But we wanted to understand like testing it out and seeing how it's actually helping people. So our research team, they interviewed around 2,000 US based developers. This is a combination of professional developers, students and teachers as well, and some people that are kind of more hobbyists. But we wanted to look at productivity more holistically. And so we use something called a space framework to understand like which aspects of productivity we want to investigate. So I'll talk about that next. But this study included a combination of like qualitative data. So how people perceive they're being more productive. As well as quantitative data around like how fast are people doing things? How many suggestions are they accepting versus suggestions they're ignoring. So there's a lot of interesting studies here. And we wanted to assess the effects in the everyday development scenarios. So there's some interesting findings. Like a lot of people were accepting things more often after the work day or over the weekends. And we're not really sure why. But it's just one of the reasons why you should check out the academic paper. And we're going to be publishing more findings about this as well. But really quickly, what is SPACE? So this is an acronym for a framework about like several different aspects of behavior. We wanted to focus on satisfaction. So the S and the E. Satisfaction and well-being as well as efficiency and flow. Both of these dimensions are pretty critical, we think. Sometimes they're overlooked and sometimes people focus on singular metrics like how many commits people are making. I've been in that situation before.

So this is kind of how we informed our survey. That we surveyed developers with. Which brings me to what does it even mean to be productive? So we looked into this even before the co-pilot study. I think back in 2021, we wanted to understand like how are people, like what do people think being productive means to them? And what we found is that it's kind of similar to having a good day. So we call this our good day project. I'm linking to another blog post about this. But what people said is that they want to be able to stay focused on a task at hand. Feel like they're making meaningful progress and feeling better at the end of the day's work that they've accomplished something that they set out to do. And something that we heard is that when you break your state of flow throughout the day, then that can be really detrimental to your own sense of productivity and feeling like you've actually finished what you're intending. So a lot of this informs staying in the flow and feeling like you're satisfied with what you're able to complete. So these are the things that we set out to look into.

And back to the actual copilot study. What developers told us is that the majority of people, when they were using copilot, they said that they felt less frustrated when coding, they spent less time searching, so where you would otherwise probably go out to Google or read through some comments in an external site, having all of the suggestions and answers within your IDE is really helpful in kind of keeping you in the same flow.

And then, yeah, so they felt like they were able to, yeah, stay in the same flow, spend less time searching externally. And what was great to hear too is that people spent less mental effort on repetitive tasks that copilot helped them do. The majority also said that they felt more fulfilled with their job while they were using it and they're able to just basically spend more time on solving more interesting problems, more satisfying work. This was good ground for us to understand that it's having a positive effect on developer happiness.

And then one of the earliest hypotheses that we had was that people can complete tasks faster, especially repetitive tasks. And there were also some experiments we did where people used copilot to complete a task and compared that against people that didn't. So, yeah. Now that I've told you about some of our studies that were done on the autocompletion aspect of copilot, I want to come back to this slide, because there's something new we're working on called GitHub copilot chat, and that's going to bridge the gap between autocompletion within the IDE and having more of a conversation around the code that you have, like, the files you have open, understand, like, what feedback it has, and you can give it feedback and further prompt it to kind of help you progress.

And these types of discussions have many use cases. So I'll just go through a few of them. But, yeah, it really helps you with some of the more repetitive boilerplate stuff, common tasks, like maybe writing a SQL query or even, like, writing a regular expression for something. But you're seeing that it's, yeah, it's helping boost productivity and at the end of the day, saving you time to work on solving more interesting problems. So the things that you actually want to be doing. And it's really useful for kind of working in unfamiliar domains as well. You can really do more discovery and, like, figure out how to make a prototype of something super quickly.

So I want to show you an example of using copilot chat that I did a demo recording of. Also, are people using copilot chat? Another show of hands. Is this something that people have access to? Okay. Not a ton. But what I did is when I was building out this demo, I went to� this is very, like, but I went to the tulip festival at Kuckenhof a few weeks ago. And I took a lot of photos of tulips. I went to the fields and I was really trying to figure out, like, how can I make a really tulipy app? So I asked chat TBT and it helped me decide what I wanted to do with it. So I started with a blank slate and I'm going to hit play on this and try to do full screen so you can see it better. But on the bottom left here is copilot chat. It knows about the files I have open. And I just started with, like, using the Create React app. I just started with a blank slate. And what I did is I said, hey, I have these images of tulips in the public images directory.

Can you make a simple gallery for me? And what it did is it gave me a nice starting point. It was like here's your array of images that you can define and here's how you can loop over that and basically add image tags for each tulip image that you want to use. So this was a nice starting point. I just went in and pasted that in.

And what it gave me is it gave me some really zoomed in versions. This is a nice photo of a tulip called Tequila Sunrise in the Flower Show. But what I needed was it to be a little more scaled out. So I was like hey, these images are really large, can you scale these back a little bit? So you can actually give it feedback and tell it to iterate on the suggestions that it's making you. So the next thing that I did is, I asked it to fix that for me and it told me, hey, you can do this with a height and a width attribute for the images. And it gave me a suggestion of starting with 300 pixels. So I did that. And then, okay, so it looks a little bit better. But I wanted to figure out what libraries I could use to basically make it look a little bit more aesthetically pleasing. So throughout this process I got a lot of suggestions for different react libraries. And the one that I found easiest to use was react components, sorry, styled components. So it actually just suggested straight away when I asked it what I could use to make it prettier. It suggested I use styled components, and what that did is it just suggested yeah, like, here's the types of components it will give you that are ready for you to use. You can use the image gallery container. You can use the image component. It's already styled for you. It gives me the CSS that I need. So once I went ahead and, like, copied that suggestion in, I refreshed my app. I'm just going to go through this a little quicker for the sake of time. But it ended up giving me more of what I was looking for. Now I have this, like, kind of nicely spread out gallery of boxes for all of my for all my tulips.

So the next thing I wanted to do is learn about other components that I could use. So I wanted to make a modal. One of the things I did was I asked it about other aesthetically pleasing components, like, how can I add headers, how can I add some more landmarks? And then I asked it about modals, like, do styled components have modal support, and it said yes, it does. Here's how you can create a new file and define your modal component and here's how you can basically reference it in another boilerplate example of an image gallery. So I went ahead and worked with that suggestion.

Something I noticed, too, is that it suggested me a very generic example. Here's how you import the modal. It didn't really apply to my file, so I wanted to take that opportunity to ask it something more specific. So I was like, can you... Let me fast-forward a little bit. But I asked it, can you actually help me add a modal to my particular image gallery? And then I decided to put, like, some random facts inside the modal, so I was more specific saying, hey, like, each modal should use a, like, this array of random tulip facts that it generated for me. I just wanted to have a paragraph of a tulip fact within each modal. And, yeah, this just shows you again, like, the more specific you can prompt it with something, it can actually generate something really easy for you to copy and paste in there. So I'm going to fast-forward through this a little bit as well. Because I'm just basically, like, copying and pasting different things in here. It was able to handle all of the, like, on click and, like, closed states for the modal. It was able to pass through all the props and, like, the actual facts from this facts array that it helps me with. So if I fast-forward just a little bit, because I think I have a minute or so left. Yeah. It was able to just kind of put something in there for me. And I tried this a few times, by the way, and it took me about 30 minutes from start to finish. So I was pretty excited about that, since I hadn't really done anything in React in a long time. But now I really want to start my own photo blog and my own website that I've been meaning to do for a long time. So yeah. This is pretty exciting for me. And I want to go through some other use cases, too. We have seen a lot of interesting patterns come up after this has been released. So it's also helping people write tests. That's another test. That's another case where it can help suggest to you what type of framework. Like Jest, for example, would be a good testing framework. It can give you the test cases that you might want to use for particular functions. It's really helpful for repractoring code. So you can select full pieces of text, tell it to refactor it in a certain way. Or translate something to a different language.

Debugging was pretty useful as well. It knows the file you have open. And if you tell it, hey, this situation is generating this type of error, it can help you get beyond that. And documentation is also something that I found it helpful doing as well. You can tell it to write some docs for all of the code that you have in a file, as well as, like, explain different aspects of the code. If you highlight something and say, hey, can you explain this for me? It can give you step by step in English what something is doing. And this can help you understand how it's really thinking about the code.

And then there's other implications for this too. So we're working on kind of putting this stuff and adding these little helpful parts of Copilot in Copilot X. It can generate a summary and a pull request of everything that you're doing. So that's what I was able to get it to do using Copilot for PRs.

But also wanted to cover really quickly, like, where just some things you should be aware and the TLDR of this list of things where these tools might not be so great, is that you just need to be skeptical of the responses and verify the suggestions that it's giving you. It's good for getting through kind of common tasks, but if you're trying to do something more complex or get it to generate an entire function for you, it's not going to be 100%. It's meant to help you automate the parts before you get to that point and help you have some sort of starting point there.

Dealing with tricky APIs, it might not give you, once again, 100% what you're looking for. Something I noticed with the chat features, whether it's copilot chat or chat GPT or something else, it can give you outdated suggestions based on libraries. I actually wanted to do this demo with GitHub primer, which is our design system with React components, but it kept giving me deprecated components so I went another direction and ended up using styled components so that's just something to be aware of as well. And these, they're never going to replace your code quality and security standards. You should always still have that scanning happening. And it's kind of like pair programming with another person, you just need to make sure it fits in and makes sense with your project.

And one thing to note too is that this is really good for getting started in a new language It's really great for junior developers, but if you don't understand the code that you're writing and you don't understand what you're trying to build, then you might not be able to like verify the responses as well. So it's just kind of like a trade-off and something to be aware of. And this leads me to my conclusion. A lot of people ask me if this is going to replace people's jobs. As a human in this day and age, I'm a little cautious with AI and the way I think about it, but I do think these are just going to be tools for you that will probably at some point just be a standard part of a developer's toolchain. They're meant to help you do things faster and help you do things more with your time, and they're not writing complete software. They're really meant to help you out in what you're doing. I, for one, really like them. I don't know how I did things without CoPilot now that I've been using it for a year now. I can just—I'm thinking back on all the time that I could have saved anxiously looking through Stack Overflow comments or just searching how to do something on Google. So yeah, I'm really hoping that I'll get started on my little photo blog and my personal site finally. But, yeah, that leads me to the end of my talk. So wanted to leave it open for Q&A if there's any questions. Thank you for your time. That was great. We have a ton of questions, so I'm gonna, instead of reading out individual ones, I've sensed a couple of themes. As with many AI topics, people are still a little bit skeptical and or need more clarification.

A lot of people have mentioned either security concerns or IP concerns and how you would convince companies that have banned Copilot for those types of things. Yes, so I can speak about the security concerns first. Something we've added for folks, especially there are two types of Copilot you can use. There's like for individuals and then there's Copilot for businesses. So for individuals, there's something called data sharing where the prompts that you prompt the service with, which is a file you have open or, again, the other tabs you have open can be used to kind of help improve the product. So I think it's off by default. But if you turn that off, none of those prompts are retained. And then for businesses, which comes into companies that we talk to that want to use it, none of that, there's no data sharing mode available. So you can be assured that nothing is being used to train any model, for businesses, at least. So that's kind of the main, that's the first thing that we usually talk about.

That is good to know. Thank you. There's also a bundle of questions about copilot for JetBrains when, and copilot for him, and copilot for other IDs, and that kind of thing. Yeah. So you can use the kind of the normal like autocompletion copilot with JetBrains now. But in terms of copilot X and that vision of features, in technical preview it's just in VS Code. I don't know when we can ship it to the other IDEs that we support, like JetBrains. But I can assume that before it hits general availability, we'll have wider support for IDEs. But yeah, I don't know when the chat and the X features will make it in there. Fair enough. Yeah.

A couple people had questions about the open tabs providing context. And do you have any other tips on providing copilot better context? Yeah. So there's a blog post that we released recently, ever since we added support for adding in more context with other open tabs other than the one that you have open. It's really improved a lot, and the types of suggestions it can make. It doesn't have any concept of a repository. It only knows what you're prompting it with the files that you have open. So yeah, a lot of people are writing interesting posts about prompt engineering and how you can prompt it with better context. Because really, it's all in the way that you ask it questions and you can get more refined responses. So I say, like, the more context, the better with how you're asking it stuff.