Who’s Faster at Building an Energy Generation Data Visualisation Platform: ChatGPT or a Developer?

So hi, everyone. So like the kind of talk title says, today I'm going to be talking about basically who's faster at building a data visualization platform. So this is a question I got quite a lot whenever I'm working on something, either from clients or from friends or even people working in the same industry when they always ask me, did you try just asking ChatGPT? Can't ChatGPT do this? And it's one of those things where on the spot, immediately I'm always like, no. No, no, no. ChatGPT can't do these things. And at one point I started doubting myself. So hence why I kind of did a little experiment.

A little bit about me. I'm going to keep it super short. So yes, as mentioned, I'm a tech lead developer, and recently I've been moving over into data engineering. And a little fun fact is I've lived in six countries and moved eight times before turning 18. No, my parents are not diplomats. That's usually the automatic question I get.

Okay. So a bit of stats about generative AI and how it's used in the workplace. All of the stats you're kind of going to see are from the U.S. The U.S. does a lot of surveys. It's kind of great when you want to show some statistics in a presentation. But in general, about 56% of U.S. workers have said they use generative AI for their work tasks. Only about 9% of those use it on a daily basis. But it's still becoming quite an important point of discussion. On the flip side of things, around 55% have said the quality of generative AI matches the output of experienced human workers. So those two statistics look very close together. They're not necessarily the same people who answered both questions in the positive. And you're going to see why from the results. I'm kind of spoiling it. But yes, you'll see that a little later on.

Okay. So ChatGPT caused a wave of interest in 2024. In March 2024, it had about 180 million users. Which is quite an impressive number. And like you can see, it kind of took only about five days to reach 1 million users. So it became very popular very fast. We developers are definitely one of those who use it most. Because it's used...

68% of people said they used it to write code. So we all love charge50. We're definitely funding that machine for sure. So today we're going to look at three things. We're first going to see, okay, how am I going to be conducting this experiment? How am I going to develop the application? And of course I did promise a competition. So a competition there will be. And I'll show you a side-to-side recording of what happens when I try to build something and charge50 tries to build the same thing.

Before that, the first time I kind of gave this talk inside my company, a lot of people asked me, why are you using charge50? There's so many other models out there that you could be using instead. Yes. Yes. You could. You could test this against another model. The main reason why I chose charge50 is you're gonna get lots of different rankings for the models out there. I used the one by Eval Plus. So they used both a human evaluation and something called MBPP, which is most basic Python problems, to just test how well these models compare to each other when they're doing code generation. And like you can see, charge50.4 is at the top at the moment. The others aren't too far behind, so they are catching up. But that's kind of the reasoning behind, you know, using this one as a comparative measure.

Okay. How are we conducting this? So you saw in the title, it's an energy generation data visualization platform. Very long thing to say. Why did I choose this? It could be for sustainability. It could be some nice cool graphics. But actually, it starts from an old story back in uni. A very stressed out student. I had exam after exam after exam after exam. And of course I found the only nice way to procrastinate, which is probably a very nerdy way to procrastinate. It's spend my days looking at the most beautifully simple but also very boring graphic of GridWatch. This looks very boring, but trust me, when it's either that or studying for mechanics, I much prefer looking at this screen all day. So what it shows you, it just shows you power generation in the UK, from different sources.

It's kind of a nice basic platform. And when I was doing this talk, I thought, oh, this is perfect. You know, some simple graphics. Some simple stats, which are available to everyone. So let's use this as the basis of, you know, building this energy generation data visualization platform. I need to come up with a catchier title than that.

Okay. So that's a good base. But let's just set some ground rules, you know, making everything plain and simple, so we know there's no cheating going on on either side. So the first thing is it can come under any format. It can be a website. It can be an app. It can be a business intelligence tool. Chargept is allowed to create any format that it wants.

The second thing is both I and Chargept, you have to use the same source. So we're going to use something called electricity map, which I'll show you later on. The same data needs to be shown. So I'm going to stick to very basic dials and graphs. I just want to see energy generation. And I want to see percentage of renewable versus carbon neutral over the last 24 hours. Mostly because the API only let me have 24 hours of data. But that's about it. And the fourth one is it has to be done in one shot. So you're going to see in the recording, I did put a little stopwatch on the side, so you can see that I didn't pause it at any time.

This is where the data's coming from. It's a platform called electricity map. I take no part in this, so this is no advertising at all from this. It's just a cool little platform, which is open source, and they let you use their API for free for 24 hours of data. Any more and you have to pay. Okay.

So how did I build the data platform? So I'll first give it from my perspective, and then I'm going to tell you how I kind of interacted with chatGPT before getting the final result. So in building data platforms, there's typically three main components. So in the end, I actually didn't need to do any transformation, so that could get rid of.

But the three stages are ingestion, storage, and visualization. So the three tools I used are airbyte. Airbyte is an open source ingestion tool that you can use. It's really nice and simple. You'll see it in the recording. I used BigQuery for storage and I used Google's Looker for visualization. So that's the three things that you're going to see me set up.

Just to tell you things to consider at each stage, in the ingestion, we are ingesting data from an API. One thing I did find out through trial and error is you do need to flatten a nested object so that Looker can read into that data. That's going to become important in the recording. That's why I'm telling you this now. And what's really nice and simple is that airbyte does have a native connection to BigQuery, so it makes it a lot faster to ingest that data. BigQuery, storage, plain and simple, single data set, nothing too complicated. And then Looker on the other side, we're just going to keep the interface nice and simple, you know, simple and usable, not going to overcomplicate it, just so you could make it more complex if I wanted to, but we're just going to stick right to the basics. And it's going to look something like this in the end. So I'm showing you the end result now, just so you can see how it compares and I'm also going to show you the same equivalent with GPT. So that's what it looks like. Pretty similar to what we were expecting. Now what about with GPT?

So when I'm using ChartGPT, what am I keeping in mind? Firstly, it's all in the prompts. Using ChartGPT is all in the prompts. I'm sure you've heard prompt engineering being mentioned at some point or other, including within this conference, but there is a really beautiful documented website about prompt engineering, about all the different forms of prompt engineering, which I highly recommend looking at it. It's actually super valuable to improve the results that you get. But the main idea is this. So firstly, tell ChartGPT that it's an expert. It loves being flattered, you know? So when you're opening up, before you're telling it what you want it to do, saying it's an expert in something just sets the ground of, you know, how is it going to respond to you. Secondly, keep your sentences short. Thirdly, tell ChartGPT to ask questions before it replies.

This is really important because it's going to be gathering context before it actually provides you with an answer. Kind of links with number four, provide code and any relevant context. Five is be assertive. This is always a fun one. You guys probably know, but it's impressive the number of people who say thank you and please to ChartGPT because they're worried about a robot uprising. It's not going to make a difference. If they take over, they take over. But, yeah. Be assertive. No need to say please and thank you. And, of course, provide feedback at the end. If there's something that doesn't, you know, isn't what it's supposed to be, just make sure you're providing feedback and hopefully it's not hallucinating, but as you'll see, it can hallucinate as well.

And that's what it looks like. I have shown you two different versions of what ChartGPT has generated. That's going to become important. But you can see it's very similar to, you know, what I built myself and what we're kind of expecting from the get-go. So that's the little setup.

Okay. Now, as I promised, I did promise there was a race, I'm going to attempt to comment this. It's six times the speed, so it's going to be very fast. At the top left, you'll see there's ChartGPT. Bottom right is going to be me. Okay. Ready, set, go. So, with on the bottom right, I'm currently setting up the database, and there you can see Airbyte connectors coming in. At the top, you see ChartGPT. I'm giving it all the different contexts. It's kind of listing me out a huge list of questions I'm going to have to answer to give it more context. Bottom right, I've now set up the connector with the API. So currently, you can see I'm flattening that nested object, so I'm doing a very repetitive work of just telling it, okay, find the data in these different localizations. Top left, still answering questions. There's a lot of talking to ChartGPT. It's giving me more questions after it's provided it some context, so that's taking it a while. Bottom right, I finished with flattening everything. I'm now doing that connection up to the API and to BigQuery, so currently putting in all that detail.

Top left, you see ChartGPT's already generated code. Great, it's even giving me a folder with all the data in it, so put that up there. On the bottom right, I got asked a question by a colleague, so I got a distraction. Top left, now running the code, we can see there's first errors, so having to go back and debug it. Bottom right, I'm still doing all of the connectioning up. Top left, you could see the first results, not looking so good, though, so kind of need to give it all that feedback. Bottom right, I'm still connecting to BigQuery, so that's kind of all of the ingestion tool coming in.

Top left, currently getting it to fix some of the graphics because they don't look quite right, but it's starting to get there. Bottom right, it's currently ingesting all of that data, so that ingestion is taking a bit of time.

Top left, it doesn't know what a dial is. I've had to give it an image for it to know what a dial is, so that's currently debugging there. Top right, building up that dashboard, getting all that data up there.

Top left, don't know if it's got on the dial yet. Yes, it knows what a dial is. Amazing. Great. So that part's debugged. Bottom right, I'm now putting in together the graph, getting everything set. Top left, still getting all that code running. Dials are there. Graphs are there. Just not looking quite right.

Bottom right, I'm building that final little breakdown of the energy generation. Top left, still giving it some feedback, you know, that that dashboard's still not looking too good. Bottom right, currently splitting up for the different... Actually, now I'm just renaming everything, so now I'm just making it look pretty. Top left, still debugging it. The graphs are looking a bit better, but still not exactly what I want. Bottom right, just cleaning everything up, and done. Great. Bottom right, 26 minutes, developers. Top left, we're seeing which at GPT could've made that code a bit shorter, but right now it's giving me all the colors are the same in that graph, so I'm having to explain those colors aren't quite right. It just changed everything from blue to purple. Okay. But it's still not quite right. And now turning it to breakdown. Colors look a bit correct, but it still doesn't get where a stacked graph is.

Now doing the stacked graph, kind of pinging it forth, you can see it's generating the same thing. At one point, you see there's a frustration. If I slowed this down, I'm asking it, do you know where the stacked graph is? And it says, yes, that. It's not. And I even gave it a picture. But it's really hallucinating here.

Frustration is mounting. Taking a very long amount of time back and forth. I'm getting passive-aggressive with it at this point, because it still doesn't get where a stacked graph is, and the time is ticking. This is six times the speed, by the way. Imagine doing that at normal time. Still turning. I'm still stuck on that stacked graph. Everything else looks great, but it's really struggling.

Oh, I've given up here. Now I'm just doing the width, I'm setting it to be nice and pretty. That stacked graph is not going to happen this time around. So getting to the end, just doing the width about right, making sure it's looking kind of similar to what we expected it. Still turning. Still giving it feedback. It's adapting, slowly but surely. We're about to get there. Is it going to be soon? And... Still that last little bit of feedback. There you go. 43 minutes. So... Massive difference there. So, I finished in 26 minutes, and Chatjipati finished in 43 minutes. So, concretely, what was happening? So, Chatjipati is lagging behind, but it's really not consistent in terms of timing.

So if you remember when I showed you the two different outcomes from Chatjipati, there was one graph which did have that stacked graph, and a second one which you just saw being built there. So the first one actually only took 25 minutes to build. It got it really quickly, it was perfectly fine, and then here we saw it was more nearing towards the 45 minutes. So Chatjipati can occasionally be faster than the developer, but it's really not consistent. So you're not going to get the same answer each time, and sometimes it's going to be incredibly frustrating to go about it.

So what happened? This is a good representation of what happens when you use Chatjipati. Yes, it's going to generate code faster, but you're going to spend way more time debugging the code it's given. You're just going to have to do back and forth again and again, cleaning things up. So what were the key differences with Chatjipati and me building this? So firstly, there's decisions based on scalability. So in Chatjipati, if you saw, it was a really basic website that it built, and it pulled the data directly from the API. On the other hand, I used an ingestion tool and I used a database. Why did I make that decision? Because I knew that the API only had 24 hours of data, and I was going to need to accumulate it if I want to see that data along the long run. So there's that decision based on scalability. Secondly, there's access and experience. So firstly, Chatjipati built everything from scratch. So like we said, a website. On the other hand, I used existing tools out there. So Airbind, BigQuery, and Looker. This is both a mix of experience, so knowing that these tools will do the job for me, and also access. Chatjipati can't access these tools. There's logins behind it. It's going to have to do things from scratch instead. This one is quite interesting. So if you saw at the end, the big thing that it got stuck on was the design. When you're first giving it context, you might know what you want. You might give it the design that you want, but something happens called subconscious preference. So for example, the colors in a graph. Do you know the fact that you prefer a stacked graph rather than the bar charts? All of that can be happening subconsciously, and you'll only realize you don't like it when you see it, and you're like, yeah, that doesn't fit quite right. Compared to when you're building that dashboard, and your subconscious is coming into play automatically. And finally, complex problem solving.

So in terms of this, they're kind of equally matched. With Chatjipati, it can't run the code directly. You can set it up to try that, but for now, you know, it can't run the code directly. So testing and debugging is really done through interfacing, through me giving it the errors, et cetera. On the other side, if I had to debug it myself, it's either I know it by experience, or I actually have to take a long time to research the error and find how to fix it.

Okay. So rating complexity-wise. You know, developers, easier to handle complexity. You've got context, you've got experience, that's much nicer to do. Chatjipati has basically the basics. And it's also, you know, it's based on probability, the answer it gives you. Customization, subconscious preferencing, so developer is kind of better in that aspect. Technical expertise, anybody can run Chatjipati. So you still need some sort of, it doesn't have five stars, because you still need to know what you're debugging or what's wrong by looking at it. Time to market, about the same. So I'm really hoping nobody is putting Chatjipati code straight into production, so you do need to debug a lot of it before it actually gets released. So it gets about the same mark as being a developer. And setup time. Chatjipati generated that folder really quickly for the basic code, and then it was just refinements, so setting up is quite quickly, while developers, just normal average time that you'd expect. And finally, scalability and security. It's all about context. You need to know what's, you know, environment you're building stuff in, what's the security protocol that you'll need, and also what is the long-term objectives of it. So being a developer is better. Chatjipati won't have that unless you try and give it to it.

So why are we talking about these two solutions as if they're non-complimentary? So you're probably doing this already under the hood, where the best way to use Chatjipati is as a tool. It's not as a replacement. So that's really, you know, the question of could you combine the best of both parts. The answer is obviously yes. So where Chatjipati is best is definitely it's basically Google on steroids.

So where Chatjipati is best is definitely it's basically Google on steroids. It can get you answers quite quickly. You don't have to do as much digging into. It's a great learning experience if used right. If you don't know something, it can give you bits and pieces of code, bits and pieces of information to help supplement your own learning. It's also amazing in documentation. I don't know how many people like writing documentation. If you do, you're a unicorn, you're going to get hired very quickly. Most people don't. Chatjipati is great to do that. And it's much faster to generate smaller pieces of code. But the advantage of being a developer is you get intuition. You know when things don't feel quite right. You know what's being used, depending on the platform, depending on the context. That's the second thing. Of course, there's more creativity in there, and also there's that debugging aspect. So knowing where things are going wrong, where things need to be fixed, et cetera. So overall, Chatjipati is definitely a great tool. But it's not ready to replace us. At least not yet.

Thank you. APPLAUSE Basically, the first question you just answered, and just pay attention to it, it was asked by Chatjipati. Will AI replace developers? Maybe you have some extra thoughts on what you just said.

So genuinely, I think as AI in general keeps progressing, it's getting better and better at automating certain tasks that we do. So it's taking away some of the things which we find tedious, but it's definitely making our job more efficient. So I remember at a conference we were having this discussion and it was very much comparing how a farmer used to use basic tools to collect his crops, and now they've got tractors. They can do all of those things automated for them and much faster. Very similar as a developer. AI is kind of supplementing our job. It's making us more efficient. It's making us go quicker, et cetera. But I don't think it's, at least not yet, going to get to that point. Totally. Totally. Yeah, so we are safe, folks. So Chatjipati, it's not your time yet.

Let's pick this one. I believe it's based on your statement about some nice resource with the prompt. So the question is, what's the best resource, you know, with prompts for development in Chatjipati? So I'm guessing that's going to be kind of what, you know, which prompts should we be choosing to make sure Chatjipati is more efficient. Yeah. So there's a whole field called prompt engineering. There is a website, I forgot the name, but if you look up prompt engineering documentation, there's a website which groups, I think there's more than 20 different techniques that you can use. It's really good. At one point I was testing it out for anomaly detection. Chatjipati as a base was only about 20% efficient. As soon as you start using prompt engineering, it became to 60% efficient. So it's definitely worth looking at those techniques, trying them out. They really do actually improve the responses that you get. Yeah, I actually can follow up on that as a person who is delivering session on prompt engineering quite often myself. Yeah, tons of different techniques and tons of marketplaces with prompts, like free prompts, paid prompts.

I'm not an active user of those ones, but maybe you can find something specifically for the development. I believe the vast majority of prompts there are for your creativity and artistism, but I have no doubt that something for developers are there as well. And also, yeah, this is actually not a question but kind of comment to make your competition more fair. And someone says that you choose requirements ahead of time and Chatjipati didn't know this information in advance. So maybe third party choosing topic is the way to go. And basically, my personal thought, you actually opened the rabbit hole to all new kinds of events. We can call it like cyber sport, right? But not about gaming, but bot developer versus human developer. So maybe again, idea for conference organizers. What do you think about that? I think it's a very interesting question with that third party choosing the topic, because this digs into a lot of the advantage of knowing context, because if you saw both, you know, choosing the requirements beforehand means that I knew what I was building, but it also meant I could answer Chatjipati's questions when it was asking me for the context. So I think going with a third party will have a very, very different outcome, because firstly, Chatjipati won't be able to ask me questions because I have no idea. So that's the one thing, is that there's a disadvantage with Chatjipati when it doesn't get the context just as much as on developer side. I think it will be an interesting thing to try out, because who knows what would happen in that situation. Absolutely. Absolutely. Yeah, maybe just a thought of mine, so maybe a rise of agents will help here, because in my understanding, AI agents or generative AI agents are exactly, I don't know, entities, how to call them, that are proactive, right? So they are not just waiting for your input. They start asking you themselves. This is a kind of, I don't know, a bit scary level of development of generative AI. Let me pick something else from our growing list of the questions, so you'll be super popular in Q&A spot after we leave this venue. Yeah, let's pick this one, kind of futuristic. Where do you see AI taking us within the next five years? So we are 2024, 2029, or maybe around number 2030. Where are we? That's a good question. I think the first thing is, I feel like the open source models will definitely be at the point where they overtake closed source models, because so many people are active in the community improving the models out there, like we saw Chatjipati 4 was at the top of that list, but all the open source models are very much improving quite quickly. So I feel like in five years, we'll definitely be at the point where there are so many open source models out there that people are going to have much more creative approaches of where they apply it. At the moment, it's mostly used in a global sense as a chatbot. There are other uses of it, but the most global one is as a chatbot. Who knows where it's going to take us in five years? Yeah, folks. Let's meet all together here again in 2030, and I'll ask organizers to find video recording of this session, and yeah, let's see where it comes. Marked. Let's pick this one. Super quick one.

Have you used Chatjipati to build your slides and text, I believe, for this session? Actually, no. No, I haven't. I use it on a regular basis, especially to write emails. I'm very socially awkward, so whenever it's to do with writing messages or emails, I write it for that. I didn't actually use it for my slides or my text.

Yeah, actually, same for me. I use different genres of products, services almost every day for multiple tasks, but never use this for the presentations. I want to keep something for myself where it's like my only sole source ideas, but it might change, of course.

Is there an alternative to Chatjipati that keeps the links to the original documents like a combo of Chatjipati with Google? I believe we are talking about stateful model, right? Maybe you know different services that support this kind of experience. To be fair, I haven't actually dug into it that much, so there's nothing that comes to the top of my mind, but I feel like there's definitely going to be something out there if you do a bit more research. Perplexity. Oh, yeah, perplexity. Perplexity is great. Whenever you're looking for any sort of sources, you kind of ask it, can you give me a statistic? And then it will tell you where it comes from. Yeah, 100% perplexity. Good, good. Thanks for your help.

Would giving Chatjipati code examples for styling and graphs have sped up the process? Yes, definitely. So that's one of the prompt engineering techniques is something called few-shot learning, or very similar to that. It's when you do give an example code, and then at that point, it's very good. It's much quicker. I think in this competition, it would be helping it a bit too much. That's why I didn't do it. And that's why it's very much that hybrid model of both, using prompt engineering to tell it in which direction to go to, will make it much, much quicker and much more. It will hallucinate less, basically. And actually, you've just answered this question, what was the meaning of one-shot prompting? So you explained it very well, one-shot versus few-shot prompting, very, very important techniques.

That's why I didn't do it. And that's why it's very much that hybrid model of both, using prompt engineering to tell it in which direction to go to, will make it much, much quicker and much more. It will hallucinate less, basically.

And actually, you've just answered this question, what was the meaning of one-shot prompting? So you explained it very well, one-shot versus few-shot prompting, very, very important techniques.

And let's pick the last one. Have you considered leveraging dedicated LLM tools for this benchmark, like Pythagora? I haven't heard about this myself, but maybe these are some other tools that might help. Yep, so I think you could definitely use more tools out there. I think one thing that would have been quite interesting is, like you saw, I was kind of loading the website by myself. One thing that's kind of good to expand on, a little side project if people want to try, is to use something like LangChain, who will automatically generate that code, run it itself, and have it more in a loop, rather than manually having to run that website. That would be quite a cool experiment to do.

Thank you very much, Chloe. And yeah, let's meet again in 2030, and thank you very much. Thank you. Round of applause!