The Rise of the AI Engineer
From Author:
We are observing a once in a generation “shift right” of applied AI, fueled by the emergent capabilities and open source/API availability of Foundation Models. A wide range of AI tasks that used to take 5 years and a research team to accomplish in 2013, now just require API docs and a spare afternoon in 2023. Emergent capabilities are creating an emerging title: to wield them, we'll have to go beyond the Prompt Engineer and write *software*. Let's explore the wide array of new opportunities in the age of Software 3.0!
This talk has been presented at React Summit US 2023, check out the latest edition of this React Conference.
FAQ
An AI engineer specializes in using AI technologies to design, build and implement solutions and improvements in various applications. They often work with AI APIs to integrate AI capabilities into systems and products, focusing on both creating AI-driven solutions and leveraging existing AI tools effectively.
AI engineering has seen significant advancements, from basic tasks like handwriting recognition to complex image generation and superhuman performance in various tasks. This evolution reflects broader technological advancements and an increased integration of AI into different sectors.
The rise of AI engineering is driven by a demand-supply gap, with a significant shortage of ML engineers compared to the demand for AI capabilities. This has led to high economic value for AI engineering roles and encourages more professionals to enter the field.
Future trends include more widespread adoption of AI across industries, increased focus on ethical AI use, and advancements in AI technology that simplify integration and usability. AI engineers will likely play a central role in shaping how AI evolves and is applied in various sectors.
ML engineers primarily focus on building and training models, whereas AI engineers are more involved in integrating these models into applications and systems through APIs, handling the practical implementation and user-facing aspects of AI.
Yes, non-technical individuals can engage with AI through tools like mid-journey prompting and other AI-powered applications that do not require deep technical knowledge, allowing a broader range of people to utilize AI technologies.
AI engineers today focus on integrating AI into user applications, enhancing AI usability in products, and improving AI interaction with end-users. They work on both the infrastructure and product side of AI, ensuring efficient and effective AI implementations.
Video Transcription
1. Introduction to the Rise of AI Engineer
We have 20 minutes to cover the rise of the AI engineer. AI is the Moore's Law of our time. The image generation capabilities have developed significantly over the past years. However, there is still a lot to be proven in the field of AI. Short-term impacts on AI are popularly blamed, but it's important to follow the money to determine what is valuable. Jasper, generative images, and chat bots are examples of successful AI projects.
All right. Thanks, everyone, for tuning in. We'll see you next time.
Okay, so we have 20 minutes to cover the rise of the AI engineer. It is a thesis I've been developing for the entirety of my career, and I'm going to be focusing on AI engineering basics. If you are new in some sense, if you are still here, you're at least somewhat interested in it, and I intend to do justice. I'm going to go very fast. This is the first time being at one of my talks. You know that I speak at the rate that I listen to podcasts on, which is at 2X and sometimes 2.5X. All my notes are on my website, as well as on the Latent space if you search rise of the AI engineer.
If you've been somewhat living under a rock, or you've been overwhelmed by too much info, these are the ones I want you to have in mind. AI is the Moore's Law of our time. These are all sort of progress metrics in the past ten to 20 years. What you have on the left is the image generation capabilities that we've had developed over the past eight to nine years. You know, in 2014, we used to sort of develop this as a grainy picture of a face, and now we complain if we can't place our dog in the middle of a generated image. In 2000, we barely solved handwriting recognition with the MNIST dataset, and now we can mostly are at superhuman levels in quite a few core human capabilities that we understand. But it's also caused a lot of AI madness. This is where I tend to distinguish myself from what some of these AI hypebeast people are. You don't find a lot of these at the top of the list of people that you might see online.
So a lot of people are sort of mentioning AI and try to have some AI strategy on their earnings calls, but there's very, very little usage, actually, and there's still a lot of stuff to be proven out. And in particular, one of the most impactful or one of the most hyped projects of the whole year is Auto GPT, which if you're on the stock market, you pretty much run into this over the years, but it's a pretty crazy case in Django, and it's pretty crazy to have that and then to have other people also commenting that no one's using this thing, which is really interesting. There's also sort of I call this sort of manic depression. In the same month, you have investors saying AI has peaked and then also AI is back, coming from San Francisco, it's very common to simultaneously have the feeling that this is what we're looking for, and that this is what we're looking for, and this is all short-term momentum against the backdrop that we are in a longer arc of history where we're just inevitably marching towards sort of a bootstrapping of digital intelligence that will surpass us within our lifetimes. And I would say it's very popular to blame sort of short-term impacts on AI, Chegg is one of them, but I think for the most part, it's not in sort of all the papers. I have a monthly recap on the In Space in my newsletter where I sort of recap the top things you should know as an AI engineer, but really it's just, follow the money. Like, that's, you know, it is the least worst form of figuring out what is valuable that we've developed in the time that humanity has been around. So who's doing a good job? Who's actually making money? Jasper's arguably not doing as well, but I think a lot of people are doing their own best. So there's a lot of people doing their best now, but they still went from zero to 80 million ARR in two years. Generative images, now reportedly at $100 million ARR with a team of 13 people. And chat bots, obviously, OpenAI has a billion dollars just on chat.GBT alone.
2. Progress and Opportunities in AI
There's an incredible amount of progress and excitement coming from AI. AI is shifting right, which is a huge shift in the past ten years. The first answer is to do all of the machine learning stuff. The second answer is to do data engineering. The third answer is to do machine learning on Coursera. There's a spectrum of roles in an AI-enabled organization, from the ML side to front-end and product engineers.
There's actually a team of 13 people on that team. And Danny Postma is a good example of that. So that's just some data points I want you to have in mind, that AI is real, AI has a lot of opportunities. There's a lot of hype, but there's also some real long-term progress here.
And real, like, the real central thesis that I want to deliver is that there's an incredible amount of progress and excitement coming from AI over the past few months. And how it's been moving over from React and JavaScript, just like myself. This is all summed up in this thesis, the rise of the AI engineer I wrote a few months ago. And the central thesis is that AI is shifting right.
This is an actual XKCD comic from 2013 where they talked about, like, oh, if I need to do bird recognition, I need five hundred percent, I need to do Bird recognition. I need to do Bird recognition. 17.7 percent zero shot performance up to state of the art. 78 percent on some metric. That is a very, very different state of AI as it is today. From, hey, you need to hire a research team to, hey, you need to prompt a little bit. That is a huge shift in the past ten years. That is a huge, huge shift in AI as it is today.
So, first answer is do all of the machine learning stuff. I have done the machine learning courses on Coursera, not helpful at all. The second answer is, I'm sorry, it looks like the screen is gone dark. The second answer is that you should do data engineering. The third answer is that you should do machine learning on Coursera, not helpful at all. After having done that, it is not really super helpful for me to understand all of the foundation model progress of today. Should I pause to get the screen back? I think the AVT is still figuring it out. I'm still trying to figure out what is happening here. It is still playing on the screen over here. I'm just going to try to keep this moving without screen assistance. I want you to imagine in your head a spectrum from left to right of the roles in an AI enabled organization. On the left you have the ML side, and on the right you have the ML side, and on the right side, you have the ML side. On the left, you have the ML side, and on the right, you have the ML side. Front-end and product engineers.
Comments