React Native AI: Bringing on-device LLMs with AI SDK to React Native

Hello everyone, it's really nice to be here today in London. Thank you Git Nation for inviting me. And today I'm going to talk about React Native AI, a library to run on-device models in React Native. And here I would like to ask you, who ever played with React Native or who works with React Native? Please raise your hand. Oh, that's quite a lot of people. I didn't expect. And who here played with AI SDK from Brazil? Please also raise your hand. Okay. So less than React Native. I'm actually shocked.

All right. So today we'll try to connect AI SDK with React Native and we'll see what will come up. But before we start, let me introduce myself. My name is Szymon Rybczak. I'm a senior React Native developer. At Callstack, I'm 19. I've created many React Native libraries. React Native AI is one of them. I post a lot of stuff on Twitter, on X. So if you like React Native content, please follow me there.

And yeah, as you know, you can run AI into places. Of course, you can run AI in the cloud. And that's where most performance models are inferred. I don't know, Gemini 3, every model is inferred there because the cloud has a lot of compute power, etc. But actually, you can also run AI on the device. And devices that we carry, all carry in pockets, are really powerful. And sometime ago, last year, last summer, I was wondering, is it possible to run on-device AI on mobile? You know, for the past four or five years, I was doing all kinds of mobile apps in SwiftUI, in React Native, and I really love to play with good mobile apps. And of course, I'm a React Native developer. So I thought, is it possible to run on-device AI in React Native? And after looking at the stats from some latest iPhones, like two years ago, the stats are really good. Like on the single core, actually, iPhone 15 Pro has the same performance as some iPad Air.

It really showed me that devices that we carry in pockets are really powerful. And another kind of point of inspiration was this tweet from CEO of Vercel, Guillermo Rauch, who tweeted about new initiative, experimental initiative in Google Chrome, that Google Chrome will now ship some Gemini model, 2 gigabytes, and everyone will be able to access in Google Chrome. So if there is a low internet connection, but internet connection, or we are offline, there will be ability to play with it, to ask some questions, and use it maybe as a fallback. And it was really good. It was super fast. And I've played with it on my Mac, and it was super good.

So I started researching this topic. And when I opened the source code for this Chrome Window AI, I learned about providers in AISDK. And the idea of providers is that AISDK contains many functions such as stream text, generate text. And basically provider is an implementation. So you have one API, one unified API. And for, I don't know, Antropic, Gemini, OpenAI APIs, you just create providers. There are some original providers like created from Vercel team or community providers. And everyone can create one.

The schema is public and you just need to create it and you can plug it under generate text, stream text, and all kind of AISDK functions. So I thought, hmm, let's create one. And I really love Vercel AISDK. And if you haven't played with it, I really encourage you to. Because this is the amount of the code that you need to plug AI model with React Native AI. That was the idea back then. And you import generate text or stream text function from AI package. Then you call it, you pass the model and you pass the prompt, some probably message history and some other configuration props.

And I thought that it would be really cool to do it. And you just need to replace two lines from your implementations on the web to use the React Native AI implementation. And I was sure that I want it on the JS site, on the TypeScript site. But I had to find a way to actually power it, to have an engine that will deliver the answers and run the inference on Android, on iOS and any other platform that React Native supports.

So I found this engine called MLCLM. And it's a really cool project because it's a universal engine. So it's the same idea as React. You have one code and you can run it basically everywhere on multiple platforms. And here you have one model and you can also run it on multiple platforms. So I thought, okay, let's hack. I really love the vibe that here we have universal React and here we have like a universal engine. Let's connect these two pieces.

And of course, it took a bit longer than expected for me to create some proof of concept. But after some time, a few longer evenings, I've created the first-ever demo of running on-device LLMs in React Native. And here is the pre-recorded demo. So I pick the model. The model is right now downloaded from a network. We need to download the model once. And then it's saved as cached. And now I will turn on the airplane mode. I don't have internet connection access. And I can ask some questions. What's React Native? And after a few seconds, the stream will come up. And it's all running on the device.

It's running on iPhone 15 Pro. And I was really shocked by the performance of it. And I don't need internet connection. I can ask any questions that I want. And it worked. And it was cool. But it required one thing. It required from every user to now load like a few gigabytes model. And that wasn't the best.

So when we take a look at the chart I have presented at the beginning of my talk, actually on-device models, there are two parts of on-device AI. There are third-party models that you download from the network. And these can be for multiple purposes. But every user needs to download it. You cannot pack it into the app store because the app would be very big. And that's not a good practice in general.

But I was always dreaming about built-in on-device models. So imagine that iOS or Android would ship the model with the operating system itself. So everyone would have it. Everyone would have the same model and we could all use it. And that was always like a dream when I was working on this topic for some time. And yeah, this year, WWDC 2025 in June, Apple announced the thing. Apple announced Foundation Models Framework. So a way to run on-device LLM created by Apple on iOS, on MacOS, on iPadOS.

And I really was excited about this one. And as usual, I was watching the WWDC with my colleagues from Caustic. We are also doing some garrison posting. And you know, we're making jokes around Flutter because we are React Native. I've tweeted about, you know, we have always this kind of competition. Who will bring the new API to React Native. New native API to React Native. So it will be accessible. And I've tweeted it after I saw it on the conference.

And I started, you know, cooking. I started playing with it. And actually after a very long evening, the same day, I've managed to plug Apple Foundation Model. I had to update Xcode. It was super experimental. I've regretted in some parts this decision, but it was worth it. So here is a demo of running Apple Foundation Model, the Apple LLM with React Native AI in React Native, and the performance of it is super cool.

So here I have generate text function. I get the Apple Model with React Native AI helper function. And on the right side, I have my iPhone 15 Pro. I turn on the airplane mode. Here is important fact. I don't need to download the model. The model is shipped with the operating system. And I don't have even streaming right now, back then I didn't have. And it's super, super quick. And even with streaming. Right now, we support streaming, of course. It answers super cool. And yeah, like we were really hyped on it. And yeah, like the answers really make sense.

And it was really nice for us to see that Apple is also investing in it. And yeah, I posted the demo on Twitter, of course. And people asked, okay, cool. Can you show the code? Of course. And this is the code that you need to run it in React Native. And it's so cool. Like we are really proud of this code, you know?

So, thanks to Vercel SDK and React Native AI Apple package. You can just pass the provider, call the function, and you have the results. Then you plug it into your UI code, into your chat app, or whatever you'd like. Pass the tool, calls, whatever you like. And people really love it on Twitter.

Like Jared Palmer, who is right now at Microsoft, working on VS Code and Copilot. CEO of Vercel on-device Apple models with AI SDK. And we've shipped it under 24 hours. Here's a tweet from Mike, our CTO. And it really presented us the iteration speed with React Native AI and all kind of helper functions and ecosystem that we've built around on-device LLMs. And we've released it after some time. We've cleaned up the code, of course.

And right now, we support a lot of things. We support structured outputs. Of course, we have the first-class Vercel ASDK support. And after some time, we are also mentioned on Vercel ASDK docs. So, we are official community provider. Text generation, streaming, tool calling. That's actually super powerful, because you can create some tool calls. And for example, create a tool called Create Calendar Event. And you will ask, without internet connection, could you please create a calendar event today for the after party? And it will access the native APIs for the calendar. And it will create the event in your calendar without internet connection. Embeddings and transcribe. So, we also support TTS and STT.

And yeah, here is... I'm really proud that it's integrated with ASDK. It's really cool software. If you haven't tried, once again, I really encourage you to play with it. And yeah, please start the repo if you want to play with React Native AI. We really look forward to have some users. Please file an issue if you have some problems with the library or question. We've essentially hit 1k stars.

And yeah, we are not stopping, and we will be shipping more providers for on-device models. And that will be it. Thank you so much. Are there any news about built-in models on Android? Yes, actually there are. Google Pixel 9 Pro actually shipped it. And like every new Google Pixel we ship, we've shipped Gemini models. So yeah, we are already trying to integrate with it and we'll provide official support in React Native AI. Cool.

And what kind of real-life implementations do you see for this in actual mobile applications and tools? Privacy. Privacy is a thing. You are not sending any data to the cloud. You are not sharing it with anyone. So, if you have some private information, you can also access this offline. So, privacy, offline needs. And yeah, we've already heard of examples of health app integrating where you can upload your PDFs, some results from your doctor or whatever, and upload it and the on-device model can read it, can analyze it, and can have some summary or whatever use case. So, this is the thing that we are seeing. Perfect.

Perfect. And are there any... Wait. Did I not just ask that? Wait a second. It's fine. Okay. Does this work on simulators? Yes. Actually, yes. If you have macOS 26 and you enable Apple Intelligence, you can also play on simulators on the iOS side. And on the Android side, you need to have the physical Pixel 9 Pro. Cool.

And are there any news about built-in models on Android? Yeah. You just asked that already. I was like, am I going crazy? It's all fine. Yeah, there will be. Okay, perfect.

Can the LLM be trained if only used offline? Is this as effective as seen in online models? So, of course, like the offline models are not that powerful as the ones that are running in the cloud. And this is a trade-off. And there are some pros and cons on this one. As I mentioned, like privacy, like it's super, super quick. We don't need to have an internet connection. So, as always, use the right tool for the task. Cool.

Someone wants to know more about you. What is your day-to-day role and what you're working on right now? Yeah. So, I'm Senior React Native Developer at ColdStack, and we build all kinds of React and React Native solutions. For the past few months, I was working on one of the React Native apps, doing text-to-app with AI. And it was recently launched, and it was a super great experience to build. And it's a super great experience to see, like, many AI apps using React Native and becoming, like, top 10 apps in App Store. So I was doing that for the last few months. And apart from that, we are doing a lot of open source projects, React Native AI, React Native frameworks, and cool new stuff. Sounds exciting.

How often do the models get updated? Actually, that's a great question. Like, Apple is constantly updating it. I'm not aware of any change log if they are not sharing it publicly. But we've filed a lot of issues at the Apple forum that, you know, like, tool calling wasn't working perfectly and they actually are, you know, implementing fixes and upstreaming it. So they are working constantly. And with, like, implementing an app, like, adding support for React Native AI Apple in your app, like, you are kind of subscribing to lifetime updates from Apple. Because, like, if they started shipping their LLM with new devices, they will come up with better LLMs. And you are, you know, you don't need to do anything, like, with this. Cool.

And can we implement RAG with this library? Yes. Yes, there is support for embeddings. And it depends on the engine that you are using. But yes, in general, we've added also support for it. Cool. And does the model on device get updated with the user's data? If iPhone knows what apps use when and suggests them and et cetera. Can we use this data somehow not to fetch it, but to flow with it? Hmm, so, OK, so the model on device learns about the user data from other apps, right? Yeah. OK, I'm not sure if this is what Apple does under the hood. I don't think so, to be honest, because that would probably be some kind of pervasive vulnerability. But yeah, I'm not sure. Apple doesn't share any information on that. We'll see.

OK, what are the tool calls that you've already implemented or tried? And can I create a custom one? Sorry? So what are the tool calls? Tool calls, yeah. So, for our example app in the repo that you can download and play over it. As I mentioned, we've built this kind of calendar tool calls. So you can read your today's schedule. You can create even in calendars. You can access any native API that is accessible on iOS or Android. So you can access battery level, you can access your notifications. You can send notifications to, you can schedule alarms, any API that you have on the native iOS. And also you can do some API requests, whatever you like. You can execute any JavaScript code there. Cool.

And does the model have access to data on the device? Think photos, files, sign up? Yes, if you create like proper and relevant tool call, like for example, get photos and user grants proper permission, it'll work. Cool. And can you supply a system prompt to the Apple Foundation model? Yeah, go ahead. Of course. Oh, you can? Okay. Go ahead. Yeah, Apple supports like a system prompt to calling structure outputs. So they really cooked on this API on this website. Nice.

Is it possible to train these models more or give them a predefined context for your specific use case? For the on-device models, I'm not really sure. Like you can provide system prompts, but I don't know if you can post train them. If you use like custom engine that we also support MLCLM, you can like if you are a bigger company and you can train your own model, you can also use it. And yeah, like with the universal engine, you can do a lot of stuff. It allows for much more, but that like the other side of this is that, like downside of it is that users need to download this model. It's not shipped with the OS. So if you want to have like custom model, train for your needs, you need to download it and every user needs to download it. But if you use the on-device model, like it's shipped with the operating system. OK. And do you think on-device LLM can have enough compute to generate images, videos, 3D outputs? Yes.

So we don't support currently like generating images, videos for React Native AI, but I've seen a lot of demos of like the API from Android, for example, allows to generate images. I'm not sure about videos, but, you know, like the current API, like the current AI that we all use is the worst that we will use, like AI is updating frequently and with every, you know, every month there is a new better model. Every year there is, you know, breakthrough in AI. So I'm really, you know, I really think that in future months there will be a possibility to do like image generation, video generation and all kinds of things. Really exciting things for the future.

Do you have any idea on the param size for these device models and how do they compare with other open source models? Yeah, so there are some benchmarks. I don't remember the numbers, but after playing with it and just using the example app and we're implementing this API, it's a really cool model and with like, this model is profiled like that. I'm mostly saying about the Apple model that is shipped with OS because I played with it the most. It's profiled for the for the like latest iPhones. So it uses proper, you know, metal, proper, you know, proper amount of CPU's and GPU and, you know, like Apple engineers spent a lot of time.

But, you know, general purpose models, which is also open source, some of them are not profiled for each device. You know, there are general models, so they are not using, sometimes they are not using like all the resources properly, not allocating proper memory usage and kind of these things. So, yeah, Apple model is really fast and it's pretty smart as for the on device. And what does this mean for other AI apps? I'm speaking of Gemini, Jacky P.T., Claude Sonnet, things like that. Yeah, I really doubt that they will implement like on device. To be honest, like, you know, Jacky P.T. to play with like latest open AI models, same for Gemini, same for any other app that we use on a daily basis. But I really see the use case for on device and AI for privacy apps, for offline apps, for instance.

You can also use, for example, if some app uses models from the cloud, but there is no internet connection, we can fallback to maybe a slightly less optimal model on the device. Implementing on-device models for such scenarios is crucial for various use cases. The library can have context awareness of the React Native component rendered if passed in the system.

MLC is not the only runtime; there is a potential plan to support MLX and other runtimes in React Native AI. Memory consumption differs significantly between built-in and downloaded models. Downloaded models can consume considerable RAM, causing performance issues, especially on Android devices. In contrast, models shipped with the OS are more optimized and consume significantly less memory.

Whisper.RN is geared towards transcription, while LLM.RN serves other purposes. Both tools are being considered for support in React Native AI and Versalia SDK. Checking device support, like for Pixel 9, is essential due to the vast array of devices available. APIs enable developers to verify device compatibility and adjust logic accordingly for a seamless user experience.