Leveraging Data Flow Programming for Scalable and Efficient AI Systems in Distributed Environments

So today, I'm going to talk about Graphi. It's spelled Graph AI, but I call it Graphi, which is the declarative dataflow programming environment for AI Asia.

So the traditional applications are typically written, some kind of a control application on top of a database, but because of the rapid advancement in AI, the way we build application is changing quite In addition to database, we use a lot of neural network-based models like large language model or image generation model, image recognition model.

So those, the models are typically available as either an API on the remote server or sometimes on the machine itself. The challenge of those, the AI module, is they are very slow. Unlike typical databases, some of the API calls takes like 20 seconds, 30 seconds. So definitely we have to use a synchronous API, right? It's not suitable to use a thread-blocking synchronous API, but I think the industry is still not recognizing the importance of asynchronous APIs and also deep programming environment. The good example is the Python API of OpenAI SDK is synchronous, which is unbelievable from our point of view, right? Our typical JavaScript programmers, nobody wants to call synchronous API, which takes 20 seconds to block the thread, but that's how people are programming today in Python.

Luckily, the JavaScript version of OpenAPI is asynchronous, so we can take advantage of it, but still dealing with multiple asynchronous calls is difficult. The application is changing from simple client-server to more distributed, which introduces challenges in writing efficient programs. My proposal is to shift from traditional imperative control flow programming to declarative dataflow programming. This programming style allows us to call multiple asynchronous API calls using async and await, making the code easier to write and read. However, executing everything sequentially can be inefficient. To solve this, we can use Promise.all to execute independent functions concurrently. But there is still room for optimization, as some functions can be executed even before others finish. The complexity of dealing with asynchronous dependencies can be messy and difficult to maintain. The solution is to go fully into Dataflow programming, which describes the dependencies of each function call in a declarative manner.

And some of them are one second, some of them are 20 seconds. So this type of programming, using the async and await, which we are familiar with, and we love it because it's a lot easier compared to a function call mechanism. And we can write the code as if they are synchronous.

But the problem is this approach, we ended up executing everything sequentially, right? Even though function A, B, C are independent. So we could execute them concurrently. This programming style execute them sequentially. So the total amount of time required to execute this function, that this operation, is the sum of each API calls.

So in order to solve that problem, we know we can use promise all to execute function A, B, C concurrently. So we actually reduce the total amount of time required. But again, this is not optimum yet because the function D and E are independent. So we can add yet another promise all to execute D and E concurrently. So much, much better than the sequentially everything, but still not optimum, right? The function D actually doesn't need to wait for the C to finish. So we could actually start D immediately after A and B are done, so that the total amount of time reduces further.

But the thing about, this is the quiz, right? How to write this code? Very complex. This is just only a sixth function calls. But dealing with this asynchronous dependency code became really messy, right? You can combine the promise all, async await, plus the function call. We could write that code. But that kind of code is really difficult to read, difficult to write, and difficult to maintain. We can easily introduce some bugs and a side effect so hard. I think the only solution is we go fully to Dataflow programming, which described the dependencies of each function call, but also declarative. The declarative meaning we don't need to invent a new language. We can use either JSON or YAML.

So here is how we describe this program in Graphi. So the left side is just a code, right? It's simply defined a graph, right? With a six node from A to F, and describe, right? That node A needs to function call function A, node B needs to call function B, and A, B, C are independent. They can just execute any time. And node D, E, F has some input, right? D has input from A and B, E has input from B and C. So the program will just easily describe those dependencies, right? The Dataflows. And then the runtime, in this case, the Graph AI runtime figured out when, how to execute. So the programmer doesn't need to explicitly say D needs to rerun R, S, Q, A, and B, it's just Dataflow. Then let's go to the actual AI programming.

So for example, when we want to call open in AI, we built a small function called the open in AI agent, which takes the prompt from the user input and then call open in AI API and return the result as a text property. So simply three node, get the user input, give it to LL, and then display the output.

And this is just one Q&A. So in order to turn into chat program, we need to have some kind of a loop, but the challenge is the Dataflow is a cyclic. You cannot have an arrow going back. So in order to solve that problem, we came up with the idea of the concept of a loop inside a graph. So you can specify this graph need to be executed three times, or sometimes the execute until a certain condition is met, just like the while loop. Then we can execute this, a cyclic graph with multiple times. And this is simply, we just execute the same code three times.

So in order to turn into the real conversation, we need to accumulate messages from the user and LLM into an array, and then give it back to LLM. So the entire code looks like this. A little bit complex, but it's not that complex. In order to show you how it works, I want to show you the demo. So the same graph is now running on the browser, right? As I said, the graph is implemented in PipeScript. So it can run on the server side or client side, and this is running inside the browser. And because of the graph, we can easily get some messages from those graphs to indicate the state of the graph. So right now, it is actually waiting for, the user input node is waiting for the user input. So green indicates it's active.

So I'm going to type in what is the data? Then I hit submit, then you will see actually the data flows and execute. Okay. Right? So when the user input, actually, two nodes, the check input receives the user input and then checks if the user wants to terminate. And in this case, it simply checks if I type in back by, and then the continue flag turns into false. But the default is continue. The another node, the user message node, received the user input, then append it to messages, the array of messages, which is originally empty, append it and then give it to LLM. So LLM receives just the one message at the beginning, execute, and then generate output. The output will be displayed by the output node, but also this reducer received that message and then append it to a messages array. And this dotted array, the arrow indicate sort of the special case only allowed in the loop case to data back to the source node. I'm going to type in one more.

And this dotted array, the arrow indicate sort of the special case only allowed in the loop case to data back to the source node. I'm going to type in one more. So I'm going to say, what is the method? And because we are appending the messages, so it knows what it means by it means the data flow programming. So the LLM says the advantage of data programming, like parallelism, the resultability, the scalability. So I didn't make a presentation, and then I simply add the end, I type by, then the LLM responds to me say goodbye, but because the checkpoint indicate I want to end the conversation, then this continue flag become false, therefore the execution of graph is ended. So this is how chat application inside graphi.

And graphi, because as I said that the typical graph the data flow programming, that because the graph needs to be a cyclic, we needed to introduce a few new concepts to make it convenient to write a complex program. So this is another example, the first we showed the loop, but this is another example, it's a sub graph. So in the big, when you started writing a large graph, then it became complex.

Sometimes you want to create a sort of the nested graph, so that we can use that portion, just like a sub routine or function. So you can have a graph inside a graph, and from the external point of view, the outer graph is point of view, it's like a special node, which inside execute a sub graph. But from the parent now from point of view, it's simply receive the data from another node on the parent side, execute some operations asynchronously, and then return the output to the parent. So this is the enables the reusable code, also make it easier to read the code.

Another one is the map reduce. So when we are reading with an array of data, like a benchmark, right, the benchmark, the LLM data benchmark has a large set of questions and answers. And we need to throw it to LLM to see if it responds correctly. And running that code is again complex. If you try to do concurrently, so today, many people just do synchronous, but using a graph, you can simply give an array of data to this map agent, we call it the map agent, the map agent, recognize that array, and execute the sub graph on each item. And then of course, the concurrently.

Another complexity comes when the graph becomes large and too heavy to execute on one machine. We can actually specify a certain portion of the graph, typically a sub graph, to be executed on the remote server. Again, it's all part of the graph extension. So you don't need to worry about how to remote execution. You simply need to specify this subset of graph need to be executed on this URL, then the graph extension will figure out and then execute. And on the parent point of view, the parent graph's point of view, it's simply just a node, right, which perform asynchronous execution. But internally, that node send the sub graph to the remote server, remote server execute concurrently, and they return the result back to that node, that node, then give the result back to the parent node. So this is how the distributed computing works inside a graph. And as I said, because the graph is implemented in pipes route, so you can run on either client side or server side or anywhere. So we can actually write a complex program to and then execute on multiple machines. In certain cases, we recognize that the like a special database need to be accessed only from a particular server.

So in that case, we give a sub graph to that server to execute on behalf of the outer server, which doesn't have access to the database, that does server execute that grass, get the result, and then only the result coming back to the parent server. So that's how we execute distributed graph. So this is the graph I, and this is new, but we open sourced.

So in that case, we give a sub graph to that server to execute on behalf of the outer server, which doesn't have access to the database, that does server execute that grass, get the result, and then only the result coming back to the parent server. So that's how we execute distributed graph.

So this is the graph I, and this is new, but we open sourced. So it's available on both GitHub and also a node package manager. And it's a fully MIT license. So anybody can access it, use it today. We actively working to improve. Actually, the engine is well done, I think very small. And we are adding multiple capabilities in extensions, like a sub graph or remote graph or maps, reduce, are all extensions. So we can easily add the features to the graph I using that mechanism.

So we are looking for other people to participate and help us. We need more improvement in tutorials and samples, but we are very open. Thank you for listening.