JavaScript File Handling Like a Pro: From ZIP Streams to Memory-Efficient Parsing

When thinking about Node.js, most developers will think of a rest API or some kind of web server. They hardly think about working with files. But the truth is, Node.js has some serious capabilities when it comes to working with files. You just need to know how to do it in the right way.

For those of you who don't know me, my name is Noam. I'm a full-time developer with over four years of experience. I've been part of large-scale production teams and also as the main architect of projects that are built from scratch. This is my family. My wife just graduated medical school, which works great for me because I'm a hypochondriac, so it works good. And in the past year, we've become parents to our lovely little daughter. So it's kind of my obligation now to tell unfunny dad jokes, so sorry in advance.

Why does it matter? Let me share you a little bit of a story of how I was introduced into Streams. So my manager gave me a task to build an app. It was quite a simple one. The app was basically need to get data from a client, storing it in some kind of database, and later on to create from it file and send it to a remote server. And it needs to be doing the other way around. So to get files from a remote server, extract the data, and then send it in a JSON to the client. So I've built just a masterpiece, a beautiful app. I've done testing, I've done classes, everything was really cool. And I managed to finish the task half of the time that they gave me. So that evening, I came on to my wife, and I said to her that she can pick the fancy restaurant that we always cannot afford because I'm going to get a raise. One thing led to another, two minutes later, my boss called me and tell me that my app was crashed on the very first day. So after I tried to figure out what happened, I realized that in my testing, I was testing files of 1 to 10 megabytes. But in the actual production, the files that I was getting was 500 megabytes or even gigabyte. So this is what made my app crash because I used only the naive so called way of read and write file. So what is actually the problem with file handling? The main problem is that it loads the entire file into memory. So this is Mr. Beans still waiting. You must wait until the whole file loads into the memory until you can use it, and it can cause some serious problem as we can see in the next slides. So I warn you about the dead jokes.

Ignore your capabilities. I'm not saying that streams are always better than the naive way. I'm just saying that you need to expand your toolbox in order to work with the file the right way because when you limit yourself just to the naive way or just to streams, you limit your code, you limit you as a developer. And this is the best case. You can cause serious problems of security, app crash, et cetera, et cetera.

Just to get everybody on the same page, I don't want to waste valuable time about streams, but just the main concept that you need to be familiar with in order to understand the rest of the talk. So streams are basically interface working with data piece by piece instead of all at once. We have readable stream, which is the source. We have writable stream. You can think of it as a destination stream. We have duplex, which is both way, and transform, which is kind of middleware, you can think of it.

Streaming OJS are event-based. So you don't need to wait for the file to fully load, and this is the big, one of the big advantages that you can react to small events as data flows. Two more main concepts that we need to understand in order to understand streams are a pipe and pipeline. A pipe is basically like a pipe connecting between two streams. One of the crucial concepts in order to understand streams and why it's so powerful and important is backpressure. Backpressure is basically when our consumer is slower than the producer. You can think of it like when a baby is drinking milk, but the milk flows faster from the bottle than the baby can hold.

So a pipe is basically like a pipe connecting between two streams. So we have the source stream and pipe connected to a destination stream. And we have one of the crucial concepts in order to understand streams and why it's so powerful and important, it's backpressure. And it's somehow getting swept under the rug many times. So backpressure is basically when our consumer is slower than the producer. You can think of it like when a baby is drinking milk, but the milk flows faster from the bottle than the baby can hold.

So actually, I have a visualization of it. So this is my baby. And although you don't cry over spilled milk, you actually do cry if the milk is spilled on your new computer. So this is basically backpressure, and we show it, examine it later in detail. What is the big difference between streams versus naive? And this really blew my mind when I was first introduced to it. In both of them, we have the file on the disk. But in the naive way, we load the entire file into memory, and then we process the data. And in streams, we read it chunk by chunk and process the data as it arrives, which gives us the ability to keep our memory stable and flat. And if we're talking about efficiency and not just speed, it makes our app way more efficient, clean, and bulletproof.

You can think of it like Netflix. If you want to think of when I watch a movie on Netflix, and you need to download the entire movie before you start watching, it takes a lot of memory. You wait forever before seeing the first time, and it completely blocks until it's done. But if you will do it with streams, so we send small chunk of data one after another while you're already watching, and it starts immediately, the memory stays flat and low. And if maybe for some reason the network or disk is slow, it just buffers small parts instead of freezing everything out. So this is actually the big advantage of streams. So let's see how it solves all of the problems that we have with the naive way. So with read file, it's loading the entire file into memory, and we're using the readable stream. It reads stream chunk by chunk, one chunk at a time, and the default is 64 KB, but of course you can configure it on yourself, and it starts emitting the data immediately. One of the big advantages of the readable stream is that it plays well with other concurrent. Even though that read file, not like read file sync, is not blocking the event loop, it can cause some serious lateness, and makes our life really difficult when trying to manage concurrency. One more thing that we have, slow start processing time. Let's say you have a CSV file, and what you are looking for, it's in the second line. But you have one million lines on it. So in streams, you can just stop the chunk of the second line, but if you use the naive way, you need to load the entire one million lines into memory before you can access it.

So it's actually really, really inefficient. And the last thing, I talked to you about the back pressure. Streams actually implement automatically back pressure. Later on, I'll show you an example of code, how we can do it, but you don't need to worry about it. So this is what makes streams way more secure, efficient, and cleaner, if we're talking about large files. It prevents the crashes and keeps memory usage stable, which is the real magic.

Okay, so I will go over the code briefly because I don't want to bore you, to make you bored. And I want to go over all of the main concepts, which is a broad subject. So you can see the code in details if you want later on, on my Git. I will have a repository for it and maybe with some more examples. So here I basically have two identical codes, I have a memory usage logger function, and the only difference is that I'm reading files with the naive way through loop, and here I do it with readable stream. And you can see, by the way, this is the way to make a loop over streams and synchronize. So every two seconds, I'm sending the memory usage to some kind of HTML code, which I'll show you now.

So as you can see, when we do it in small file, okay, so I have a small file path here, let's do the small file path here. So this is a small file of, I think, five megabytes or something like that. So when we start running the code, you can see it's logging the memory that it uses, here are the chunks, here is, yeah, four megabytes at a time, which is the whole memory. And let me show you a graph of the use of it. Okay, and both of them, it's kind of stay flat with a little movement because of the garbage collector. But our memory are basically the same, which you can see, actually streams takes a little bit more memory with the small file. But the amazing part is that if we will do it with larger file, and this was the main difference in my project.

Let's do the large file here, and large file here. Now, when I'm going to run the code, you will see the magic, which just blew my mind. Now, if we go to the graph and refresh, you can see how the memory of streams just second. Yeah, to save, it's kind of important. Now, I switched to the large file path. And as you can see, after I will run the code, if we go to the graph of the memory usage, and we refresh the page, you can see how the streams stay flat. And it's load into memory. You can see how the memory here is go spike and go to 400,000 while in streams. It's react for one gigabyte file as it reacts to five megabyte file. So we can predict our code very carefully. When you can see here, how it's slow way too much memory. And this is the main difference. This is the big power of streams. Because most people, by the way, doesn't know it. But streams initially came to solve the problem what we need to do if we have more, we have data with more memory than we actually have on our process. So it's actually slow it down, wait until we have the memory, and then flow and send the next chunk of data. So this is a really amazing part, at least for my opinion. And this is what make my app really better and improved it and fixed all my app problems. In my case, and I want to share it with you.

After I show you that, let me go back to the presentation. Okay, now I want to go over production proof tips. For these tips, what helped me create way more efficient apps, as I said, because it's a broad subject. And I want to show you demonstrate as many examples as I can. So I'll go briefly over the code, but you have all the code and details in my GitHub repo. So let's go to our examples. Or as I say to my daughter, when we need to change a diaper, let's get our hands dirty. I know I need to stop doing that. Okay, so I want to start by showing you the big thing that backpressure can do. Here I demonstrate when the producer is much faster than the consumer. So I'll configure it a little bit so you can see what can happen.

Let's say we have way more than 64 kilobytes chunk, so it's really the data that flows goes faster. So we have a bit of class of fast producer to send a readable. I use the read method and I made the consumer really slow. So here we have the slower consumer. Here I want to again show you the memory usage of that. So when I run it manually, sorry, we have run manually and when I run in the bed way, so this is the bed way when we don't handle backpressure. So after I run it, you will see how much memory it takes and how it's not efficient because if we not handling backpressure, so we're kind of canceling all the effect that strings had. But if we run it on the manual mode, which do handle the backpressure, you can see how the memory stays flat and low. And this is pretty amazing because we're talking about the same file, the same amount of memory and data. But look what a backpressure can do to our app if we handle it correctly, instead of using all this memory.

And if you want to take your apps from fine and maybe work, maybe not, and unpredictable, it's okay maybe for small apps. But if you want to take it to the next level, which what was happening to me, look at the main difference. And if we're talking about efficiency, this is just a game changer, at least for me. So let's go through one more example. As I said, if I have a CSV file, all right, so here is the naive way when I try to find, let's say I have a CSV file with a lot of ID and users. And I want to find, let's say, ID number 10, because it's New York and we want to respect the money from the giants, or for the Jetfans. We want to say, let's say, the user ID of 10. So if we have millions of users in the naive way, we need to load the whole file while with the streams, we can stop after we find what we needed.

So let's say we have a small CSV file of, I don't know, like five megabytes, I think. And we run this example. You can see how the process of streams stopped after 12 lines with nine millisecond running. In the naive way, it's a total of 2,000 lines into memory, but 15 milliseconds, not so bad. But when the file path is a much larger file, I think here we have a 15 megabyte file, I think. And we save and run it. You can see now, this was taking 13 milliseconds. And here, 77. And you can see how the memory goes and grows in the naive way, and how it keeps stable, because after we found what we needed, we can access it from the code. And if we do one gigabyte file, and the amazing part, that our app crashed because we used more memory than what we actually had. So this is also a very useful way of how to do it. And the last thing I want to show you is the difference between pipe and pipeline.

90% of the times, you will want to use pipeline because the problem with pipe, which is connect the streams, that it doesn't forward errors. So you can think of it, if you have a lot of streams connecting with pipes, and you have a leak in one of them, but you don't know that there is a leak, you think everything works great until you find out a very big problem. But with pipeline, you can think of it as a system that tells you there is a leak, shut down what you need, and everything else continues to work incorrectly.

Just to show you the difference, here we have a function when I'm using pipe, and we have a writable stream, some kind of transform stream, which by the way, Zlib is a transform stream, and we have writable streams. So say here we have input file that I know exists, and here I'm creating an error when I'm using missing file that we have. And when I run the code, as you can see, our app crashed because it doesn't know how to handle the error, it doesn't forward the error, and we don't have error handling, and our app will crash.

But if we will do it the other way around, and here we have our existing file, and in the pipeline, we'll have the same error. You can see this is the same file, I use one for the demo pipe and one for the demo pipeline, and we do it with pipelines. So it's still, we have an error, but it's handled our error gracefully and can help us maintain scalable and large-able projects. This is another example of how to unzip stream when we're using the unzip your package as the transform stream, which is kind of cool.