Optimising Images in Web and Native

Hi, my name is Kati Cremmann. I'm the lead developer of ReactJS. And this is a speech from our new guest speaker, Kati. Hello! Thank you. You are my true friends. Thank you, everyone, who stayed just before lunch. I will try to keep it brief. Thank you for that intro, Janne. It was amazing, as always.

So, hi, my name's Kati Cremmann. I'm the director of engineering for mobile at the moment at Promotable. We're an amazing JavaScript consultancy. We build things in React and React Native. And a lot of these things include images, which brings us to the topic of my talk, which is about optimizing images for mobile and web.

So, what exactly do I mean by optimizing images? You can kind of think of it in two ways. One way is around optimizing the content of the image. So, optimizing the cropping of the image. Are you going to use portrait or landscape? Where do you want to fit the subject of the image so it fits best the position you have on the screen? So, to give you an example, so, on the left, we have an original image, which is in portrait mode. And say that our goal was to use it as a hero image on a website or maybe like a banner image on Twitter. This is not an optimal image for this use case, because, as you can see, there isn't really a good way to crop it, so we end up with half a child. But this talk is not about cropping. If it was about cropping, the TLDR at the end would be to use an aspect ratio. So, as developers, we are not usually in charge of actually taking the images and positioning the images. That's for a job for the experts. But the way that we communicate what we need, the positions that we have on the screens, are using an aspect ratio. So, an aspect ratio is a ratio between the width and the height of an image. So, rather than saying, I need an image that's 750 by 500, you can tell your designer or a photographer that you need an image with an aspect ratio of three to two, and they can then give you the highest quality image they're able.

The other way to look at images is to look at the file size. So, this is something that we are more used to optimizing, because we are well aware that when we are rendering a mobile app or rendering a website, then the user has to download all the assets from whatever server before we can display them. So, we want to have these assets, images, JavaScript bundles, CSS, as small as possible.

And the other thing I want to talk about is the image dimensions. So, specifically, the width and the height of the image, which also matters. And I've added a question mark here that's because you might not realize just how much it matters.

When I was talking about optimizing images yesterday with a couple of people, a lot of people wondered how come we're still solving an image optimizing problem? People were trying to optimize images ten years ago. When I was talking about the topic of my talk, we kind of came to the conclusion that people have been doing this similar kind of thing for years and years and years. But the problem is that as technology advances, the problem we are solving keeps getting harder.

So to give you an example, this is an image that my friend Taz took of me a couple of weeks ago. Taz is a very good camera, top of the range. And it's great. I can zoom in, go all CSI, and figure out exactly what time it was when this image was taken. However, this kind of resolution comes with the cost of file size. So if I dig into the metadata of the file, I see that this image alone is 21 megabytes. And it's 8,000 by 5,000 pixels. So this is the source image that your photographer these days with a good camera is going to give you to use on your websites and mobile apps. And obviously you can't plop that in straight away because that is not going to give a good user experience. So if I wanted to use this on my website, I might reduce the dimensions, run it through the trustworthy Panda at TinyPNG, and I might end up with something much smaller. So this is 100 megabytes. Still not ideal, but it's usable. And I've reduced the image dimensions. And you notice that I've mentioned image dimensions several times. And you might be wondering why it's even important. And I'm going to tell you a story, and this is actually the story that prompted this talk.

And this is from a couple of years ago now, quite a few years ago, when I was building a mobile app. The mobile app was ready. It wasn't live yet, it wasn't live to users. But it was ready for content managers to add content and test things out to make sure everything was ready before releasing it. And the actual app doesn't really matter for the purposes of this story. But all you need to know is that it was very image heavy. The whole app, like every single page, had lots of really professionally shot custom images in various configurations. And what we found is that the content managers came back to us, and they kept saying that after a couple of pages, the app keeps crashing.

Not just after hours of use. It would literally be after five or six pages. And immediately, we were asking, oh, what device are you using? Is it a low end Android device? Because as React native developers, we tend to blame everything on low end Android devices. Unfortunately, in this case, they were all pretty much using the latest iPhones. So, we couldn't blame Android. But we were obviously thinking, if the iPhones can't do it, Android phones have no chance. So, we have to solve this.

So, what we did is, we, well, basically commented out code until we figured out what keeps breaking the app. And we found that if you comment out all the images, all the performance problems magically go away. This is obviously great, but not a good go to market strategy for an application that's all about images. So, then my friend console logged to the rescue. And what I did is, I just console logged the sizes of all the images that were being sent from the API. And I actually managed to find the actual screen shots I took at the time. And this is basically what we arrived at. This is just opening the app and navigating to a couple of pages. And it's kind of a mess, but I've highlighted some of the biggest culprits. And as you can see, based on the image that I showed before, these are clearly the source images that their photographers gave them as a source, as the beginning image, rather than cropped versions. And what you might not know, especially if you come from the web, is that rendering images in native apps is more expensive. So the way that rendering happens in native is that we calculate the size of the image, the area that you have available, and then we convert the image to fit that size. And this is done in memory. And the larger the difference between the image and the size that we're rendering it in, the more expensive the calculation. And obviously, if you're doing tons and tons of these expensive in-memory calculations all at the same time, you're going to run out of memory and your app's going to crash. So to fix this, the straightforward solution is to size the image based on what you actually need. So I'm going to talk you through how I would go about it. We're going to use this beautiful green image throughout. I'm going to give myself two constraints. We're going to use the screen width as the width of the image. And we're going to use an aspect ratio of 3.2 for the height. This is React Native code. We have a handy hook called use Window Dimensions.

You can also use dimensions directly as an export, but using a hook obviously means that you're running it on the web. Someone resizes the browser, you get the new value. I'm using a screen width from using window dimensions, and I'm calculating the relative height based on the width. And in the third arrow, I'm passing it straight into the image.

Now, this is a screen shot from my phone. And I've also done a crop so you can see it a bit, the quality of the image a bit better. And I printed out the width and the height that I get from React Native. So it's telling me that my width is 390 and my height is 260. So I go away, I take this image, and I crop it to 390 by 260. And I end up with this.

I'm not sure how well you can see it, but the image on the left is really blurry. But why? I've just taken the dimensions, exactly what I need from my phone, and I'll crop my image. So why is the image blurry? Is there anyone here that can tell me? Yes. Yes. Correct. Pixel density. I have a whole slide for it. Yeah, that is spot on. And if you're coming from the web, especially, that's not something that you usually have to think about. But on mobile, it is very important. So pixel density basically tells you how many... Sorry. Let me reverse it a little bit. So when you come from the web, especially, and you're starting writing styles, you might be thinking, oh, I no longer need to write pixels, or Ms, or Rems, after the style numbers. And the reason you don't have to write those is because you're not talking in pixels and not talking in Ms. You're talking in display points. And a display point may equal a pixel, but it may not. And pixel density basically tells you how many pixels does it take to make up a display point. So one-to-one, so all the iPhones would have used this, would be one pixel is one display point. However, most iPhones these days will use a pixel density of 3, which means that one point is actually nine pixels.

So, and if you ever had, for example, added an app icon, you would have been very familiar with this page, where you have to upload each image in 1x1x2x3. The reason you do that is to account for the pixel densities of the different iPhones. You'll have a similar thing on Android, but I think you have about six different versions to account for. And this is basically the formula. It's pretty straightforward. The size of the image in pixels is the size of the area in display points times the pixel ratio. There is an export in React Native, so you can import pixel ratio and then get the pixel ratio of the device or website you're currently on. And there's some examples. I just took the screenshot from the React Native website. We can see that the pixel ratio tends to vary between one and 3.5. It doesn't usually go higher, but knowing technology, it probably will eventually. And now, if I multiply the display points with the pixel ratio, I'll get the actual image that I needed, which was 1170 by 780. And now this will look crisp for my screen.

Now, the problem is not everyone uses the same phone as me. I wish they did. That would make my life much simpler. However, even if you just look at iPhones, there are tons and tons of different sizes. So here, if I compare an iPhone 6 and an iPhone 12 Pro Max, you can see that the same image using the width of the screen and a 3 to ratio will be wildly different.

So if you only have one source of truth image that you're going to upload, you might be tempted to just upload the image in 1290 by 860 and everyone else will have to scale. The problem with that is that now your lower end devices that are already struggling for the ones bearing the burden of having to convert this image to the size you have available. Thankfully, this is a problem that has been solved. If you're using a CDN, pretty much all modern CDNs will have this functionality or you can build your own where basically the device calculates the area that they need, and they ask the image in specific size. So, your server would go, oh, you want 750 by 500, I've got this in the cache, there you go. Or if they don't have it, they would calculate it on the fly, save it in the cache for the next time, and return it.

The problem with this solution is that there is still too many different images. So, the first time, firstly, the first time we generate the image is going to have a couple seconds of cost where the image is being generated, so the user has to wait. And secondly, if you're using a managed service, it gets really expensive because most of these services will charge you based on transformations, so you're much cheaper, much better off using the cache if you can. So, the way to get around that is usually you'll end up having an array of supported sizes. So, I've gone for quite a few here. So, I've gone for like 15, but you will have supported stops where you say that I'm going to support an image that's 600 width or 1,200 width. And so, you calculate the width that you need the image in and you'll take the next best image, which means that you're still not having to rent anything too difficult, but you're still getting the optimization of not having to do so many on the fly transformations.

All right. I've talked a lot about this. I did do a demo as well. So, you can try it out. So, this is something that's the most fun, I think, if you can try it on your phone. I wasn't expecting this to be quite so big. Oh wow. Okay. I'm not sure I can do this. I guess you can see the bits on the right, which is the important part. So, the code is on the left. If you can't see it, just trust me.

It's there. But, I've got three examples here. So, on the right, I'm rendering it in React Native Web at the moment. But if you go here, and you have the Expo Snack app. I'm going to put this slide with the QR code up later. You can just run it on your phone, and you can see how these images run on your own phone. But on the web, so I'm rendering three different images.

So, I've got the full size image in all its glory. Then, I've got... So, at the top here, you can see what the pixel ratio is. And here, we've got the pixel ratio of 1, thank God. But if I'm running it just normally on my display, it's a pixel ratio of 2. You can see the screen width is 333, and you can see this is the full-size image. This is the exact size image. This one, on your phone, if you try it, it's going to look blurry. This is the exact size image with the pixel ratio. And this is the exact size image with the particular sizes. So, for example, here, you can see that the width in pixels that we're looking for the image for is 333, but we go for the next best width, which is 350. Let me just go back to my slides. There you go. This is the slides to take a picture of, if you want to try it out. and you can just try it out on your phone, and you can see how the same image renders to you versus your friends. Hopefully, you found it useful, and thank you very much.

So, firstly, in this solution, you have this array of fixed sizes, so you actually generate all of those images during the build time? Yes. So, you can choose. Depending on your service as well, you can create the images in advance. If, for example, you don't want to use a service at all, you can just decide you're going to use these three images, you're going to generate them in advance, and you know that your API is always going to support them. But the more optimal solution would be to create them on the fly. But in your case, can you recommend any tools, for example, bundlers, or things that do this image optimization for you, or is that exercise left for the reader? Personally, I like using Cloudinary, but they can be quite expensive. So, I think it's also the tools that you have available.

If you already have an image CDN, then they will probably do it for you. Nice.

Here's the next question. Is there a proper way to size images without having to choose the next best version, requiring you to have an array of sizes at the ready? So, basically, if you don't do this fixed size kind of thing, what is the best way to go about it? So, it would be the original method that I showed, which is exactly the size for the screen. So, you will get the best size always for the area that's available, but it could be slower if you don't have that many users with that device. So, if you have one user on an iPhone X and 10,000 users on an iPhone 12, the iPhone X user will always, whenever they go on the page, the server will have to generate those images, whereas the iPhone 12 users will be all using the same cache. Nice.

And the other way, of course, to optimize image loading is to choose an appropriate image format for the type of image. Have you played around with these modern image formats and how those sort of effects solve the same problem? I think, to be honest, image is one of those things where I tend to pay to use the good tool for the job. For example, not that I'm advertising them in particular, but if you were to use Cloudinary, they have a parameter called format auto, which will serve the best format for the platform that you're requesting from. If you go on Chrome versus native iPhone versus native Android, it actually sends the same image in a different file format. Yeah, but I like the point about splurging on tools a little bit. This is not something that you want to try to solve for yourself if you do have the budget for it. Yeah, absolutely. I mean, it's like time versus money, right? In this scenario, I feel like my time is more important than some people's money. Yes. Image is nothing. Thirst is everything. Exactly.

Okay, you're thirsty. All right. Thank you, Cady. Speaking of thirst and hunger, we'll let people go for lunch, but you can find Cady at the speaker Q&A room after this. Thank you so much, Cady, for joining us. Awesome. Thank you, Yanni. I appreciate it.