Why Writing a Pinch-to-Zoom Component Is Harder Than You Think

Hello everyone. Thank you very much for coming to my talk on why writing a pinch-to-zoom component is a lot harder than you think it is. So let me first give a quick introduction of myself. I am Oli, and I work on the AgCharts team, which is basically just a front-end library for rendering data visualization charts, such as pie charts, bar charts, line charts, those sort of things.

And we've got lots of different features. One of the key features is that our charts are interactive. So you can obviously click it with the mouse. You can also use it with a keyboard if you have accessibility needs. And obviously, you can also use it with touch input if you happen to be on a mobile device.

And so a couple of months ago, I had a feature request coming in that we wanted to add support for not just one finger, but two fingers. And broadly speaking, this is the problem that we wanted to solve. We wanted some sort of, you know, zooming and panning features. And so I'm sure you're all kind of familiar with this. You probably use something similar, like Google Maps. But essentially, we wanted a panning feature, where the user could put two fingers, move those around and adjust the pan of the viewport. And something similar with zooming, where the user could put down two fingers, spread them apart, bring them close together to adjust their scaling.

So when this feature request came in to me, obviously, me being the good programmer that I am and all, I did what any good programmer would do and I said to ChatGPT, how do I do this thing? And, you know, ChatGPT said, oh, you do the thing like this. Okay? So this is what I like to call the classical solution. So basically, the way that it works is when you've got two fingers, they've both got X and Y coordinates, and you use those two touches to calculate two different variables. One variable will be sort of like the center, or you can think of it like the average X and Y positioning. You track that average, and as that changes, you can adjust the pan with that. Then the second thing you calculate is the distance between those two touches. You can calculate that using the Pythagoras theorem, and as that changes, you can adjust the zoom. And so that actually works for most use cases. So here, as you can see, an example on the left-hand side. So that's, you know, a zoom pan feature that I'm sure you're all very familiar with. As the user pans up and down or left and right, it changes the zoom, but critically, it keeps the X and Y-axis scaling values in proportion with one another, which is good. That's actually what you want, because you don't want to disorder the image. You still want it to kind of look like Manhattan Island. You don't want it to look like some other weird shape. But here's the thing. I thought, with a chart, wouldn't it be nice if we could disorder the image, because the user, as you can see, may want to adjust the X-axis or just the Y-axis, or they may want to have some more fine-tuned control as to what data they view. So obviously, the classical solution is not going to work for us, because we want independent X and Y-axis.

So then, you know, me, a good programmer, I go back to chat.gpt, and I'm like, oh, no, no, no, no. I want the X and Y-axis to be independent. And then, chat.gpt, obviously, is like, oh, that's a great idea. That can really improve the user experience if the X and Y-axis is a very enthusiastic guy. Then he gave me a solution, and it didn't work. And I was like, chat.gpt, I need a solution that works. He's like, oh, yes, I understand. It's very frustrating when the solution doesn't work. Here's a solution. Another solution. Doesn't work. Doesn't work. I did that, like, five, six times, and eventually I was like, OK, shut up.

Let me solve the problem myself. So I kind of thought, OK, so let's try to forget about the code for the time being and try to think from the user's perspective. How do we design a good user experience with touch? So I did some research, and I came across this really useful guideline written by Android. It's very good. I recommend that you read it if you're interested. But essentially, it explains that you have to have something called a realistic touch UX. And so as you can see, the good example there on the left-hand side, as the touches move, the UI components move perfectly well in sync with that. Whereas the bad example, you can see the UI components are lagging behind, or they're moving too quickly or too fast. And so that's bad. That's bad. We don't want that. So how do we do that on the chart?

OK, let's look at a use case. So for the panning, let's say, for example, the user puts down two fingers on these two markers, on the star and on the plus sign. And then they move those two fingers to the right-hand side. Well, we want realistic UX design, so they should still be touching the same points. What about for zoom? OK, let's look at a similar case. The user is pressing down the plus sign and the multiplication sign, and then they move those two fingers closer to one another. Oh, well, that's interesting. We want exactly the same thing. The user is still touching those two same markers. So yeah, that is very interesting, because initially, we thought that the zoom and the pan, they were two different features. But as it turns out, they're both the same thing.

What we want here is that we want a feature that when the user touches the stuff and moves those touches around, we want the touches to still be touching the same stuff. And now I've said a lot of stuff, but you're probably thinking, what is he talking about? What are we actually trying to solve here? So let's narrow it down. Let's just think about one dimension for the time being, just focus on the x-axis. So here we've got this HTML number line, a one-dimensional component, with two moments in time, start and touch move. Let's hone in on what we're supposed to do when we receive a touch start event. So what are our inputs at this moment in time? Well, obviously, we know the screen offset and the width of our number line. We'll call that rx and rw. That's in pixels.

OK, what else do we know? Well, we also know the offsets of the touches, because we received that from the touch start event. We'll call that a1 and a2. That's also in pixels. And obviously, we know the zoom constraints of our component, because we've initialized that from the get-go. It'll probably be like 0 and 1 or something. Yeah, we know what are the zoom constraints, zmin and zmax. And so what we're basically looking for at this point in time is we need to find some way to remember that the user is touching the numbers 4 and 6. So basically, what we want is some kind of normalized coordinates between 0 and 1. We'll call that x1 and x2. And basically, we can calculate that using some interpolation equation. We don't need to worry too much about the details of this equation. But basically, what's happening here is we're adjusting the pixel values and the zoom values such that we have a normalized coordinate between 0 and 1. I've also added in here this end value, which basically, for our use case, just means 1. I've added it there for completeness, because you may in some cases, because you're dealing with floating-point errors and whatnot, you may want a larger value in some conditions. For us, let's just say n is there. It's 1. I've put it there for completeness.

Now, what happens when we receive a touch move event? What are the inputs at this moment in time? Well, obviously, we still have the viewport offset and the width. That hasn't changed. The user hasn't gone to a new tab or whatever. They're still touching stuff. We know what are the new coordinates of our touches because we get that from the start event. Sorry, the touch move event. We could, if we wanted to, calculate the new normalized coordinates of these values using the same equation and just simply inputting all the new values. But hang on a minute. What is this Z-min, Z-max with apostrophes? What are these things? We don't know these values. That's what we're looking for here, right? We're looking for a new Z-min and Z-max value such that the X values haven't changed, such that we're still touching the same numbers that we were touching before. So this is great. I know this does not look groundbreaking, but it is because think about it, right? This is not one equation. It's two equations because we've got two fingers.

And so now everyone, I'm sure you're all shaking because we know what's happening, right? We've got two equations and two unknowns. So what we can do is we can do a lot of very boring math, lots of stuff. This is my favorite slide in the entire deck, by the way. I had to like scribble loads of notes on a piece of paper and it took me ages to figure out how to put this on a Google slide. But I did the hard work. It's there. We have an equation for Z-min and we have one for Z-max. We can just use that. That's it. That's all we have to do.

Okay. Let's try and code this up now. Let's start first with the touchStart event. So what are we doing here? Well, as you can see, the important thing here is the toNormal function. That basically just implements the mathematical equation that I was talking about. And then our touchStart handler, that basically, well, we have to keep track of the identifier of the touches, obviously, but we receive as inputs the CSS bounds and the two touches, which also contain the offsets of the touches. And here's what's interesting, right? Because if you remember, we specifically want the X axis and the Y axis to be independent of one another. So what that means is all of this math that we've done also applies to the X axis. Obviously, we have to use the height instead of the width, but we can just plug in all of these inputs into the math equation solver and it will just do the same, do everything as we want for the Y axis.

Now what happens with a touchMove event? What do we want to do here? Well, interestingly enough, we do something fairly similar. So we've got this one function here, solveTwoUnknowns, which is solving these two equations. It's a little bit more convoluted. There's a bit more math going on in there, but we're doing the same thing. It's just giving us a Zmin and a Zmax. And here we do the same thing. Basically, we're tracking our numbers. So we find where they are by comparing the identifiers. And then we input all the pixel values that we have from our CSS bounds and our touches. And we input these normalized values that we calculated at the touchStart event. That's going to give us the new Zmin and Zmax, and we'll always be touching what we want. This is great.

So now I've rambled for like 15 minutes, and I haven't even said the word React. So you guys are thinking, what is this guy talking about? But this is not what we came here for. But it's okay. Okay. We'll take all this. We've done the hard part. Let's try to integrate this into React. If we're going to design a React component that's very robust and reusable, we want it to kind of work on any component, any sub-component of a component, any sub-component of a sub-component. We want it to work basically on any HTML element.

Fortunately, there is this feature called a CSS transform. And that works on any HTML element. It has two components that we care about. It has a translation. We can use that for adjusting the panning. And it has a scale. We can use that to adjust the zooming.

And it has a scale. We can use that to adjust the zooming. And obviously, you know, those values come with the X and Y component to it. So this CSS transform, that's gonna be our use state for our component.

All right. Good. Let's have a look at the component. So this is not a complete implementation. This is if you want the skeleton of our component. What we've got here is our use state, which is just a CSS transform. And for this example, I've kept it simple and I'm just using an image tag. In practice, this works on any HTML element. But I've kept it simple for the time being.

We put in our CSS transform and set the right origin. And that's the core of our component. Now let's integrate in our math solver. What do we need to do that? So this is the entire code of our component. What have we added here? Well, we've added a couple of refs. Obviously, we've added the container ref here. Because if you remember, our equation solver requires the offsets, the widths and the heights to do the math. So we've gotta keep track of that. And then I've also added an instance of our zoom pan solver. And so what's nice about that is we can call the use ref a function. Because really, we don't actually care about the internal state of all the math that's happening. The only thing that really matters is if our CSS transformation changes, then okay, we need to redraw something. Because that's the only state we need. And then I've also added some input handlers for touch start and touch move. But the handlers are actually nothing we're unfamiliar with. So we're just checking, have we got two numbers? Is the HTML element null? Don't do the default browser behavior. Because browsers really love touch events.

They really want to change stuff. So don't. Just shut up. Shut up for a sec. And then you get the bounds. Because we need that for the math solver. And then you do the same thing on both the handlers. We're very familiar with that.

So the only thing here that's missing is how do we calculate that CSS transformation. Let's zoom in on that. So this is how you would do it. What I've done here, basically, I've added this two client function. Which is essentially just the inverse function of our interpolation function. Because think a bit about it, right?

If you've got a normalized coordinate zero, and you input that into the inverse function, it tells you where that zero is in screen pixel coordinates. Which is what you want to translate the CSS. So that's good. We can do that to calculate the transformations for the X and Y axis. And then for the scale, it's pretty straightforward as well.

So N is basically R1. That's the maximum in the normalized coordinate space. You find the difference between the min and the max. And voila. That just gives you the scale.

So let's try to have a look at it. Just give me a moment to load up the demo on the iPad. Now I'll get a popup any minute. Yes, there we go. Yes. Accept that. Here we go. So this here is the component in full.

So let's see that it does actually what it's supposed to. We can zoom in here. That's nice. That's cool. We can put two fingers on there and pan them around. That's good. Yes. That's panning. That's what we want. We can also adjust only one axis. For example, only adjust the X axis by moving the fingers up and down. Yes, that's great. It does that. We can also do it left and right. Okay. All right.

All right. It does distort the image in this case, but that's what we want, especially for the chart. And let's check that it actually is realistic touch gestures. So this is the tricky part of the demo. I have to make sure that I touch exactly where I want to. Let's try to touch the faces of these two sheep here. So if you put the faces on these two sheep, as I move this around, you can see that the two touches are basically always touching the same part of the image, even if I distort it. It's still touching more or less the same place.

So that's great. It's doing what we want it to do. There's a couple of caveats with this demo. So I'll go through them. So one of the caveats is that obviously a common use case is that you may want to, say, for example, only adjust the x-axis. You'll put your two fingers like this close to one another and you'll move them apart. But here's the thing. So this may come to you as a bit of a shock, but I am actually not a machine. I'm a human. And so when I put down the fingers at the start, there was a little bit of y difference between them. And it was very, very tiny.

So that means when I put the two fingers down like this, that causes the y-axis to be very sensitive. So we can actually fix this. And that's what we do on AG charts. If the user puts those two fingers like this, but the difference, the y difference is very, very small, we just ignore that. We pretend that it's zero. And that lets you basically just adjust the x-axis. Then there's a couple of other caveats. So let's say, for example, you put two fingers down and you move them around. Again, like I said, I'm a human. I'm not a robot.

I'm not a robot. So obviously when I move my fingers around, I'm not perfectly moving them in sync. They sort of go a little bit like... They move around a little bit. So you can kind of notice that the scale is a little bit like fidgety. It's like make this big, make this small. It looks weird. But we can fix that as well. So we can add some sort of like snapping logic so that when you move it around, if the fingers are not moving by much, just ignore that movement. You could add some kind of logic for that.

Then another feature you could do is what happens if the two fingers swap positioning. So let's say, for example, you go like this. In here, this is not very good. I've added a hack to basically say ignore this case. So when they flip around, my code is like if the hack is like, pretend that didn't happen. But you could add that as another feature. So you could add some sort of mirroring effect to actually swap the image around if you wanted to. So that's broadly speaking how you implement a pinch to zoom component with independent axes in React.

Here is a QR code to scan, to check out the source code. If you want to play around with the demo a little bit. It also includes the slides and some additional documents that explains how the math works. You can also try out the AG charts library if you want. Because we've actually got an implementation of this. If you want to have a more realistic use case than just a bunch of sheep in England, you can try it out there. Definitely highly recommend. And I'll be available for any questions at this point in time.

In the meantime, I'll ask you just a little bit about the libraries that you're working on. So AG charts and AG grid, are those the two main AG libraries? That's correct. And when you say main, I should emphasize the only two libraries. The only two. Yes. Perfect. Cool. And so AG charts, is that based on anything? Is it based on D3? Is it just entirely from scratch? It's completely self-contained. So we don't have any external dependencies on it. We have some internal dependencies. So we have a couple of packages that depend on each other, but no third party dependencies. It's all built from the ground up. It's also built in vanilla TypeScript, so you're not constrained to just React. You could use it in any other framework. But we have a wrapper for React if you wanted to. Awesome.

First question in here. What other touch gestures does AG charts support? Panned with momentum? I'm not actually sure what pan with momentum. Oh, pan with momentum. So like that... Oh, yes. I love that idea. I love that. No, it doesn't do that. But maybe it might one day. That's a really great idea. Thanks. Yeah. Yeah, all those physics-based gesture stuff.

Yeah, all those physics-based gesture stuff. Yeah, yeah. Yeah. All right. Next one. Any performance considerations using a CSS transform? I've heard rendering painting can be impacted as soon as you add transform rules.

Yes. I mean, that is true. So you do have to repaint it whenever there's a change. But then at the end of the day, there isn't really a way around it. So like in that specific example, you're distorting the image. So like at some point, you're going to have to redraw everything because all of those pixels have moved. So you could add some sort of optimization. So the way we optimize this in AG charts is that we only need to apply the transformations to the actual data in the chart. But other things like the legend and the title and all that, that doesn't need to be redrawn. So we just redraw the relevant parts of the canvas. Yeah. So for people using transforms, you can use a 3D transform to invoke the GPU.

All right. Would this be a good integration for maps? And would this work for trackpads or only just touch screens? I don't think it would. Maybe. Depends on what kind of trackpad technology you're using. I don't think it would work on a Mac, for example, because that's basically a... It doesn't export touch events, but there might be different on different hardware. This would not be a good integration for maps. It would be terrible because you would distort the image completely and it would look terrible. In that case, you would want to use the classical approach of just tracking the average positioning and the distance and use that basically to implement ZoomPan. How can keyboard-only users use Zoom and Pan? More broadly, how would you make accessibility intuitive in a domain like DataViz without strong standards? I like this question because I spend a lot of time working on the accessibility stuff. And, I mean, sadly, there isn't a very good way to deal with this for accessibility purposes. So the alternative that we have is that we have different ways of adjusting the Pan.

So you can, for example, use the plus sign or the minus sign to adjust the zoom. And there's other key bindings and other features that we have, such as the zoom buttons, the navigators, there are other features you can help to make your chart more accessible. But I think for this specific use case, I don't think it's very feasible for us to make it accessible, unfortunately. Yeah.

How often do you implement feature requests from users and which one was the most difficult? This one was pretty difficult. I had to argue with ChatGpt and write a bunch of notes. How often do we get the request? I'm actually not sure. I don't have much exposure on our ticketing system. But I think it's quite substantial because, obviously, the library is very mature now. So we have more time to implement feature requests. Yeah.

Are there any trends or expectations for what should happen for three or four finger interactions? So I don't think there's any expectations right now or any precedents or whatever. But we could, in theory, add support for that. You may want to have three fingers adjust some other part of the view. But I don't think there's any trend going on that I'm aware of. Yeah. I think one of the Google Maps or whatever adjusts the pitch of your viewpoint. That always gets me when I'm using it. I'm like, why are you doing this? Yeah. There might be a use case for it. Because obviously you're dealing with multiple axes and you have different series. So maybe three fingers could adjust which series are visible or something like that. There could be a use case. But there's no trend that I'm aware of.

Given the state of AI, how important do you think it is to understand what happens under the hood and know the underlying math? So yeah. I think it's absolutely fundamental. So I don't want to open up a can of worms because I could talk about this for hours and hours and hours. But essentially, I think it's very, very crucial to understand the theory and to be able to actually do the hard work yourself.

So I don't want to open up a can of worms because I could talk about this for hours and hours and hours. But essentially, I think it's very, very crucial to understand the theory and to be able to actually do the hard work yourself. And you can use AI to speed up your workflow, but at the end of the day, you have to kind of understand what it's doing. Because if you don't understand, then you won't be able to identify bad output generated by the AI. Yeah.

I totally agree. We have been researching using a plug-in feature approach for our internal front-end platform. What are some pitfalls or concerns you would think about for this use case? So I mean, I've talked already about a couple of the pitfalls. So some of them was like, for example, the y-axis is very fidgety. And when you pan around, that's very fidgety. So there's still some work that needs to be done to try and make it a bit more usable because humans are sadly very imperfect creatures. And then the other issue would be that obviously this isn't applicable to every kind of UI element. It's specifically only usable when you want the x- and y-axis to be independent, which is actually probably a minority of cases. I think most cases you generally would want to keep that aspect ratio locked. Yeah.

And what about with screen readers? Reading table data in charts is always a problem. I don't actually understand the question. How to work with screen readers? Well, I think... Yeah, you've got a lot of data. Yeah, so we do have a built-in mechanism to give updates to the user. So I've got a whole separate talk about how we deal with screen readers. But essentially, yeah, as the zoom values changes, we could announce new values. So for example, as that changes, we could say like, oh, it's now 20% zoom and 40% zoom. It becomes more of a, I think, UX problem rather than a technical problem, because how would you make all of these changes accessible? Because there's a lot going on and you don't want to obviously bombard the user with lots and lots of cases. So it's another UX problem I don't really know. I don't really have a good answer at this point, unfortunately. Yeah. Well, thank you so much. That was awesome. And I really enjoyed it. Oh, thank you very much. Yeah.