Sketching with Code: Integrating React and p5.js

Hello React Summit! I'm excited to be here. Like it was said, I'm a senior software engineer. I'm currently building educational experiences at NewZella. I'd love to chat more after my talk about creative coding or your favorite open source libraries. And I became a GitHub star a few years ago based off of my involvement in the tech community and I'm super passionate about helping people if they're interested in getting more involved with open source.

So today's talk is a creative coding talk. I would say my creative coding journey started on this website, Neopets, specifically their HTML guide. This was the first time I was writing CSS. And now I'm still writing CSS and exploring creative coding and other mediums. So I have a CodePen where I have a ton of examples that are written in p5.js, svg.js, just plain CSS. So if you are interested in seeing more examples after today's talk, I encourage you to look at my CodePen. And I also have on this slide a video of the pen plotter that was mentioned. So this is a little machine where you can give it instructions based off of a vector file. And you can use a pen like this gel pen. You can use paint to transfer it to paper.

So today's talk is about creative coding. I will be introducing folks to p5.js, why you should consider using React with p5.js. Actually showing you how to get started using those two libraries together. And we'll cover a little bit of animations, gradients, and more. And over the course of this talk, we'll be building up this application or this small application to generate a grid pattern. So this is available at tiles.monica.dev. If you want to play around with it. And if you scroll down on the page, there is a link to the GitHub source code. So you can look in depth at all of the logic of how this works. Which we'll be covering a good portion of it today. But I don't have time to go through everything line by line. And for today, our working definition of creative coding is about using algorithms or a predetermined set of rules to generate imagery. So these four images are these lava lamp-inspired images that I created using an algorithm called the marching squares algorithm. And so you can tell that these are all different shapes, but part of the same system. So that's what I mean by having, like, this predetermined set of rules that will determine the output.

So they're all unique. But there's some commonality based off of the rules behind it. And today, we'll be incorporating user inputs into our creative process. So what a user types in is going to control the shapes, the colors, and the text that appear. And I really like this quote by the Variable Design Studio about generative art and artwork that grows, changes, and transforms over time. I think that's, like, one of the really cool things about creative coding is that it can respond to users or data or changes over time. It's not just, like, this fixed piece.

And an art exhibit that I really enjoyed seeing is this decoding happiness exhibit at the Wellcome Museum where attendees were invited to check in. And based off of how they were feeling in that particular moment, they have a piece of art that was created. So it's very specific to that point in time. And there's actually a guide behind the different visuals that you see in terms of the colors, the shapes, and the sizes. And once you have that key, when you see the different images that were created, you can kind of gather at a glance more information about each of the people and really take a deeper look. Our example today is going to be more abstract. But you can use those same principles if you wanted to create something similar to the decoding happiness in terms of being able to create art that is unique to an individual or certain data.

And so we will be using p5.js, a creative coding library, to draw on the canvas and using React to manage the state of user inputs. p5.js is a JavaScript library and it's designed for creative coding. It's accessible to folks whether you have a background in software engineering or you're coming from education or more of a traditional artist, which is something that I really like about the project. And all of the concepts that are within p5.js you can of course apply to other creative coding libraries as well. And under the hood, p5.js is an abstraction of the canvas API. So that is a way that you can use JavaScript to add elements to the HTML canvas element. And the canvas element or API can be used for animation, game graphics, data visualization, and more. There's an application called JS paint that you can check out on GitHub where it's built using the canvas API and you can recreate the experience of if you were younger, using MS paint to create art.

And now I want to talk about specifically what inspired the pattern that our example we'll be using today, which is true shade tiles. So these are different tiles that when they're arranged in different orientations, you start to see patterns emerge. Similarly to true shade tiles, there's an algorithm called 10 print, which is a one line program that is randomly alternating between a forward slash and a backwards slash. And when you run that program, you'll actually start to see these labyrinth like patterns emerge just from the random orientations. And there's a whole book dedicated to this algorithm if you're interested in learning more.

One of the true shade tile variations that's most popular is this Smith tile variation, which is composed of quarter circles and it helped make this pattern more popular in the 1980s. And similar to the original true shade tiles, as you rearrange the Smith tiles, you start to see different patterns emerge. So let's create a Smith tile in p5.js. So the first thing to know about p5.js is we need to create a setup function that's called one time per function to set up things such as the color, background, size, etc. Things that do not need to change from frame to frame.

So this is a basic p5.js program where we're calling the setup function, creating a canvas, setting the width and height to the same as the window. We're setting the background color of the whole canvas to navy, positioning the element, and then we're drawing a square. We're saying at the 0, 0, x, y coordinate that is 400 wide. And you'll see that there's just like a blank square that appears on the canvas. Now, if we updated that and added in this call to no fill and set the stroke to white, then instead of seeing a filled in square, we'll just see a blank square. This is one or that's one way we can create graphics in p5.js. But there's a create graphics function that is recommended when you're creating or drawing on the canvas because it's able to optimize the performance of your application and draw those off screen on the background. So I would recommend that we use that as we're building out our project today.

So in order to use the create graphics function, you'll make a p5 graphics object by calling the create graphics function and passing in the size of your element. So those are like the width and the height of the graphics objects. Then we can call those same functions we were calling before to create the element. But you'll see there is no element that is appearing on the page. That's because we actually need to call the image function and pass in our graphics objects for it to render on the canvas. Now to create the actual trusset tile, which is not just the outline of the square, we need to also create arcs. So there's an arc function in p5.js where you can pass in the X and Y coordinates of the arc and have it drawn to the canvas. And so you'll see the top arc is hot pink and the bottom one is teal. So how p5.js works is whenever you make any changes to the colors or sizes, it applies to any elements or functions that you call after that until you change it. So originally we set the stroke color to white and then we drew the square. But then once we changed it to hot pink, the very next element we drew turned to hot pink. And then it only affected that one element because we then changed the color again to teal.

So order is super important in p5.js. And like I had mentioned earlier, you'll start to see these different patterns that emerge as we put these tiles together into a grid. If you rotate them, you'll start to see different shapes. And you can play around with, like, the color and even, like, what an individual tile looks like.

So back to what we're building today, this trusset tile generator. So like I said earlier, we're going to be using React to manage the state of user inputs and p5.js to draw on the canvas. So I find that React is a lot better at managing complex state than p5.js. P5.js' primary purpose is for drawing on a canvas. And React also will allow us if we want to create user input elements, we can position them using, like, a tree. We can position them using, like, CSS for the positioning as opposed to in the context of p5.js. Everything is within the canvas, and we have to determine, like, what the X and the Y coordinates are. So it's a lot more challenging to deal with, like, reflowing the layout if you're using p5.js. So I definitely recommend if you have more than one or two fields to consider using something like React or another library to manage that.

So when using p5.js with React, there is a handy p5 wrapper package that I would recommend using to help make your life easier. So with the p5.js React p5 wrapper, we can import that wrapper and then we can render it within our React application. And you may notice here that I'm checking that the window is not undefined because if it is, I don't want to render or I don't want to try to call the p5 library because it's a client-side only library. There is a node version of p5 if you wanted to create a, like, I created a blue sky bot at one point that was using p5 to post images. So there is, like, a node package, but typically it's a client-side only library.

So in addition to the setup function, there's also the draw function, and p5.js is one of the core functions that you would normally see when you have a p5.js sketch. And that's something that's continuously drawing on the canvas and by default it runs 60 times per second. And when using p5.js with React or with any other library, you want to use p5.js in instance mode. So that means everything is going to be namespaced to a p5 object, whereas if you want to quickly use p5.js, you would just use it in global mode, which means you don't have to namespace it. So you would just call set up and draw. But in instance mode, you have to put p5.setup, p5.draw, and that p5 can really be named anything. It just needs to be referencing a p5 object. So with instance mode, we can have multiple sketches on a page. It's compatible with React or other libraries. The issue when you don't namespace it is that there could be a collision. If there's more than one JavaScript library that has the same naming conventions, things just start to fall apart.

If there's more than one JavaScript library that has the same naming conventions, things just start to fall apart. And while it's more complex to get started with instance mode, it's definitely overall safer and helpful as your project grows. With global mode, if you did want to have multiple sketches on a frame, you would have to use an iframe. It is a quick way to get started, but like I said, I think as your needs evolve, you would likely end up having to move over to instance mode.

So back to our React application. So we have the React P5 wrapper, and within that, like I said earlier, we have this sketch component. So the sketch component is using p5 in instance mode. So you'll see we have this p5.setup function, p5.draw, and there's this new function, p5.updateWithProps. So within our sketch, we have these local variables like the current color, but we can override them with the values that we're passing in from our props. But we do need to define them separately within the sketch so that the variable has already been declared.

And I had mentioned earlier that draw runs 60 times per second by default. That's something we can always adjust. You can either call no loop if you never want it to run. You can pass in a specific frame rate. But it's really important to be mindful of how often it's running and if you have any performance bottlenecks. Because it runs 60 times per second, if you have something particularly slow in the draw function, that really only needs to happen once in the setup, that can make a big difference for how your program is operating.

And in our application, we can use an object to represent the state of our application. So we can have a name that's a string color, which is an object of string because there's three different colors we want to capture. The size of the tiles being a number. Pattern mode. We can have multiple pattern modes. We can use Booleans for whether or not we're saving the image if we want to show grid lines or a banner, and also having two different art modes that we support.

Now in terms of actually getting that information into our p5.js sketch, how I have it set up is we have a reducer. So the sketch has access to the input values, the state of the input values. The sketch is never going to update the react state, so it just needs read-only access. Whereas our input fields, we want to actually dispatch actions to update the state as the user interacts with our application. So the input fields component is going to have access to both the input values and the dispatch function. So if we have an input component for our three colors, we can have a handle color change function that's called whenever that color changes as the user interacts with it. And the handle color change is going to dispatch an action of the type colors, and then it's destructuring the current colors that are in the state so that we can keep those and just override the one that's been selected by using the event name and event value. Then within the sketch file, we can use that update with props function to make sure that our input values are reflecting what we've received from the props.

And we can destructure the colors or any other values that we're getting back from the input values and use that, for example, to set the background color of the canvas. So you'll see here in this graphic that as the state is changing in the input component, the colors on the canvas are also reflecting those changes as well.

And speaking of colors, one of my favorite P5.js functions is this lerp color function, so it's using linear interpolation to create color gradients. And linear interpolation is a way of, if you have two known data points, you draw an imaginary line between them, and then you can say I want a value that is like 50% between these two values or some other amount between the two values, and then it will guesstimate where that value would fall. So we can use that with numbers, but we can also use that with colors.

And using lerp color in action, in this example, we have a grid of the touche tiles. And so you'll see within this grid, we have this nested for loop to create the grid. We're randomly rotating each of the tiles so we can get those patterns that show up. And then for changing the color of the tile, we can use p5 tint since we're working with image objects. And how the lerp color function is set up, we have first the from color, then the to color, and the amount of where that color should lie between the two is set up based off of which row we're looking at.

And using lerp color in action, in this example, we have a grid of the touche tiles. And so you'll see within this grid, we have this nested for loop to create the grid. We're randomly rotating each of the tiles so we can get those patterns that show up. And then for changing the color of the tile, we can use p5 tint since we're working with image objects. And how the lerp color function is set up, we have first the from color, then the to color, and the amount of where that color should lie between the two is set up based off of which row we're looking at.

So the higher the row is, then the closer it will be to the to color because we're dividing the row by the height. So the highest row, that would equal one, and the lowest row, it would be closer to zero. But we can also completely change how we handle the colors by having something like a handle color mode change function. Where we update the pattern as the user interacts with the application from a gradient to this random distribution. So with the gradient distribution, we went over before we're determining which color to tint it based off of which row we're looking at. But we could just say, okay, if it's not in the gradient mode, we just want to pick a random color. And so we're going to return a number between zero and one and then pass that into the tint function which is calling lerp color. So that's what you see on the right-hand side where the colors are just randomly distributed. And you'll notice that there's more than two colors even though we only set two colors.

We've talked a little bit about randomness. But I think it is important to note this quote that if there's one thing computers are good at, it's being predictable. They can execute the same code a million times. And so long as they are given the same inputs each time, they will always come up with the same outputs. This is a quote from Joshua Leboeuffuser at CloudFlare. And this applies to random functions. So if you're calling math.random or something, it's not as unpredictable as it seems. So CloudFlare, they actually have a wall of lava lamps that they use to help make their random number generation more cryptographically secure. So the quote that I put on this slide is actually from an article if you want to learn more about how they're using lava lamps for that. And within P5.js, like I said before, the numbers aren't... they're not unpredictable, the random numbers that we're getting. But the cool thing about P5.js, unlike just vanilla JavaScript, is that we can actually set the seed value that is being used for... or easily set the seed value that is being used by the random function. So on the left-hand side, this is a picture of a canvas where every single time the canvas redraws, we're selecting a new random number. And you'll see that the numbers keep changing. There's no random seed set, so every single time it reruns, we get new random numbers.

And then on the right-hand side, if we have the same random seed being set, meaning we're calling this random seed function in P5.js, then every single time we rerun the application, we're going to get back the same random numbers. So it looks like nothing is happening from frame to frame just because those values are staying consistent. And both of these would be considered pseudo-random numbers because of the fact that if you give it the same seed value, it's going to give you back the same numbers. So we can use that to our advantage for animation or lack of animation.

Similar to what I went over before, we could have an action... an event handler focusing on the animation. So this handle is animated. It's just making a Boolean value that's changing from isAnimated to not isAnimated when the element is checked or unchecked. And then within our sketch, we can say, okay, if we know from the input values that the camera is animated, then we can call the p5.loop function. So this is generally what happens by default in P5 sketches, but if you ever call no loop and then want to resume a sketch, you need to call p5.loop. And then we can say, if it's not animated, then call the no loop, and that will just stop everything. But another way we could control the animation is by saying, if it's not animated, we're going to set a random seed. So we can make that a... for example, a name could be the seed. You just have to convert it to a number. And then if that seed doesn't change, it doesn't appear to animate unless there's changes that happen from frame-to-frame that don't rely on the random function. So you'll see, if you turn off animation, but then start typing your name as you type the letters, then it's going to change and then stop if that is how we have it set up.

So that's my presentation for today. I just wanted to introduce folks to P5.js and show how you can use React if you want to manage the state of P5.js for the sketch and how it can help make it easier to render the UI, especially as you build more complex P5.js sketches. And P5.js has a ton of built-in functions for creating data visualizations and creative coding. I didn't have enough time to go through everything today, but I encourage you to check out their documentation if you're interested in learning more. So thank you, React Summit!

I have struggled a bit just with the Canvas. It's not as easy to make it mobile-friendly as it is with CSS. So that's why I would maybe encourage folks, if they want to have something mobile-friendly, to consider using something aside from P5.js for the mobile part. But there are definitely workarounds in terms of how you scale things to work on mobile, but I think it takes more work than relying on CSS. That it's more designed for that use case.

Okay, cool. We have more CSS questions. Can we animate P5.js sketches using CSS? To my knowledge, no, but there's a lot of ways within P5.js to do animation. I had said you can set the frame rate. You can set that. You can also determine how much time has passed on a P5.js sketch. Based off of how much time has passed, you can say, okay, this element should grow or shrink over time. So there's plenty of functions within the library that can control the animations. So I don't think you would need to rely on CSS. I think that's one of the strong points of P5.

A couple more minutes of questions. Thank you, ShadowDanny. Does having P5.js visuals or could P5.js visuals slow down a website's performance considerably? I think that's definitely something you would want to benchmark. There's things that you can do in P5.js to optimize the performance. Like I had mentioned, you can create the graphics object, so that it's drawing it off screen as one optimization. There's definitely certain, if you have things that are more computationally intensive, you may be able to do them in advance. So I've had some sketches where I wanted to manipulate an image. So instead of loading the image data, I just ended up converting it into X and Y coordinate values based off of the RGB values. And then gave it a data file instead of the image. Because that was more performant than working with the image directly. So definitely depending on your use case, definitely be mindful of how it could affect performance. And there's also a clear function in P5.js where you can remove elements after a certain amount of time. Even if they're not visible. Just to make sure that you're not having extra things on the canvas if you don't need them to be there.

Even if they're not visible. Just to make sure that you're not having extra things on the canvas if you don't need them to be there. That was a very senior dev answer. It depends on the user.

Cool, I think one more question. Does it use WebGL under the hood? Is it a convenience wrapper to what HTML5 canvas stuff? It's definitely a wrapper to the HTML canvas API. But there is also another version or mode of P5.js, if you are interested in WebGL and more 3D rendering. It's slightly different syntax than the regular 2D version of P5.js, but it's all part of the same library. So if you go to their documentation, you'll see information on that mode as well.

Cool, well I have so many more questions as we all do. But that was all the time we have. Thank you again so much, Monica. Let's get one more round of applause. Applause. Music.