Remix Persistence With DynamoDB

Hi, everybody, it's a real pleasure to be here, and I'm stoked to be part of this remix movement with you. And today I'm going to be talking to you about persistence with DynamoDB. My name is Brian LaRue, you can find me on various socials, so it still exists under that name, and I work for Begin.com.

So, before I get into Dynamo, I'm just going to talk about persistence in general. So, persistence is a really important requirement for dynamic web applications. It is an essential complexity for anything that's personalized, so anything that's got an authentication step and we're saving some data about a person and we need to do that in a secure way and in a fast way, you're going to need a database. And you can't do this with a flat file system. You don't want to do this with a flat file system because you'll get into concurrency issues.

So, traditionally, people would choose a relational database, and DynamoDB is a kind of next-generation database that is key-value based. And so, most organizations out there these days have settled on Amazon for infrastructure, and Dynamo is the flagship key-value database for AWS. So, if you're using AWS, you probably would like to learn more about Dynamo.

So, what exactly is Dynamo? It's a low-latency, wide-column key-value store that's a really fancy way of saying that we query by key, and we store values as JSON, and we can have different shapes of different columns on every row. So, for every row in my database, I could have different item attributes and it's all good. Dynamo is a completely managed database, and this means that there's no patching. There's no software to upgrade, and that's really nice. And also really nice is that it's scaled to zero, which is a fancy way of saying that you only pay for what you use, and you don't pay for anything else. So, it's 100% utilization. You're not keeping a big cluster of servers around just to meet demand that you may or may not have. You just use what you pay for, and happy days. You move on from there.

So, the next kind of big question that I often get is like, well, why would I choose DynamoDB out of the millions of database options that we have available to us out there? And the key one for me is it's the flagship managed database for the pioneer of the cloud, and they use it themselves at Amazon to back Amazon.com's retail business. So, it's just a good choice. From that regard, there's great local development options. It's got a huge free tier. It's fast no matter how much data you store, which is kind of a science fiction dream for databases. Used to be, as you added servers, you would add latency, and you definitely would add problems, which we just don't experience with a managed database. It's very small, so there's not a whole lot of API to learn. There's really only about six key API calls to deal with Dynamo, which is really great. It's got a good SDK for just about every runtime you could think of. And the sort of sleeper feature, my favorite feature, that it works really well with AWS Lambda, single digit millisecond latency for queries, no matter how much data you're storing. It's generally the line that gets tossed around, and this is a really big deal because a lot of databases are quite slow today, especially traditional relational databases. They can be made fast, but they're never gonna be that fast. And they just don't work as well with Lambda for a variety of reasons.

So, the next question would be, well, why not DynamoDB? Obviously there's trade-offs, this isn't gonna be a trade-off-free decision. So, the first big kind of thing is that it's an Amazon-only thing, so people get worried about that. This isn't actually technically totally accurate, though there are examples of people cloning Dynamo now for other key-value stores, so we saw this happen with a lot of other Amazon services, like S3 very famously, it's cloned quite a bit now. So, that'll probably continue to happen.

A very valid criticism, though, is that it's just a bit weird to learn. So, modeling for a wide-column key-value store is very query-centric, you'd have to think about how you wanna get the data before you store it, unlike relational data, where we denormalize everything and we can kind of query ad hoc. And the other thing that's a bit weird is that the query syntax is a bit strange. It's not as same as relational, although I would argue that this is just a familiarity thing if you had learned Dynamo first and then learned Postgres after you'd probably find Postgres weird, so not really a major thing.

Why non-DynamoDB also has a bunch of myths, and I wanna address these head-on. So one of the first myths is that it's expensive, another one is that you need to know all of your access patterns completely upfront before you do anything, that's not true. I've heard that it's hard to modify and migrate, that's also not true. Sometimes people say you can't use SQL, that's not true, and we'll address the biggest red herrings of all, lock-in and scaling at the end, cause it'll be funny.

So, first one is, is it expensive? Well, the DynamoDB Free Tier gives you 25 gigabytes stored on disk per month for free, and it's 25 cents for every gigabyte after that. That's a lot of starter data, 25 gigs, it's a ton. From a, that's just on disk, what it costs. You also have to pay for read and write, and basically it works out to about 200 million requests per month is gonna be in the free tier. So, storing 25 gigs, 200 million requests per month. I think this is not expensive at all, this is very cheap. And so, even if, as you scale you find these numbers to just get out of hand, it still speaks to using Dynamo for at least prototyping your application. It's gonna get you really far, for very little. As a side, the joke I used to like to say is like, yeah, Dynamo's expensive but so are DBAs. And that joke really only lands if you've had to shard a database. So, I'm not gonna make the assumption that anyone knows what I'm talking about there.

Another big myth is that you need to know access patterns upfront. So DynamoDB modeling is different than relational database modeling, and it's just different, it's not better. It's just different. And so, in both databases, relational or key-value, you need to have a schema of some kind. You need to say that these values are expected and that's how I'm gonna query for stuff. So, it really doesn't matter which database you choose. At some point, you're going to have to declare a schema.

You're gonna have to say I have an accounts table or I have a songs table or whatever. And this is true, whether you're dealing with relational or a wide column store. So, I don't think it's a great argument. You're going to be migrating and learning about your app, and as it gets more big, and you understand the access patterns more, you're gonna be able to store the data in more efficient ways. And that's true no matter what database you choose.

Sometimes people say you can't use SQL with DynamoDB. That used to be true. That's no longer the case. We now have a query language called PartiQL. Sounds fun. It looks like SQL. So, I guess if you find that fun, then it is. And you can click through and check it out. It's not just limited to Dynamo. It's actually supposed to be a broader idea. I think right now, Dynamo is probably the only people using it, though.

So, the other big red herring that gets thrown around is whether or not you're locked in. And at a certain point, you're always locked in. And I would argue that the biggest lock-in of all is choosing upfront complexity. And sharding and managing a running instance for a database is definitely a lot of complexity. Sometimes people think that if you use a database like Postgres, that means your data is inherently portable. And I think that's a lie. You're going to have to go through a huge amount of pain to move that data between different database providers, regardless of the database you choose. So, to me, it's not a very good argument. There's a much longer, more rational written argument here on Martin Fowler's website that you can check out. I just don't think lock-in's a good reason to choose anything. If you've already selected Amazon as a vendor, you're probably going to be there for a while.

This one's fun, and it comes up all the time. People are like, well, does it scale? Yeah. It super does.

So Jeff Barr last year says, Amazon DynamoDB powers multiple high-traffic Amazon properties and systems, including Alexa and Amazon.com sites and all the Amazon fulfillment centers. Over the course of the 66-hour prime day, these sources made trillions of API calls while maintaining high availability with single-digit millisecond performance peaking at 82 million requests per second. That scales.

So it's also not just Amazon. Social proof is how you like to make decisions. There are a ton of logos here of people using DynamoDB at scale for a variety of applications. I think we can put that one to bed.

So as a very fast recap, DynamoDB scales up and down dynamically, generous free tier, supports an SQL-like syntax. It's fairly straightforward to migrate around, works really well with AWS Lambda. Yeah, it scales and yeah, it's a bit weird to learn and we're gonna get into that.

Before I get into it too much further, if you just wanna zone out and not pay attention to the rest of my talk, a thing I would recommend you do is pick up this book and follow Alex DeBrie on Twitter if you're still there, this is just an immediate buy for your whole team if you're using Dynamo, it is the best docs written for DynamoDB by far. When we started adopting Dynamo, I wish this book had come out because we would have saved a lot of time if it had. It is the cheapest thing you'll ever be thankful that you bought. It's gonna give you super powers and I can't recommend it enough.

So Dynamo, let's get into it a little bit deeper. Some terminology to be aware of, first of all, so Dynamo has tables and tables are collections of items. You could think of these like pages in a spreadsheet and items are like a row in spreadsheet or maybe in a database, if that's what you're familiar with. And now items have attributes. And so like if a row, if the table's name is accounts and a row is a user in the accounts, it might have attributes like name, email and address. Partition key is how we query for a specific item or a collection of items. So partition key is kind of like a primary key and relational terms. It's usually a fairly unique value, but it doesn't have to be. And it's how we get stuff. A sort key is a secondary way to query. And so a sort key might be a way to narrow down the collection of items or a way to get a more exact representation of hierarchical data. So if you want to get like a lower leaf in the tree, as it were. And then the last concept is indexes. And indexes are actually, they're just basically like tables. They're another table with a different primary key, sort key schema so that you can query by different stuff. And I'll explain what I mean by that.

So basic modeling. A partition key is a unique value for accessing an item. So this right here is what we call an Arc file. And it's a document we use to provision DynamoDB tables in the architect open source project.

So we say, Hey, table, I want a table named accounts and I want a partition key of account ID. And I'm going to have an extra index for the accounts table. Oh, I have a typo here. That should be accounts with an S, that's terrible. But this should be accounts. And so it matches here, and it would be a different way to query. So I might want to get my user by account ID, but I might only have the email, such as when they're registering or signing in or logging into the system. So I need an index to query that by email. I like how my typo carried over to here, so sorry about that.

So basically in 101.2, so there's a primary key concept and then there's a secondary key concept or the sort key, sometimes called a secondary key. Sort key is for querying sort of one to many relations. So if I was modeling a blog in my system and I've got accounts and accounts have many blogs. And so one account ID would have more than one blog post. And we could have a secondary key of slug to get an individual posts. Now this is cool because we might also have to query blogs by when it was published. And so this is a secondary key in the index where we're gonna go by published instead. And so it's kind of nice. So now we can get all the blogs. We can get a blog by its title and we can get whether or not it's been published. Cool. So primary keys and sort keys, tables and indexes. That's like the most basic high-level stuff. There's a lot more to it. One of the things to know about DynamoDB and you hear this come up all the time. So I'm gonna address it now. What about single table design? So Amazon recommends that you model your application with as few tables as possible.

And ideally that means just one table and you can get really fancy with partition keys and sort keys to store a really complex relationships inside of your tables, inside of one table.

Now, I don't like to do this with ephemeral data and durable data. And what I mean by that is if I have API keys or tokens which I'm expiring with a time to live, I don't wanna store that in the same place where I have user accounts, which are durable data that I wanna keep around forever and maybe back up on a regular cadence. I don't wanna back up short-lived tokens. I want to back up user accounts. And so we don't wanna put those tables together. So you typically will have more than one table depending on the life cycle of the data involved.

Now, this is a personal opinion and everybody's got one. So take it for what it's worth. But I think single table design is an optimization step once you know your data access patterns. Until you know them, you should probably just lean towards trying stuff out and figuring out the best design for your app at the time. And that might mean that there's gonna be opportunities to improve it later. And that's not a bad thing. That's actually what will happen regardless of whether you design it for a single table or not.

Okay. That's a lot of hype. Let's get into modeling some stuff. So first of all, if you're gonna create a remix app to work with this, you're gonna wanna choose the Grunge Stack. That uses Architect Under the Hood, which gives you all these AWS superpowers. You don't have to do that though. And I'm gonna show you how to work with DynamoDB directly on your machine locally without an Amazon account because that makes things a lot easier. If you do check out the Grunge Stack, you will see in the generated code that it gives you this awesome note-taking application that is really close to how I would recommend you'd wanna work anyways.

So typically, you'd have some kind of folder called Models and you would wanna put all of your data access logic in those models. You don't want your application layer to be interacting with DynamoDB. You want it to be interacting with the entities of your system. And so in this case, the entities are User and Note. And so, whatever you wanna call them, whatever those things are that you're persisting, you wanna collect those into like an object so that your application layer reads really cleanly and it's more focused on business logic and not on data access logic. And that's really, really important to separate those concerns. And once this is all compiled, it'll all be mashed together in the right places. But if you sort of dig into like these models, you'll see that they kind of encapsulate all the ugliness of working directly with the database, which isn't that ugly.

But db.note.get with these PK and SK things is a lot less clear than the interface of just getNote, right? So that's a nicer way to work. Now, I was gonna go through this and just sort of explain how all this works, but I feel like it starts with almost too much context. And I don't wanna confuse you with all this extra business that's going on. I just wanna show you the most basic, basic stuff.

So with that, I'm actually just gonna go jump over to my terminal, and I should fix my example. Super funny. I'm gonna create a new tab and I'm gonna make a directory and I'm gonna call it Remix DynamoDB. And I'm gonna jump in there. I'm going to echo a pair of parentheses into a file called package.json, and I'm gonna NPM install tape, tap spec, architect sandbox, architect functions. And I'm gonna save them for development. It's gonna take a sec. We're all kind of used to the NPM tax, but it's worth maybe talking about what's going on here.

So I've just created a package.json file and I added some dependencies to it. Those dependencies are for testing, and that's it. And there we go. Now, I'm gonna touch a file called app.arc and modify it. I'm just gonna call it myapp. And we will start by defining a DynamoDB table called cats. And we'll give it a primary, or a partition key, sorry. I always wanna call it a primary key. We'll give it a partition key of cat ID. So we have, typically I like to name the tables in plural and then each row would be like a singular type thing. And so now with the way that Architect works, we'll be able to run this locally, which is really, really nice. But before I do that, let's go pop over to package.json. Cool, I got all my depths. So we'll just add a script. And and tap spec. Okay, so this is gonna run tape against a file called test.mjs. And it's gonna pipe the results to tap spec. Now test.mjs doesn't exist yet, so let's create that.

And we'll just add a silly test. Oops. Works. Now you can add all kinds of tooling to this, obviously, and you can get really, you know, buck nutty if you want, but I'm just trying to get this working. And I just want to like, do like, one little step at a time and see how it works, you know. Just taking baby steps. So we have one passing test that feels pretty good. But there's not a whole lot going on here.

So remember when I went into this app.arc file and I defined this table called cats. It would be cool if we could, you know, take a look at that. So why don't we do that? So before we do that, we're going to need the Arc sandbox. So architect ships with a DynamoDB runtime based on a thing called Dynolate, and it lets us run it locally, which is nice. So you don't have to deploy anything to the cloud to test out your DynamoDB. And I'm gonna say, now it runs like a dev server. So, you know, we have to start and stop it and we'll do that before and after our tests run. And I'm not sure why I get this screen thing going on. There we go. And end it. And we'll see some weird output when we do this. So let's take a look at that. Oh, missing a comma. I should install ESLint, but I don't want this to turn into a configure my stuff story. So you'll see here, we got some output. And when the sandbox started said tables created in local database, which is nice. Actually don't want to see that every time. I'm gonna tell it to be quiet true. And instead of just saying works, why don't we take a look at the architect DynamoClient. Architects are of a collection of small functions that you work with AWS. And so one of those functions is a data access layer. And so we're gonna say let data equals await Arc.tables.

So this is gonna reflect back in the tables that we defined in our arc file. And if we were to deploy this, that arc file would generate the cloud formation needed to create the database at AWS. So let's take a look at that.

So let's create a new data access layer. So we're gonna say, we're gonna create a new cloud formation, and then we're gonna say, we're gonna use this data access layer that we've written out. So we'll just create a new cloud formation. And we're gonna say, we're gonna create a new data access layer and we're gonna say this is the AX. And then we're gonna say, we're gonna invoke this data access layer and we're gonna do a new query.

The reason this is useful, I'm gonna just quickly jump over to the architect website and I'll show you. Here is on the left, an arc file, oops. And this on the right is the generated cloud formation for that arc file. And you'll see down here somewhere, it's gonna define a DynamoDB table. And that's great. We've got like scope and data and IDs and it's got a TTL on it. This table when it gets generated is gonna just have a meaningless GUID for a name. If I call it cats, for example, there's no cats in here. It's gonna be mashed in with what gets generated. So at runtime, we actually will look it up by key and allow you to have this Dynamo client that is a little more easy to work with. So you're just working with arc. or data.cats, as opposed to working with some GUID.

Okay, so we got our table. Let's write some data to it. So, write a cat. result equals await data.cats.put. And I think I said in the schema that it would have a cat ID. And I'll just, you know, this isn't intended to be production code. So, we'll just do that for the cat ID, give it a name. Sutro's my cat's name. Yeah, a birthday. Okay.

And instead of saying T.compass, we'll say T.OK result.cat ID has a cat log out result here so we can take a look at it. Cool. So, we wrote a cat to the database and then we exited right away and it's gone because you don't persist anything in our local file system sandbox. By default, this will actually just run in memory and it's just great because as you add more and more tests and you build out a bigger and bigger data access layer, the tests are gonna run fast. In our case, for begin.com, I think we have something like thousands of tests and they run in milliseconds, which is how you want this to work.

So, why don't we read, oops, read all cats and we can add a couple of cats. Instead, okay. Another cat in tuxedo, how about that? And I'm gonna, I'm gonna wanna get this ID later, so I'm just gonna say, my cat ID up here outside the scope of my test and I'm gonna copy this chunk of code up here. And so this cat ID will be outside of the scope and then later on I'll be able to look it up. But for now, let's just get all the cats.

So data.cats.scan, as cats. Okay, so scan is typically considered bad practice. Typically. So what scan will do is it'll go through and it'll read everything. And if your table is really big, that's gonna take a long time and it's gonna have to paginate through multiple pages of data. So typically you wanna grab things by querying or by directly giving it a key. So that's just the sloppy, fast way to work. But you know what? This also scales for quite a long time. So don't feel bad about using the tools you have available to you, and then iterating to improve it. Doesn't have to be perfect upfront.

So let's read one cat, data.cats.oops! You get, and we're gonna query by cat ID. So if we see here by cat ID, we pulled back this one, and this is just screaming fast, like we're just adding queries and no big deal. Reading and writing, all that takes place in a few milliseconds. Not bad, kind of exactly what you want. So as far as Crud goes, we've got create, we've got read, we've got a list with this, oops, with this scan business. Let's do a query. Just to round it out. And so this is kind of the, read cats by cat ID. So this is kind of the recommended way to work with with rows. Key, condition, expression. Now you can do this in multiple different ways, this is just sort of one way and there's a whole lot to unpack about how to learn how to do proper modeling and building up a larger application. I'm trying to give you a sense here, I'm not trying to say that this is the only way or the best way, this is a, I'm introducing you to Dynamo for the first time. So we have a key condition expression, we're saying, give me all the cats where the cat ID equals this value. And then we have to have an expression attribute value. And one of them is going to be cat ID and we'll pass the cat ID we generated above.

That's gonna get us a result and we'll inspect the value of that result. So you can see it's really actually quite similar to running a scan. But it was prefiltered on the ID that we were looking at. Now as you design and learn how to model out DynamoDB applications, you'll use IDs as a way to like query on groups of items in hierarchies. And this is how you get more than one item by ID. And you can add all kinds of conditions to this, it doesn't have to be too complicated.

I'm gonna do another one of these, say like cat, maybe two, and we'll use it for this one. And so read all cats by cat ID, has a bunch of cats, cool, cool, cool. Let's delete a row, so we'll have all the crud now, we've got create, read, update, delete. So destroy a cat, that sounds sad. Delete a cat row, that's a little more benign. I am a friend of the cats and I want to see them get hurt.

Okay, so I'm using tape by the way here, very deliberately. Sometimes people in the React community think you should use Jest, don't use Jest to test node projects, it's not very good for that, it's bloated, it's slow and it patches globals. You want something as close to a metal as possible for your tests. You don't want your tests to be a murder mystery, you want them to be a place where you can trust what's going on, what you thought was going on. All right, so let's grab some code from there, we're gonna get the cats, and we're gonna say delete, and it's just by cat ID, so we can just steal that code. And then we'll see if we get a result of some kind. And sadly, yes, we will. The right way to do this would actually be, let's do a scan afterwards, and we'll see how many items we have in the scan. No, it's the right way, but the right way to do this to learn, I should say. Call this one result. Oh, and delete the second one. Okay. I think I know where I messed up. It was back up here when it's creating this. Yeah. Nope, still have an error. Delete cat row, hot cats. Oh, read cat by cat ID.

I got one. And then delete row has cats. Let's just see what I got here. Oh, we deleted both cats, didn't we? I deleted this cat. And did I delete both cats? Where'd the other cat go? Going off script. Read cats by cat ID. We have a tuxedo, we have a tuxedo, tuxedo. We're in cat. That's interesting. It's almost like we overwrote our cat, up here. Did not cats.put, did not cats.put. Oh, I think we did. Because these might have happened so close that they shared an ID. Yeah. That's what happened. They shared an ID. Now this is obviously for demo purposes. You could use UUIDs here. Probably wanna use some other data like email and stuff too. But let's just see. So yeah, now we've got just one back. And it's doing what we expect. Cool. So that's a very fast tour of building a DynamoDB client in one file with three dependencies or four dependencies. As you were to mature this project, you would likely wanna have a thing called cats, right? And you wouldn't be interacting with the Dynamo code in here. You would be interacting with the cats code. And so the exercise for the reader would be to move this into some kind of model here and then your application would consume that model, would be tested independently and save your UI of your application and you wouldn't have to worry about how these things compose. They can compose nicely together as you build out your app. Okay, so that was like the highest level, fastest speed run of learning Dynamo I could throw at you. It's obviously only just barely scratches the surface.

A big recap here. So DynamoDB scales up and down dynamically. It's got a huge free tier, supports SQL like syntax. It's very easy to migrate, it works super well Lambda. Yes, it scales and yeah, it's a bit weird to learn. I'm out of time, so please follow me on the various socials at Bryan LaRue. If you want to give this a try, give the grunch stack a go, remix and then if you're having any problems, just join us at the Architect Discord. You can find us at arc.codes. We're always happy to help. Thanks so much.

Let's have Bryan join me for Q&A and let's first go over the poll that we had which is, what database do you use? And we see the results. Most people chose Postgres, DynamoDB did beat out MongoDB and then nobody chose other. So very interesting response and I know we were talking earlier and you kind of said like, oh, everybody just wants to play it safe and stuff like that. Keeping all the jokes aside, I do want to talk about, how does DynamoDB complement Remix in your opinion? Why is it such a great combination? Yeah and I didn't want my talk to be like, here's how to build a forum with Remix. I felt that everybody kind of understood that part but to me the most exciting part about Remix is that we've got loaders and actions co-located with our forums and those forums are real database forums. So they're going to talk to a backend directly. And so the hard part then becomes, what's that forum going to do? And how do I talk to a database? In my view, Dynamo is probably the best in class of managed databases today. I get into that pretty heavy in the talk but like just empirically speaking it's got probably the best performance and the best characteristics for scaling, maybe not the best pricing characteristics and tools like Postgres are great and they're fine and they're gonna get you a really long way. And I think it's totally a safe choice everyone's using a Postgres. But the problem with the Postgres is it doesn't speak to things with ephemeral compute that well. So things that are talking through Cloud Functions and Remix is really designed to work on the edge with Cloud Functions. And so it's gonna work better with a key value store like a Dynamo or Mongo. I'm actually surprised that Dynamo beat Mongo, maybe it's just cause it was a Dynamo talk but Dynamo is still a bit new, it's also still in this like corner around AWS and not everybody's super comfortable going to AWS. They want higher level abstraction. And so it doesn't surprise me that we'd see these kinds of scores in the poll and yeah, and I think Remix users would be, whatever it is you choose to put in your loader or action, whether it's Postgres or a DynamoDB, that that's the beauty of Remix is it you get this nice layer that makes data access patterns really slick, clean.