Demystifying IPFS: A Web Developer's Guide to Content Distribution

Welcome to Demystifying IPFS, and thank you for joining JS Nation and this talk. So the way that this talk is going to be structured is into three parts. The first part, we're going to talk about the web platform. And then we're going to move on to IPFS and where that fits in with the web. And finally, we're going to take a look at IPFS for JS developers and how you can actually use IPFS in practice.

The web is amazing. The web platform empowers billions of people, and it has the widest reach with over 5 billion users across all continents via browsers or web views and mobile apps. It also is the most ubiquitous platform which practically works on every device spanning from your desktop, smartphone, laptop, TVs, and even fridges these days. And finally, it's also an open ecosystem where anyone can publish content or build sites with minimal gatekeeping.

But the web has its flaws. Link rot is a big problem on the web whereby links are no longer functional. So there's research that shows that one in five scientific papers has at least one broken link within a few years of publication. In fact, other research suggests that 30 to 70% of web links rot within a couple of years from getting published. There's also the problem of censorship. A great example of this is the censorship of Wikipedia, of which there are many examples in countries like China, Turkey, India, Russia, and other authoritarian regimes. And the nature of the web is such that it's possible for these state actors to block large swaths of the Internet.

There's also the problem of how security is anchored to origin, which means that you lack tamper proofing. A great example for this is when you go to your online banking web app, how do you know that it hasn't been tampered with? The way that this works on the web is using the same origin policy, whereby essentially your browser validates the certificate authority signed certificate that is presented to you by your online banking. So essentially you're trusting who rather than what you're actually getting. This is in contrast to how package managers implement tamper proofing using hashes and signatures. Now the same origin policy of course delineates the trust and security boundaries, and that's typically tied to the protocol and the domain name. But the key idea here is that you're trusting who rather than what.

The one exception to this of course is subresource integrity, which is a relatively recent feature that was introduced in order to allow importing packages from other origins and having an integrity attribute which validates essentially the hash of that script that you might be importing. This has been a great introduction. Of course the root of a lot of these problems is in the fact that the web is location addressed. Location addressing is foundational to the web and how the Internet works. Every network device has an IP, and DNS is used to map human friendly names to IPs. Now this works extremely well because the network makes it possible to find multiple paths between any two IPs on the Internet. But this is also the cause of many of these problems because essentially for a domain name you typically have a single source, and as we mentioned before, IPs and DNS can be easily blocked by state actors. If a copy of data or a website disappears, there's no way to fundamentally know where other copies might be. Moreover, it's not possible for you to verify the integrity of the content that you're getting, like in this sort of visualization or diagram that we have here.

This brings us to the second part of this talk, IPFS. IPFS is a set of protocols for addressing and sharing data on the web. It has popular implementations in TypeScript and Go, and it is designed to work natively on the web. What do we mean by sharing? Sharing typically means two things, publishing and retrieving. There are two core ideas behind IPFS that are key to understanding IPFS. The first is content addressing, and the second is peer-to-peer networking. But it's worth flagging that IPFS fuses a lot of ideas from Git, the source code management tool, and BitTorrent, the peer-to-peer file sharing. Content addressing, which is in contrast to location addressing, is this idea that you derive the address of content from the content itself, using cryptographic hashes, rather than from the location where it is stored, as is common with Ahrefs and links on the web. Hash functions are functions that take arbitrary length data and produce a fixed length fingerprint of that data. Content addressing is not a new idea; it is very prevalent in a system known as Git, whereby each commit hash is a hash of the repo's full file tree at the time of the commit. PNPM also uses a content addressed store to deduplicate dependencies and make package management faster in the JavaScript ecosystem.

Hash functions are functions that take arbitrary length data and produce a fixed length fingerprint of that data. Now content addressing is not a new idea. In fact, it is very prevalent in a system known as Git, which I've previously mentioned, whereby each commit hash is a hash of the repo's full file tree at the time of the commit. PNPM also uses a content addressed store to deduplicate dependencies and make package management faster in the JavaScript ecosystem.

Now the way that IPFS implements content addressing is using content identifiers, commonly referred to as SIDs. A SID is essentially a hash which contains some additional metadata, like the codec and the hash function and the length of that hash. This is a property that of course allows it to be upgradable and self-describing. Now the typical URI scheme that is used is IPFS colon slash slash and then a SID. And of course a SID can be represented in binary form or in string form. And the example that we have here with the IPFS URI is in fact a string representation.

The key benefit that content addressing gives you is that it enables you to have multiple providers for a piece of data. So a provider is essentially just an IPFS node that is reachable and is serving data that is addressed by that SID. Of course you still need to fetch from a location, so content addressing can be thought of as something that is overlaid over a location. Addressed internet, which still uses IPs and ports. So there's no magic here, you still need to fetch from a location. And this is where the Cademlia DHT comes into play here. So the way that you go from a SID to an actual IP of a node that has the data for that SID, is using this Cademlia DHT. Which is quite complex to understand but we're not going to get into all of the details of that now.

But the key idea is that the DHT essentially allows you to go from a SID to a provider or multiple providers. As we have in the diagram here we have a JPEG that has three providers. So how do you encode data in IPFS? Because you can't just for example hash a directory. UnixFS is the format that is used in IPFS in order to encode files and directories. And the idea with UnixFS is that everything is essentially a block that is addressed by a SID. And large files are chunked into smaller chunks, which then comprise a larger data structure that is known as a Merkle DAG.

This is the same data structure that is used by Git. And the benefit of a Merkle DAG is that you can have a single SID that represents a whole file tree. And in this example it's a very minimal example of a build directory of a static website. And you can see that it has an index.html file and some static assets and a style.css. The reason that I use this as an example is because this emphasizes how UnixFS and in general IPFS is ideal for publishing static websites and apps. Where you essentially just take your build output and encode it with UnixFS, thereby getting a single SID that represents the whole website.

Now the second key principle idea behind IPFS that is important to understanding IPFS is peer-to-peer networking. And the key idea behind peer-to-peer networking is that every client is also a server. Now there's a little star there because web browsers turn out to be very bad servers. You're very constrained in terms of what you can do in a browser. So it's useful to sometimes think about IPFS as being a network in which you have some nodes which are just doing retrieval because they can't do the job of serving. Now peer-to-peer networking in principle compliments content addressing because as I mentioned you can have multiple providers. And if you have multiple providers for a SID, you want to be able to connect to them directly.

You also do away with this idea of canonical or trusted servers that have a certificate that has been given by a certificate authority that has authenticated their identity as we have with the traditional web. Instead we take the approach of verifying and treating everyone with a minimal level of trust. Now peer-to-peer networking is hard. It's hard because networking is hard and you have NATs and you have firewalls and you also have limited transports in browsers which I mentioned in the beginning. So a lot of this stuff is actually handled by a library that's spun out of IPFS called Lip2P.

And it handles a lot of the magic of NAT traversal and overcoming firewalls and of course implements this Cademlia DHT, the distributed hash table, where we store the information of who has what. So what benefits do we get from IPFS and these ideas of content addressing and peer-to-peer networking? Well the first is we get censorship resistance and resilience because we can have multiple providers. We prevent link rot because these SIDs, they represent the data. Now this doesn't guarantee you that you actually have providers for data.

So it doesn't guarantee you the persistence of data. But it ensures that the links actually represent the data so that you can always bring it back and then those links are fixed so they don't break. You also get tamper-proofing because you're essentially verifying every piece of content that you get back. And because everything is immutable, because it's hashed, it's highly cacheable, which enables you to do things like local first static websites, which we're going to take a look at in a moment. And of course you get deduplication across data sets. This is also the same feature that is in Git.

So Git uses these Merkle DAGs that essentially deduplicate data that is persistent across commits. DNS link is another technology that is quite important to IPFS because SIDs are not very human-friendly. They're long and they're not very memorable. What DNS link allows you to do is it introduces mutability into IPFS and allows you to link DNS names, domains, to SIDs. And you do that using a simple TXT record at the underscore DNS link dot the domain. And you store in that record a path to the SID that you have.

This brings us to the third and final part of this talk, IPFS for JS developers. Of course, after all this is JS Nation, a JavaScript conference. So let's talk about the three key operations that you have in IPFS when you're interacting with the system. The first one is addressing data by SID. So you take input data and you encode it such that it's addressed by a SID. Under the hood, of course, it's Merkle-ized into this Merkle DAG data structure. But of course, a lot of that is handled by the libraries and SDKs that IPFS has. The second operation is providing or also often known as pinning, where you run a node that announces itself as a provider for the SID or multiple SIDs. And it serves that data over the network. It's by analogy similar to running an HTTP server, but obviously a bit more involved because you have to actively announce the data to this DHT.

The third operation is retrieval. And retrieval by SID, whereby you fetch data using the SID as an address. And this can happen using two main protocols. The first one is bit swap, which is a block exchange protocol, which happens over these libPTP connections. Or, as we'll take a look at in more detail in a moment, HTTP from gateways. Now, I've sort of hinted at this earlier in the talk. IPFS on the web is pretty hard. The web platform is amazing, but it is heavily constrained, which means that if you're running an application in a browser, you're limited in terms of the number of connections that you can open. And usually when you want to connect to a server, if you're running in a secure context, you need the server to have a TLS certificate. And to have a TLS certificate, you typically need a domain. And finally, you're limited with the transport, so you can't just open arbitrary TCP or UDP connections. You're constrained to HTTP, web transport, web sockets, and web RTC. So this brings us to IPFS HTTP gateways and how they came to be. IPFS HTTP gateways are an HTTP API for retrieving data from IPFS nodes.

And probably the most common example of this is IPFS.io, which is a public good CDN, essentially, that we provide. And you can just pass it an IPFS path with the SID, and it will abstract all of the complexities of peer-to-peer and IPFS in a browser, and essentially do the retrieval for you and just hand it back to you. Now, this turned out to be a huge success for the IPFS ecosystem over the last 10 years since IPFS was conceived, but it turns out that this was a double-edged sword. It was very convenient, and developers loved this, but it turned IPFS.io and similar gateways into a point of centralization, thereby eliminating and undermining many of the core principles of IPFS.

Having multiple providers and an open and decentralized network. This is why one of the big initiatives that we're currently working on is to eliminate the need for these centralized gateways, and the way that we're approaching this is by essentially turning every IPFS node into an IPFS gateway that also has a TLS certificate so it can be reached directly from a web browser. And there's a number of ways that we're doing this, which I won't go into great detail, but feel free to reach out or read some of our blog posts that go into more details around web transports and how we approach this problem. But the key idea is that, as we speak, it's becoming increasingly possible to do direct retrieval from providers without going through a centralized gateway.

Now I want to introduce you to Helia. A lot of this was a long preamble to get to the meat of this talk, which is Helia. Helia is a lean and modular TypeScript implementation of IPFS, and it is intended for Node.js and the web, so it uses all of the standard web APIs. You can check it out on GitHub, and here we just have an example where I show you how you can just take text and essentially encode it as a file with UnixFS. You get the SID for that, and you provide that to the network, and then you can actually use that SID in order to retrieve it directly from this Helia node that is running in Node.js.