Scaling Databases For Global Serverless Applications

Rate this content
Bookmark

This workshop discusses the challenges Enterprises are facing when scaling the data tier to support multi-region deployments and serverless environments. Serverless edge functions and lightweight container orchestration enables applications and business logic to be easily deployed globally, often leaving the database as the latency and scaling bottleneck.


Join us to understand how PolyScale.ai solves these scaling challenges intelligently caching database data at the edge, without sacrificing transactionality or consistency. Get hands on with PolyScale for implementation, query observability and global latency testing with edge functions.


Table of contents

  • - Introduction to PolyScale.ai
  • - Enterprise Data Gravity
  • - Why data scaling is hard
  • - Options for Scaling the data tier
  • - Database Observability
  • - Cache Management AI
  • - Hands on with PolyScale.ai

This workshop has been presented at Node Congress 2022, check out the latest edition of this Tech Conference.

FAQ

Polyscale is a no-code database edge cache that distributes data and query compute closer to the user. It functions by caching database data at the edge or closer to the user and also offers the compute itself. This means if you have a unique SQL query, it will execute against Polyscale's Points of Presence (POPs) and return the data from Polyscale if it is in the cache.

To connect a database to Polyscale, update your database credentials such as the host name and database username to route through Polyscale. For MySQL and MariaDB, include your cache ID in the database username. For Postgres, pass the cache ID as the application name parameter.

As of the information provided, Polyscale currently supports MySQL, MariaDB, and Postgres. Support for Microsoft SQL Server is planned and forthcoming, while MongoDB support is also under consideration for future updates.

Polyscale can potentially replace Redis or other application-level caching solutions, especially in scenarios where you want to reduce complexity and avoid managing caching logic at the application level. Polyscale manages caching at the data tier, which can simplify operations.

Polyscale caches DQL (Data Query Language) queries, which include 'select' and 'show' queries. It automatically handles and optimizes caching for these queries based on frequency and other factors using AI and machine learning technologies.

Polyscale ensures data security by not storing database usernames or passwords, and all data remains SSL encrypted during transit. Users can also deploy Polyscale within their own networks for enhanced compliance and control, ensuring that cached data never leaves their environment.

Yes, Polyscale allows manual configuration of caching rules where you can specify custom rules for individual SQL queries or set default behaviors for caching. This provides flexibility in managing how data is cached based on specific application needs.

Polyscale offers significant benefits for global applications by reducing latency and improving data access speed across different regions. It caches data closer to users and distributes query compute globally, which helps in maintaining high performance even when scaling applications internationally.

Ben Hagan
Ben Hagan
83 min
11 Feb, 2022

Comments

Sign in or register to post your comment.

Video Summary and Transcription

Polyscale is a no-code database edge cache that aims to make caching and speeding up the data layer effortless for developers. It uses AI and machine learning to adjust caching behavior, selectively choosing what to cache and automatically setting cache duration. Polyscale supports multiple databases and provides detailed observability. It eliminates the need for caching at the application tier and offers significant performance gains.

1. Introduction to Polyscale and Scaling Data

Short description:

Welcome to this Node Congress talk on scaling databases for global serverless applications. Today, I'm going to introduce you to Polyscale and talk about the challenges in managing and scaling data. We'll also explore different options for scaling database systems and dig into how Polyscale works. Feel free to ask questions and get hands-on with the code.

♪♪ Yeah, welcome, everybody. I can still see a few people coming in, but we'll get kicked off anyway, so welcome to this Node Congress talk on scaling databases for global serverless applications. Let's just show the agenda for today. Just a couple of housekeeping rules, everybody feel free to use the chat, you can also unmute, there's quite a lot of people on this, so stay muted if you're not talking, but please unmute yourself, ask questions, let's make this as interactive as possible, because I think there's lots we can learn by collaborating our questions.

So, yeah, feel free to use the chat, but obviously, unmute and introduce yourself and, by all means, jump in at any point if you have questions. Those are very welcome. So, yeah, hopefully, everybody can hear me OK, and this is our agenda for today. A quick bit about myself, I'm the founder of PolyScale and my background's really been in working with sort of large data companies and database companies. And I've sort of been in the solution architecture and sales engineering space for many years. And sort of the last few companies I was at, you know, Alfresco was an open source document management company. So we dealt with very large amounts of data repositories for specifically the documents of web content management. DataSift was one of the early adopters of the Twitter firehose and ingested all of Twitter's data and later on, that worked with Facebook and LinkedIn to ingest all of that data and serve insights in a privacy safe way. And then from there, I moved to Elastic here in the Bay Area, in San Francisco, where I was working with some large enterprises with the Elastic Solution and scaling those and then before founding Polyscale, I was at a company called Rockset, which is a cloud-based Database for realtime analytics. So throughout those last few years, I've always been focused on managing large volumes of data and understanding how to scale those globally. That's really been the challenges I've been working on.

But today, we've got a great talk lined up. I'm going to introduce you to Polyscale and we'll get hands on with that for people who want to actually test it out. And I'll talk a bit about the challenges really in the enterprise and small businesses as well around managing data and scaling data. And why is that hard? Why is it hard to move data to different regions and to keep and maintain your transactionality? And so we're also going to talk about what are your options, what do people do today to scale these database systems and data platforms. And then, as I mentioned, we'll dig into Polyscale and what Polyscale is and how it works. And as I say, everyone's welcome to get hands on. We've got some code to play with and get started with. So, yeah, as I say, everybody who would, you know, any questions, feel free to chime in, unmute yourself, introduce yourself, you know, very welcome to take questions.

2. Challenges of Scaling Data and Data Gravity

Short description:

Polyscale exists to address the challenges of scaling stateful cloud-native data. Scaling databases to different countries or independently presents difficulties, from CAP theorem considerations to maintaining transactionality and managing data sharding. Accessing data across continents in a low latency environment is also complex. Data gravity, the creation of data across multiple systems, poses challenges for enterprises. Applications and users require access to data from different locations, making data movement difficult. Location, data regulations, and global growth further complicate the situation. However, there are options for deploying web apps and replicating static content and business logic locally with companies like Netlify and Vercel. CDNs and containerization tools like Fly and Section provide flexibility in deploying to multiple regions.

So, really, the premise for, you know, why Polyscale exists and why, you know, our focus is really scaling stateful cloud-native data is hard. So, anyone who's been in a role where you've had to, you know, take a database, for example, and scale that to different countries, or even just scale it independently, you know, you'll understand the challenges of doing that. And there's a whole bunch of reasons around like why that is hard, from things along the CAP theorem lines to, you know, maintaining my transactionality, you know, how do I shard my data, where does that reside? And then how do I sort of manage just the laws of physics around accessing that in a low latency environment across continents, for example?

So, great example that we've got people from all parts of the planet dialed in today. And making data fast for everyone is a hard and complex thing, and specifically here, we're talking also about cloud native data. So, what happens when you are in a situation where, you know, you have a data center outage, where one region goes down, or a part or component of your architecture goes down. You know, how do you support those environments? And that's what makes it hard. It's maintaining that state and it's doing that in an unforgiving environment like the cloud.

So, you know, I talk a lot about this concept of enterprise data gravity. And you know, kind of what is that, and how does it affect us? The reality is data is being created everywhere. And it can be across multiple disparate systems. And inside a single enterprise, there are likely many, many different data persistence layers. It could be things from data lakes, databases, warehouses. And that stuff is being created all day every day. It's just an ongoing thing. With sort of AI and machine learning, sensor data and IoT, that's just accelerating massively. So what tends to happen is, within enterprises, more and more applications want to use that data. And people are requesting access to that data from different departments, from different locations. And that then gives us this inability to move the data. We find ourselves kind of shackled to, once somebody is using... Let's say we have a large transactional database that says Postgres, and that's running from a data center in New York. And we now have, you know, large numbers of users actually actively using that. It becomes harder and harder to kind of access that data in different locations and move that data. So location becomes critical depending on where your users are. And then the other sort of big consideration around data and usages, obviously, what are the data regulations around this? Things like privacy and compliance, and PII data, what are you storing in what countries? You know, is it GDPR compliant? And this is becoming more and more what is incredibly critical today, but it's only becoming more critical as we as we progress with these data systems. And then lastly, you know, we've got to consider global growth. So it could be a simple web application. We find that we now have a growing audience that are spanning different countries or parts of countries. And how do we make sure they can access that data and how do we deal with that global growth? And the same within sort of a non-customer facing application. Maybe it's something like a be a environment whereby people are accessing internal intelligence data using tools like click tech and tableau. So same types of problems. We're putting new offices in different locations. People are working from home, working remotely. How do we support that growth? And that's really the essence of the problems that come with sort of data gravity.

I think probably a lot of people on this this talk will be very familiar with the application tier has a lot of agility. And so I think that's all the time I have today. Like if I said to everyone on this call, like, how would you deploy a web app? I think we have a lot of different options. And if I said, how do I deploy a web app to different regions? I think there's a huge number of options as well. And I kind of think about this, the content and the code. So static content and business logic can easily be replicated locally. And when I think about, if I had a server where we have these amazing companies like, you know, Netlify and Vercel, who can, you know, push my application logic and my static content everywhere. And I can run functions at the edge. And I can knit that together with, you know, a simple load balancing plan that will allow me to resolve to my closest application server. So it's really quite achievable now for any size of business or individual. Just so, you know, an indie hacker to be able to set up global load balancing. We can deploy to CDNs for our static content and our functions. And we're also seeing the ability to, you know, these new players pop up like Fly and Section who allow you to containerize your code and push that everywhere into multiple regions. So as far as the business logic and your static content goes, there really is a huge amount of flexibility. And that's a great space to be in. I'll just pause there. Any questions before we move on? Anything in the chat or anyone wants to chime in with a question? Good.

QnA

Watch more workshops on topic

AI on Demand: Serverless AI
DevOps.js Conf 2024DevOps.js Conf 2024
163 min
AI on Demand: Serverless AI
Top Content
Featured WorkshopFree
Nathan Disidore
Nathan Disidore
In this workshop, we discuss the merits of serverless architecture and how it can be applied to the AI space. We'll explore options around building serverless RAG applications for a more lambda-esque approach to AI. Next, we'll get hands on and build a sample CRUD app that allows you to store information and query it using an LLM with Workers AI, Vectorize, D1, and Cloudflare Workers.
Node.js Masterclass
Node Congress 2023Node Congress 2023
109 min
Node.js Masterclass
Top Content
Workshop
Matteo Collina
Matteo Collina
Have you ever struggled with designing and structuring your Node.js applications? Building applications that are well organised, testable and extendable is not always easy. It can often turn out to be a lot more complicated than you expect it to be. In this live event Matteo will show you how he builds Node.js applications from scratch. You’ll learn how he approaches application design, and the philosophies that he applies to create modular, maintainable and effective applications.

Level: intermediate
Build and Deploy a Backend With Fastify & Platformatic
JSNation 2023JSNation 2023
104 min
Build and Deploy a Backend With Fastify & Platformatic
WorkshopFree
Matteo Collina
Matteo Collina
Platformatic allows you to rapidly develop GraphQL and REST APIs with minimal effort. The best part is that it also allows you to unleash the full potential of Node.js and Fastify whenever you need to. You can fully customise a Platformatic application by writing your own additional features and plugins. In the workshop, we’ll cover both our Open Source modules and our Cloud offering:- Platformatic OSS (open-source software) — Tools and libraries for rapidly building robust applications with Node.js (https://oss.platformatic.dev/).- Platformatic Cloud (currently in beta) — Our hosting platform that includes features such as preview apps, built-in metrics and integration with your Git flow (https://platformatic.dev/). 
In this workshop you'll learn how to develop APIs with Fastify and deploy them to the Platformatic Cloud.
Building a Hyper Fast Web Server with Deno
JSNation Live 2021JSNation Live 2021
156 min
Building a Hyper Fast Web Server with Deno
WorkshopFree
Matt Landers
Will Johnston
2 authors
Deno 1.9 introduced a new web server API that takes advantage of Hyper, a fast and correct HTTP implementation for Rust. Using this API instead of the std/http implementation increases performance and provides support for HTTP2. In this workshop, learn how to create a web server utilizing Hyper under the hood and boost the performance for your web apps.
0 to Auth in an Hour Using NodeJS SDK
Node Congress 2023Node Congress 2023
63 min
0 to Auth in an Hour Using NodeJS SDK
WorkshopFree
Asaf Shen
Asaf Shen
Passwordless authentication may seem complex, but it is simple to add it to any app using the right tool.
We will enhance a full-stack JS application (Node.JS backend + React frontend) to authenticate users with OAuth (social login) and One Time Passwords (email), including:- User authentication - Managing user interactions, returning session / refresh JWTs- Session management and validation - Storing the session for subsequent client requests, validating / refreshing sessions
At the end of the workshop, we will also touch on another approach to code authentication using frontend Descope Flows (drag-and-drop workflows), while keeping only session validation in the backend. With this, we will also show how easy it is to enable biometrics and other passwordless authentication methods.
Table of contents- A quick intro to core authentication concepts- Coding- Why passwordless matters
Prerequisites- IDE for your choice- Node 18 or higher
GraphQL - From Zero to Hero in 3 hours
React Summit 2022React Summit 2022
164 min
GraphQL - From Zero to Hero in 3 hours
Workshop
Pawel Sawicki
Pawel Sawicki
How to build a fullstack GraphQL application (Postgres + NestJs + React) in the shortest time possible.
All beginnings are hard. Even harder than choosing the technology is often developing a suitable architecture. Especially when it comes to GraphQL.
In this workshop, you will get a variety of best practices that you would normally have to work through over a number of projects - all in just three hours.
If you've always wanted to participate in a hackathon to get something up and running in the shortest amount of time - then take an active part in this workshop, and participate in the thought processes of the trainer.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

It's a Jungle Out There: What's Really Going on Inside Your Node_Modules Folder
Node Congress 2022Node Congress 2022
26 min
It's a Jungle Out There: What's Really Going on Inside Your Node_Modules Folder
Top Content
The talk discusses the importance of supply chain security in the open source ecosystem, highlighting the risks of relying on open source code without proper code review. It explores the trend of supply chain attacks and the need for a new approach to detect and block malicious dependencies. The talk also introduces Socket, a tool that assesses the security of packages and provides automation and analysis to protect against malware and supply chain attacks. It emphasizes the need to prioritize security in software development and offers insights into potential solutions such as realms and Deno's command line flags.
Towards a Standard Library for JavaScript Runtimes
Node Congress 2022Node Congress 2022
34 min
Towards a Standard Library for JavaScript Runtimes
Top Content
There is a need for a standard library of APIs for JavaScript runtimes, as there are currently multiple ways to perform fundamental tasks like base64 encoding. JavaScript runtimes have historically lacked a standard library, causing friction and difficulty for developers. The idea of a small core has both benefits and drawbacks, with some runtimes abusing it to limit innovation. There is a misalignment between Node and web browsers in terms of functionality and API standards. The proposal is to involve browser developers in conversations about API standardization and to create a common standard library for JavaScript runtimes.
ESM Loaders: Enhancing Module Loading in Node.js
JSNation 2023JSNation 2023
22 min
ESM Loaders: Enhancing Module Loading in Node.js
ESM Loaders enhance module loading in Node.js by resolving URLs and reading files from the disk. Module loaders can override modules and change how they are found. Enhancing the loading phase involves loading directly from HTTP and loading TypeScript code without building it. The loader in the module URL handles URL resolution and uses fetch to fetch the source code. Loaders can be chained together to load from different sources, transform source code, and resolve URLs differently. The future of module loading enhancements is promising and simple to use.
Out of the Box Node.js Diagnostics
Node Congress 2022Node Congress 2022
34 min
Out of the Box Node.js Diagnostics
This talk covers various techniques for getting diagnostics information out of Node.js, including debugging with environment variables, handling warnings and deprecations, tracing uncaught exceptions and process exit, using the v8 inspector and dev tools, and generating diagnostic reports. The speaker also mentions areas for improvement in Node.js diagnostics and provides resources for learning and contributing. Additionally, the responsibilities of the Technical Steering Committee in the TS community are discussed.
You Don’t Know How to SSR
DevOps.js Conf 2024DevOps.js Conf 2024
23 min
You Don’t Know How to SSR
The Talk covers the speaker's personal journey into server-side rendering (SSR) and the evolution of web development frameworks. It explores the use of jQuery for animations in SSR, the challenges faced in integrating React with Umbraco, and the creation of a custom SSR framework. The Talk also discusses the benefits of Next.js and the use of serverless artifacts for deployment. Finally, it highlights the features of Astro, including its function per route capability.
Adopting GraphQL in an Enterprise
GraphQL Galaxy 2021GraphQL Galaxy 2021
32 min
Adopting GraphQL in an Enterprise
Today's Talk is about adopting GraphQL in an enterprise. It discusses the challenges of using REST APIs and the benefits of GraphQL. The Talk explores different approaches to adopting GraphQL, including coexistence with REST APIs. It emphasizes the power of GraphQL and provides tips for successful adoption. Overall, the Talk highlights the advantages of GraphQL in terms of efficiency, collaboration, and control over APIs.