Apache Kafka Simply Explained With TypeScript Examples

Rate this content
Bookmark

You’re curious about what Apache Kafka does and how it works, but between the terminology and explanations that seem to start at a complex level, it's been difficult to embark. This session is different. We'll talk about what Kafka is, what it does and how it works in simple terms with easy to understand and funny examples that you can share later at a dinner table with your family.


This session is for curious minds, who might have never worked with distributed streaming systems before, or are beginners to event streaming applications.


But let simplicity not deceive you - by the end of the session you’ll be equipped to create your own Apache Kafka event stream!

This talk has been presented at JSNation 2023, check out the latest edition of this JavaScript Conference.

FAQ

Apache Kafka is an event streaming platform that is distributed, scalable, high-throughput, low-latency, and has an amazing ecosystem and community. It handles the transportation of messages across multiple systems, including microservices, IoT devices, and more.

Apache Kafka is known for being distributed, scalable, high-throughput, low-latency, and having a strong ecosystem and community. It can handle trillions of messages per day and store data persistently across multiple servers.

Apache Kafka simplifies the handling of real-time data by untangling data flows and supporting real-time monitoring, processing, and reporting. It uses a push-pull model where producers push data into the cluster, and consumers pull data from the cluster.

Event-driven architecture in Apache Kafka involves handling data as a continuous flow of events rather than static objects. This allows for real-time data processing, replaying events, and answering complex queries based on the event stream.

In Apache Kafka, producers are applications that create and push data into the cluster, while consumers are applications that pull and read data from the cluster. Producers and consumers can be written in different programming languages and run independently.

Apache Kafka ensures data persistence and reliability by storing data on multiple servers (brokers) with replication. This means if any server goes down, the data is still available. Data is stored persistently on disks and can be read multiple times by different applications.

A topic in Apache Kafka is an abstract term for a set of events that come from one or more sources. It can be seen as a table in a database, with messages ordered by offset numbers. Topics are divided into partitions for distributed storage.

Apache Kafka uses partitions to split topics into chunks, each with its own offset numbers. It ensures data ordering within partitions using keys, such as a customer ID. Data is replicated across brokers to ensure reliability.

Common use cases for Apache Kafka include real-time data streaming for e-commerce platforms, monitoring and reporting systems, IoT device data handling, and any application requiring high-throughput, low-latency message transportation.

Apache Kafka supports different programming languages by allowing producers and consumers to be written in various languages. This flexibility helps in integrating Kafka with diverse systems and applications.

Olena Kutsenko
Olena Kutsenko
27 min
01 Jun, 2023

Comments

Sign in or register to post your comment.
Video Summary and Transcription
Apache Kafka is a distributed, scalable, and high-throughput event streaming platform that plays a key role in event-driven architecture. It allows for the division of monolithic applications into independent microservices for scalability and maintainability. Producers and consumers are the key components in Kafka, allowing for a decoupled system. Kafka's replication and persistent storage capabilities set it apart from alternatives like Redis and RabbitMQ. Kafka provides easy access to real-time data and simplifies real-time data handling.

1. Introduction to Apache Kafka and Shoputopia

Short description:

Hello everyone. Today I wanted to talk to you about Apache Kafka, an amazing project that has become the default standard for data streaming. Let me give you an example of how Apache Kafka can make a significant difference in a project. Imagine building an e-commerce product based on the movie Zootopia, called Shoputopia. As the project grows, it's important to avoid putting everything into a single monolith. Instead, we should consider dividing the monolith into independent microservices to ensure scalability and maintainability.

Hello everyone. My name is Elena. I work at Ivan where we support and contribute a lot to open source projects. Today I wanted to talk to you about one of those amazing projects which exists already for over a decade and became default standard for data streaming.

This is obviously Apache Kafka. But before we give a definition for Apache Kafka, I wanted to give you an example of a project where Apache Kafka makes a significant difference both to the users of the system as well as to developers. And my ingenious project idea is based on an animation movie which you might have seen, Zootopia. If you haven't seen it, no worries. However, if you have, you will recognize some of our characters because today, you and me, we are going to build the first e-commerce product of Zootopia and we'll call it Shoputopia. And like in any e-commerce project, we want to have some inventory of products. We are going to sell some simple user interface to start with where our lovely customers will be able to search for products, select what they need, put an order and wait for delivery.

And at start, maybe during MVP stage, you might be tempted to put everything into a single monolith where your frontend and your backend will be next to each other. You will have some data source there as well, and there is nothing bad about monoliths per se. However, once you have more customers and your shop becomes more popular and you start adding more and more modules into this monolith, very soon the architecture flow and the information flow of the system have a risk to become a mess. A mess that is difficult to support and difficult to expand. And assuming our development team is growing, no single individual will be able to keep up with the information flow of the system. And you might have been on those shoes when you are joining a project and they bring you the architecture, you're like, Oh my God, how do I navigate it? Whom I should talk to to understand this whole system? At this point of time, we'll have to make a tough conversation on how we can divide our monolith into a set of independent microservices with clear communication interfaces.

2. Importance of Real-Time Data and Apache Kafka

Short description:

Our architecture needs to rely on real-time events for meaningful recommendations. We also want easy access to real-time data without over-complicating our lives. That's where Apache Kafka comes in, untangling data flows and simplifying real-time data handling.

What's even more crucial, our architecture must be as close to real time communication as it is possible to rely on real time events so that our users don't have to wait till tomorrow to get meaningful recommendations based on their purchases done today or yesterday. What is also important would be really cool to have a support for real time monitoring, processing and reporting that is coming as a set package of functionality.

Also as engineers, we want to get the work with real-time data in an easy fashion, which doesn't really over-complicate our life. And this is a lot to ask, however, that's why we actually have Apache Kafka and Apache Kafka is great at untangling data flows and simplifying the way that we handle real-time data.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

React's Most Useful Types
React Day Berlin 2023React Day Berlin 2023
21 min
React's Most Useful Types
Top Content
Watch video: React's Most Useful Types
Today's Talk focuses on React's best types and JSX. It covers the types of JSX and React components, including React.fc and React.reactnode. The discussion also explores JSX intrinsic elements and react.component props, highlighting their differences and use cases. The Talk concludes with insights on using React.componentType and passing components, as well as utilizing the react.element ref type for external libraries like React-Select.
TypeScript and React: Secrets of a Happy Marriage
React Advanced 2022React Advanced 2022
21 min
TypeScript and React: Secrets of a Happy Marriage
Top Content
React and TypeScript have a strong relationship, with TypeScript offering benefits like better type checking and contract enforcement. Failing early and failing hard is important in software development to catch errors and debug effectively. TypeScript provides early detection of errors and ensures data accuracy in components and hooks. It offers superior type safety but can become complex as the codebase grows. Using union types in props can resolve errors and address dependencies. Dynamic communication and type contracts can be achieved through generics. Understanding React's built-in types and hooks like useState and useRef is crucial for leveraging their functionality.
Making Magic: Building a TypeScript-First Framework
TypeScript Congress 2023TypeScript Congress 2023
31 min
Making Magic: Building a TypeScript-First Framework
Top Content
Daniel Rowe discusses building a TypeScript-first framework at TypeScript Congress and shares his involvement in various projects. Nuxt is a progressive framework built on Vue.js, aiming to reduce friction and distraction for developers. It leverages TypeScript for inference and aims to be the source of truth for projects. Nuxt provides type safety and extensibility through integration with TypeScript. Migrating to TypeScript offers long-term maintenance benefits and can uncover hidden bugs. Nuxt focuses on improving existing tools and finds inspiration in frameworks like TRPC.
Stop Writing Your Routes
Vue.js London 2023Vue.js London 2023
30 min
Stop Writing Your Routes
Designing APIs is a challenge, and it's important to consider the language used and different versions of the API. API ergonomics focus on ease of use and trade-offs. Routing is a misunderstood aspect of API design, and file-based routing can simplify it. Unplugging View Router provides typed routes and eliminates the need to pass routes when creating the router. Data loading and handling can be improved with data loaders and predictable routes. Handling protected routes and index and ID files are also discussed.
Faster TypeScript builds with --isolatedDeclarations
TypeScript Congress 2023TypeScript Congress 2023
24 min
Faster TypeScript builds with --isolatedDeclarations
Top Content
This talk discusses the performance issues in TypeScript builds and introduces a new feature called isolated declarations. By running the compiler in parallel and using isolated modules, significant performance gains can be achieved. Isolated declarations improve build speed, compatibility with other tools, and require developers to write types in code. This feature has the potential to further increase performance and may be available in TypeScript soon.
Full-stack & typesafe React (+Native) apps with tRPC.io
React Advanced 2021React Advanced 2021
6 min
Full-stack & typesafe React (+Native) apps with tRPC.io
Top Content
Alex introduces tRPC, a toolkit for making end-to-end type-safe APIs easily, with auto-completion of API endpoints and inferred data from backend to frontend. tRPC works the same way in React Native and can be adopted incrementally. The example showcases backend communication with a database using queries and validators, with types inferred to the frontend and data retrieval done using Prisma ORM.

Workshops on related topic

React, TypeScript, and TDD
React Advanced 2021React Advanced 2021
174 min
React, TypeScript, and TDD
Top Content
Featured WorkshopFree
Paul Everitt
Paul Everitt
ReactJS is wildly popular and thus wildly supported. TypeScript is increasingly popular, and thus increasingly supported.

The two together? Not as much. Given that they both change quickly, it's hard to find accurate learning materials.

React+TypeScript, with JetBrains IDEs? That three-part combination is the topic of this series. We'll show a little about a lot. Meaning, the key steps to getting productive, in the IDE, for React projects using TypeScript. Along the way we'll show test-driven development and emphasize tips-and-tricks in the IDE.
Mastering advanced concepts in TypeScript
React Summit US 2023React Summit US 2023
132 min
Mastering advanced concepts in TypeScript
Top Content
Featured WorkshopFree
Jiri Lojda
Jiri Lojda
TypeScript is not just types and interfaces. Join this workshop to master more advanced features of TypeScript that will make your code bullet-proof. We will cover conditional types and infer notation, template strings and how to map over union types and object/array properties. Each topic will be demonstrated on a sample application that was written with basic types or no types at all and we will together improve the code so you get more familiar with each feature and can bring this new knowledge directly into your projects.
You will learn:- - What are conditional types and infer notation- What are template strings- How to map over union types and object/array properties.
Deep TypeScript Tips & Tricks
Node Congress 2024Node Congress 2024
83 min
Deep TypeScript Tips & Tricks
Top Content
Featured Workshop
Josh Goldberg
Josh Goldberg
TypeScript has a powerful type system with all sorts of fancy features for representing wild and wacky JavaScript states. But the syntax to do so isn't always straightforward, and the error messages aren't always precise in telling you what's wrong. Let's dive into how many of TypeScript's more powerful features really work, what kinds of real-world problems they solve, and how to wrestle the type system into submission so you can write truly excellent TypeScript code.
Best Practices and Advanced TypeScript Tips for React Developers
React Advanced 2022React Advanced 2022
148 min
Best Practices and Advanced TypeScript Tips for React Developers
Top Content
Featured Workshop
Maurice de Beijer
Maurice de Beijer
Are you a React developer trying to get the most benefits from TypeScript? Then this is the workshop for you.In this interactive workshop, we will start at the basics and examine the pros and cons of different ways you can declare React components using TypeScript. After that we will move to more advanced concepts where we will go beyond the strict setting of TypeScript. You will learn when to use types like any, unknown and never. We will explore the use of type predicates, guards and exhaustive checking. You will learn about the built-in mapped types as well as how to create your own new type map utilities. And we will start programming in the TypeScript type system using conditional types and type inferring.
Building Your Own Custom Type System
React Summit 2024React Summit 2024
38 min
Building Your Own Custom Type System
Featured Workshop
Kunal Dubey
Kunal Dubey
I'll introduce the audience to a concept where they can have end-to-end type systems that helps ensure typesafety across the teams Such a system not only improves communication between teams but also helps teams collaborate effectively and ship way faster than they used to before. By having a custom type system, teams can also identify the errors and modify the API contracts on their IDE, which contributes to a better Developer Experience. The workshop would primarily leverage TS to showcase the concept and use tools like OpenAPI to generate the typesystem on the client side. 
Frictionless Development With Unified Type System
JSNation 2024JSNation 2024
113 min
Frictionless Development With Unified Type System
Featured Workshop
Ejiro Asiuwhu
Ejiro Asiuwhu
Imagine developing where frontend and backend sing in harmony, types dance in perfect sync, and errors become a distant memory. That's the magic of TypeScript Nirvana!
Join me on a journey to unveil the secrets of unified type definitions, the key to unlocking frictionless development. We'll dive into:
- Shared language, shared love: Define types once, share them everywhere. Consistency becomes your BFF, errors your worst nightmare (one you'll rarely see).- Effortless coding: Ditch the manual grind of type checking. TypeScript's got your back, freeing you to focus on building awesomeness.- Maintainability magic: With crystal-clear types guiding your code, maintaining it becomes a walk in the park. More time innovating, less time debugging.- Security fortress: TypeScript's type system shields your app from common vulnerabilities, making it a fortress against security threats.