DEV Community: Dozer

Creating a Low-Latency Cache with Dozer and PostgreSQL

Abhishek Mishra — Tue, 30 May 2023 05:56:29 +0000

In the realm of software development and data management, the efficiency of data retrieval plays a crucial role in the overall performance of an application. One effective way to enhance this efficiency is by implementing a cache. A cache is a high-speed data storage layer that stores a subset of data, typically transient in nature so that future requests for that data are served up faster than is possible by accessing the data’s primary storage location.

This blog post explores how to enhance these aspects by implementing a low-latency cache using Dozer and PostgreSQL. We delve into the importance of caching in data management, the limitations of not having a cache, and the unique advantages of using Dozer as a cache layer. We also provide a guide on how to use Dozer for real-time data ingestion from PostgreSQL. This post is a must-read for developers seeking to improve their application's performance and responsiveness, and for those looking to simplify their data infrastructure.

Why Do We Need a Cache on Top of PostgreSQL?

PostgreSQL is a powerful, open-source object-relational database system. It is highly extensible and enables high performance on complex queries and heavy loads. However, like any other database, it has its limitations.

When a database grows in size and complexity, the time taken to retrieve data can increase significantly. This is especially true for complex queries that involve multiple joins and aggregations. In such scenarios, every millisecond counts, and the latency can add up quickly, leading to a slow and unresponsive application.

This is where a cache comes into play. A cache is a high-speed data storage layer that stores a subset of data so that future requests for that data are served up faster than is possible by accessing the data’s primary storage location. By storing frequently accessed data in a cache, we can significantly reduce the time taken to access this data.

What's the Problem with Not Having a Cache?

Without a cache, every time a client requests data, the application has to query the database. This can be a time-consuming operation, especially if the database is large or the query is complex. This can lead to increased latency and a poor user experience.

Moreover, without a cache, the database can become a bottleneck, especially under heavy load. Every read and write operation involves a significant amount of work, and as the number of operations increases, the time taken to complete these operations also increases. This can lead to increased CPU and memory usage, and in extreme cases, it can even lead to database crashes.

Why Not PostgreSQL Read Replica?

While a PostgreSQL read replica can certainly help offload some of the read operations from your primary database, it's not without its limitations. The process of querying a read replica still involves a network round trip and the overhead of a database query. These factors can contribute to latency, especially when dealing with large volumes of data or complex queries. A cache like the one provided by Dozer, on the other hand, can serve data much faster than a read replica.

In terms of using a read replica instead of a cache layer like Dozer, it depends on your specific use case and requirements. A read replica can help improve read performance by distributing read traffic across multiple instances, but it may not provide the same level of performance improvement as a cache layer like Dozer.

Moreover, Dozer doesn't make a distinction between types of data sources. Developers can get a seamless experience building products with application databases such as Postgres and MySQL, data warehouses such as SnowFlake and cloud storage such as S3 and Deltalake. Dozer can also consume real-time events and Ethereum data.

Why Dozer?

In the context of database management, most OLTP workloads involve random disk I/O usage. Given that disks, including SSDs, are slower in performance than RAM, database systems use caching to increase performance. Caching is all about storing data in memory (RAM) for faster access at a later point in time.

One approach to database caching involves replicating data using Change Data Capture (CDC) to an alternative database optimized for your queries. This method is increasingly being adopted for operations that require extensive data reading. However, it comes with its own set of challenges such as ensuring real-time data, managing indexing, guaranteeing availability, handling schema changes, and considering costs. Dozer addresses these pain points by providing an end-to-end system that takes care of the complexity of caching logic, allowing developers to focus on building their applications.

Dozer, an open-source data API backend, provides a solution to this. It connects to any of your data sources, transforms and stores the data in an embedded cache powered by LMDB, automatically creates secondary indexes, and instantly generates low-latency REST and gRPC APIs. Dozer is implemented fully in Rust for fast processing of data and is less resource intensive.

Without a cache layer, data retrieval would rely solely on the performance of the PostgreSQL database and its internal caching mechanisms. This can result in slower response times and increased load on the database server. By using a cache layer like Dozer, you can improve the performance of data retrieval and reduce the load on the database server.

How to Use Dozer as a Cache

Architecture Diagram

Here's a simple architecture diagram showing how Dozer can be used as a cache layer for a PostgreSQL database:

In a nutshell:

Dozer begins by ingesting data in real-time from the PostgreSQL database using the PostgreSQL connector and stores this data in its cache.
When a client sends a request for data, Dozer processes the request using the low-latency APIs it has generated, and retrieves the required data from its cache to serve the client.
For write requests, these are facilitated through the backend API, which directly communicates with the PostgreSQL database. Once the write operation is confirmed, the PostgreSQL database sends a write confirmation back to the backend API.

The key point here is that data ingestion into Dozer's cache and the serving of data to the client through Dozer's APIs occur in parallel. This setup ensures quick and efficient access to data, enhancing the performance of your application."

Dozer provides a PostgreSQL connector that allows it to ingest data in real-time from your PostgreSQL database. The configuration can be as simple as the following:

app_name: movie-booking
connections:
  - config: !Postgres
      user: postgres
      password: postgres
      host: 0.0.0.0
      port: 5438
      database: database_name
    name: connection_name

sql: |
  -- transformation and
  -- analytical queries here.

sources:
  - name: source_name_1
    table_name: table_name
    columns:
    connection: !Ref connection_name

  ...

endpoints:
  - name: endpoint_name
    path: /endpoint_path
    table_name: table_name
    index:
      primary_key:
        - primary_key_column_name

This configuration allows Dozer to ingest data in real-time from your PostgreSQL database and store it in a high-performance cache. This cache can then be used to serve low-latency data APIs to your application.

Conclusion

Implementing Dozer with its ability to ingest data in real-time from PostgreSQL can significantly improve the performance of your application by reducing the time taken to access frequently used data. This approach can significantly enhance your application's performance, making it more efficient and responsive in today's demanding tech scenario. By leveraging Dozer's real-time SQL engine, automatic secondary indexing, and instant API generation, developers can drastically lower the cost, complexity, and effort involved in putting together the data infrastructure necessary to build data applications.

Happy coding, Happy Data APIng! 🚀👩‍💻👨‍💻

Secure Your Data APIs with Dozer: Adding Authorization to APIs

Abhishek Mishra — Thu, 25 May 2023 14:35:50 +0000

In today's world of API-driven applications, securing access to data is very important. As the amount of data generated and consumed continues to grow, the need for reliable, and efficient data access management becomes increasingly critical. One essential aspect of data security is ensuring proper authorization for API calls, allowing only the right people, applications, and services to access the required data. Managing access control for your APIs is crucial to guarantee that only authorized users can access and perform specific actions. With the increasing complexity of API ecosystems, secure and efficient authentication and authorization mechanisms are necessary.

Dozer addresses this challenge by adding authorization layer to your data APIs in an efficient way. One of the ways Dozer provides authorization is through the use of JSON Web Tokens (JWT). JWT is a widely used industry-standard method for representing claims securely between two parties. You can learn more about JWT here. Using JWT tokens simplifies the process of adding authorization to your data APIs.

To demonstrate this, we'll build a simple movie ticket booking app that has two roles - user (public) and admin. The user can only book tickets and view their bookings, whereas the admin has full access to all the APIs. We will use @dozerjs/dozer and dozer-react to enhance the functionality of our app. @dozerjs/dozer is a TypeScript wrapper over gRPC APIs generated by Dozer, facilitating seamless communication with Dozer, while dozer-react provides a set of React helpers for using Dozer as a data provider. The sample application's repository can be found here, you can aslo find the instructions to run the application directly.

By following the step-by-step guide we've provided, you can safeguard your application from unauthorized access, ensuring data integrity and user trust.

TL;DR

Clone the repo and change to the project directory:

git clone git@github.com:getdozer/dozer-samples.git && cd dozer-samples/usecases/api-auth/

Start the PostgreSQL database:

cd database && docker-compose up -d && cd ..

Generate and export a master token for auth:

dozer api generate-token
export MASTER_TOKEN=your_token_here

Start the Dozer app:
```
dozer
```
In a new terminal, install server dependencies and run the server app:
```
cd server && yarn install && node index.js
```
In a new terminal, navigate to the client directory, install client dependencies and start the app:

    cd ../client && yarn install && yarn start

Visit localhost:3000 to use the app. For the admin dashboard, go to localhost:3000/admin/dashboard with "admin" as username and password.

User Page

Admin dashboard

Introduction

In this blog, we'll guide you through the process of constructing a movie ticket booking application using Dozer, React, and PostgreSQL. The PostgreSQL database is utilized directly in our server-side application to manage user creation and ticket bookings.

To elaborate, the server app is specifically designed to handle user login, create new user entries, and register ticket bookings. It interacts directly with the PostgreSQL database to handle these tasks. The information required for booking tickets, such as booking details, is sent from the React frontend application to the Node.js server app, which then inserts these details into the PostgreSQL database.

However, when it comes to consuming real-time data like the movie list, dashboard updates, or ongoing bookings, we use Dozer, which fetches the data from PostgreSQL using the PostgreSQL connector. This strategy allows us to get the most recent booking updates and display them on the dashboard in real-time.

The PostgreSQL connector in Dozer bridges our database with the Dozer server and enables the seamless transfer of data in real-time. You can learn more about this connector at Dozer's documentation page here: PostgreSQL Connector.

On the client side, we're using a React app to display the movie data and booking interface, pulling data from Dozer. We employ the @dozerjs/dozer and dozer-react libraries to simplify interactions with Dozer and access real-time data without having to hardcode API endpoints in the code.

The @dozerjs/dozer library manages gRPC requests between the frontend and the Dozer server, allowing effective communication between the two. In contrast, the dozer-react library provides a set of React helpers, making it simpler to use Dozer as a data provider in our React app. Find more about these libraries at their respective GitHub repositories: @dozerjs/dozer and dozer-react.

Let's dive into the process of building this application, demonstrating how you can leverage Dozer to amplify your app development experience.

Server-Side Setup (Express.js)

First, let's set up the server-side code using Express.js. The code is available in server/index.js:

const express = require('express')
const app = express()
const {ApiClient, AuthClient} = require("@dozerjs/dozer");
const axios = require('axios');
const cors = require('cors');
const { faker } = require('@faker-js/faker');

//... Rest of the code

We import the required libraries, including express, @dozerjs/dozer, axios, cors, and @faker-js/faker.

Database Connection for Server-Side

Setting up a connection with a PostgreSQL database is crucial to perform operations like user creation and booking entries. We accomplish this using the pg client.

const { Client } = require('pg')

const client = new Client({
    host: '127.0.0.1',
    port: 5432,
    database: 'omdb',
    user: 'postgres',
    password: 'postgres',
});

client.connect();

In the configuration object, we specify the parameters like host, port, database name, user, and password. With the successful connection established via client.connect(), we can execute queries to our database in our Express.js routes.

This setup enables two primary functionalities in our application:

User creation: With the database connection ready, we can insert new user details into the users table. This is used in the /public/new_user endpoint, where a new user is created with a unique UUID and a randomly generated name.
Booking entries: The connection also allows us to create new entries in the bookings table. This functionality is used in the /public/book_movie endpoint, where a user can book a movie, resulting in a new booking record in the database with the user's id and the provided movie id.

Admin Login Endpoint

This endpoint validates the provided username and password. If valid, a post request is made to the authentication server to get a token. The token is then sent back to the client. Error handling is also set up to send a 500 status code in case of a server error and a 400 status code for incorrect credentials.

app.post('/admin/login', function(req, res){
    // Check if the provided credentials are valid
    // Generate JWT token with full access to all API endpoints
    // Send the JWT token to the client
})

Public New User Endpoint

The new user endpoint generates a random username and UUID using the faker library. It then inserts this new user into the database. After successful insertion, a JWT token is generated for the new user with limited access. Error handling is set up in case the server encounters an error while processing the request.

app.get('/public/new_user', function(req, response){
    // Generate random username and UUID
    // Insert new user into the database
    // Generate JWT token with limited access for the new user
    // Handle server errors
})

Public Book Movie Endpoint

The book movie endpoint allows users to book a movie by sending their JWT token in the headers. The server first validates the user's token. If valid, it inserts a new record in the bookings table in the database with the user's id and the provided movie id. The response includes error handling as well.

app.post('/public/book_movie', function(req, response){
    // Validate the user's token
    // Insert a new record in the bookings table
    // Handle server errors
})

Error Handling Middleware

Lastly, we also add a middleware function to handle any errors that may slip through our route handlers. This middleware logs the error and sends a generic error message to the client.

app.use((err, request, response, next) => {
    // log the error, for now just console.log
    console.log(err)
    response.status(500).send('Something broke!')
})

After setting up these routes, the Express.js application is ready to listen for requests on port 4000.

app.listen(4000)

Our error handling middleware ensures that any errors that occur during these database operations are effectively captured and appropriately communicated to the user.
In this way, our server is set up to handle the creation of new users, allow users to book movies, and allow an admin to log in, all while ensuring errors are handled appropriately. .

Database Setup For Movie data (PostgreSQL)

We use PostgreSQL as our database, and the configuration is available in the docker-compose.yml file. The file defines a postgres service, which includes the container name, image, volumes, and environment variables.

version: '3.8'
services:
  postgres:
    container_name: omdb-postgres
    image: debezium/postgres:13
    volumes:
      - ./pg_hba.conf:/var/lib/foo/pg_hba.conf
      - ./data/init.sql:/docker-entrypoint-initdb.d/init.sql
      - ./data:/data
    command: postgres -c hba_file=/var/lib/foo/pg_hba.conf
    environment:
      POSTGRES_DB: omdb
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      ALLOW_IP_RANGE: 0.0.0.0/0
    ports:
      - '5438:5432'
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -U postgres -d omdb" ]
      interval: 5s
      timeout: 5s
      retries: 5

The database is populated with the Open Movie Database (OMDB) dataset, which provides a comprehensive list of movie-related data. For further details about this dataset, you can visit the official OMDB repository here: OMDB on GitHub

Front-End Setup (React)

The frontend of our application is designed with React, a popular JavaScript library used for building interactive UIs. The frontend setup for our movie booking application involves a series of React components that interact with our backend server using HTTP requests. These components handle user authentication, data fetching, and real-time updates to ensure users have an interactive and up-to-date interface.

Let's take a look at the structure of our project:

├── README.md
├── package.json
├── src
│   ├── App.css
│   ├── App.js
│   ├── App.test.js
│   ├── components
│   │   ├── admin
│   │   │   └── dashboard.js
│   │   ├── login.js
│   │   └── public
│   │       ├── movies.js
│   │       └── moviesList.js
│   ├── index.css
│   ├── index.js
│   ...
└── yarn.lock

Our React components are placed in the src/components directory, separated by user type (public users and admins).

Login Component

The login.js component facilitates the login functionality for the admin users. It includes text fields for the admin to enter their username and password, and a login button to submit the login details.

On submission, a POST request is made to the http://localhost:4000/admin/login endpoint with the username and password as the body of the request. If the credentials are valid, the response will contain a token, which is stored in the browser's local storage and used to authenticate future requests.

Admin Dashboard Component

The dashboard.js component acts as the main interface for admin users. On load, it checks if a token exists in local storage. If not, it redirects the user to the login page.

Using the movies_with_bookings query, it fetches the list of movies with their booking counts and displays them in a table. This component uses real-time updates to ensure the admin always has the most current booking data.

Movie List Component

The moviesList.js component displays the list of movies and their current booking counts to public users. It allows users to "buy" a ticket for a movie, which is simulated by making a POST request to the http://localhost:4000/public/book_movie endpoint with the selected movie's id. The state of this component is updated in real-time as bookings are made.

Movies Component

The movies.js component acts as a wrapper for the moviesList.js component. On load, it checks for a token in local storage. If a token doesn't exist, it makes a GET request to the http://localhost:4000/public/new_user endpoint to generate a new token. Once a token is obtained, it passes the token as a prop to the MoviesList component.

Integrating Dozer JWT in the Application

Having established the foundation of our application in the previous sections, it's time to delve into the secure integration of JWT authentication and Role-Based Access Control (RBAC) using Dozer. Our focus will be on illustrating how this can be applied in our movie ticket booking application.

Outlined below are the key steps to enable secure API authorization in your application:

Generate a master token using Dozer's API server.
Using the master token, generate custom access tokens for each set of APIs to establish granular-level access.
Incorporate the generated custom access tokens in subsequent client-side API calls.

1. Master JWT Token Generation

Kick off the process by running the following command to generate a master JWT token:

dozer api generate-token

This command triggers the generation of the master token, which is outputted to stdout. The master token has the privilege of accessing all API endpoints specified in the config file. To access the APIs, include the token in your request header as demonstrated below:

MASTER_TOKEN = 'your_master_token_here'

2. Custom Access Token Generation

For a more granular access control, Dozer allows for the creation of custom access tokens for specific API calls. These tokens can be generated through a POST request to the /auth/token endpoint, providing an access filter and the previously generated MASTER_TOKEN in the request header.

This affords you the opportunity to designate specific access controls for different users or roles within your application. For a more detailed guide on this, you may refer to the Dozer documentation page on Authentication & Authorization.

In the context of our movie ticket booking app, we generate a JWT token comprising the necessary access controls whenever an admin logs in or a new user is registered. This token is then dispatched to the client and utilized in future API calls.

Here is an illustration of the token generation process in the index.js file located in the server directory:

//...other code
const MASTER_TOKEN = process.env.MASTER_TOKEN;
app.post('/admin/login', function(req, res){
    //...other code
    const data = JSON.stringify("All");
    const config = {
        method: 'post',
        url: 'http://127.0.0.1:8080/auth/token',
        headers: {
            'Authorization': 'Bearer ' + MASTER_TOKEN,
            'Content-Type': 'application/json'
        },
        data : data
    };
//...other code

Furthermore, we can customize the access token to limit access based on the user's role:

//...other code
// Generate token with limited access based on user role
const data = JSON.stringify({
   //...other code
});
//...other code

In the code snippet above, the custom access token restricts access to only the need_to_Add_endpoint_name API endpoint. The access filter can be tweaked to include other endpoints and set specific permissions anchored on user roles.

3. Incorporating Tokens in Client-Side API Calls

Once the JWT tokens are generated, they can be utilized client-side for secure API access. This can be seen in the movies.js file and other components housed in the public directory.

//...other code
function Movies() {
    let [token, setToken] = useState(null);

    useEffect(() => {
        if (localStorage.getItem('public_jwt')) {
            setToken(localStorage.getItem('public_jwt'));
        } else {
            //...other code
        }
    }, []);
//...other code

Whenever the client makes an API call, the server validates the JWT token and checks if the user has the necessary permissions to access the requested endpoint. If the token is valid and the user will get the valid response.

Wrapping Up:

Using Dozer's powerful security capabilities, you can easily add authorization to your data APIs, ensuring that your application remains secure and accessible only to authorized users. Whether you're building a movie ticket app or any other application, Dozer provides a flexible and straightforward way to manage access control for your APIs.

The beauty of Dozer lies in its flexibility—it moulds itself according to your needs, allowing you to craft custom access controls for different user roles, thereby ensuring the right balance between functionality and security.

Through our movie ticket booking application, we've demonstrated just a fragment of the potential that Dozer holds. Regardless of your application's domain—be it e-commerce, social media, healthcare, or any other field—you can leverage the power of Dozer to secure your data APIs.

In conclusion, Dozer presents a robust, scalable, and straightforward solution to facilitate API security and data management. It's a testament to the flexibility of modern software tools and their ability to adapt to a wide range of application needs. We hope this guide has clarified the process of secure API management and inspired you to integrate Dozer into your next project.

Happy coding, Happy Data APIng! 🚀👩‍💻👨‍💻

Announcing the Dozer Appathon!

Abhishek Mishra — Wed, 03 May 2023 05:59:55 +0000

👋🏼 Are you looking for a platform that simplifies the process of building and deploying data-driven applications? Look no further than Dozer ! 💻 is a powerful Data API backend that simplifies the process of building and deploying data-driven applications & APIs for real-time data processing and analysis, predictive analytics, data caching, and much more.

And now, you have the chance to showcase your coding skills and creativity by participating in the Dozer Appathon! 🚀 This exciting event is open to everyone who is interested in building applications with Dozer. The submission period runs from May 3 to May 9, 2023; so you still have time to get your app ready! ⏰

Dozer Appathon Hacker Guide 👩‍💻👨‍💻

Introduction 🚀

Welcome to the Dozer Appathon! This is your chance to showcase your coding skills and creativity by building amazing applications using our platform. Dozer is a powerful Data API backend that simplifies the process of building and deploying data-driven applications & APIs.

Sample use cases📝

To help you get started, we have prepared a list of sample use cases and ideas for your app. You can use these as inspiration for your project, or come up with your own unique ideas.

Some sample use cases are

- Real-time data processing and analysis
- Data aggregation and consolidation from multiple sources
- Machine learning and predictive analytics
- Data caching and optimization for faster performance
- IoT data processing and analysis

Some wonderful project ideas💡

Real-time Eth layer 2 crypto dashboard📈
NodeJS-based Twilio notifications with real-time alerts using Dozer📲
Real-time ML predictions using ONNX model🧠
React Native push notifications📱
Klarna clone with user profiles and transactions in React Native💳
Real-time vehicle tracking on Google Maps with traffic density alerts (Web or React Native)🚗
Real-time inventory dashboard for a retail chain with multiple warehouses🛍️
Fraud detection for credit card transactions using ML models on Python Lambda💳🕵️‍♂️
Model training and scoring with data injection through Python Lambda 🧑‍💻📈

Prizes 🎁

We have some exciting prizes up for grabs!

The top 5 submissions will win an amazing Mechanical Keyboard and a Dozer branded Swagpack 🏆
The next 10 successful submissions win Dozer Branded swagpack which includes an awesome tee and stickers. 🎉
Additionally, all participants will receive a certificate of participation.📜

Resources 📚

To help you build your app, we have some resources for you:

Website: https://getdozer.io/ 🌐
Docs: https://getdozer.io/docs/dozer 📖
Blogs: https://getdozer.io/blog 📰
Blog Tutorial: https://getdozer.io/blog/building-real-time-data-app-using-dozer-react-postgresql/ 📚
Video Tutorial: https://youtu.be/pVfyMfNrlD4 📹
Discord Community support: If you have any questions or need help with your project, you can reach out to #dozer-appathon channel of the Dozer Discord for support. 💬

Judging Criteria 🧐

We will rate as per optimal utilisation of Dozer in your app on a scale of 0-10, based on factors like real-time data, multiple data sources, and caching requirements. We'll also consider the problem you're solving and the code quality of your app.

Submission Method 📩

Submit a PR following the Contribution Guidelines with your project detail.
Fill out this form post PR submission.
Winners will be announced in the Discord Server.💪🏆

Stargazers’ Raffle✨

Star our repository and get a chance to win an amazing Swagpack!!🎁🎉
We will announce the winners in our Discord server. So, don’t forget to join.🤗

We're excited to see what you can create with Dozer!
Good luck and Happy Hacking!, Happy Data APIng 🚀 👩‍💻👨‍💻

For more details, checkout: https://getdozer.io/blog/dozer-appathon

Unleashing the Power of Dozer Lambda Runtime for Real-time and Event-driven Data Apps

Abhishek Mishra — Tue, 02 May 2023 14:21:50 +0000

Real-time data processing is essential for modern applications, as it ensures that insights are timely and accurate. The open-source project Dozer now provides a runtime environment that allows developers to execute lambda functions or custom user-defined functions based on specific data conditions.
This new functionality allows for more dynamic and event-driven applications, as Lambda functions can be triggered based on specific data change conditions. Just imagine triggering a Twilio notification based on a SQL statement (data change condition)!

In this blog post, we will discuss how you can use Dozer to respond to real-time data changes by triggering functions based on data conditions.
In this example, we will create a React app that generates a real-time graph based on room temperature data (room temperature monitoring). We will use the pydozer_log module to react to data change conditions and trigger a Twilio message if the room temperature goes above 22.9°C.

Dozer Lambda Runtime: A Glimpse

The Dozer Lambda runtime is a major enhancement to the Dozer project, enabling developers to write Lambda functions in JavaScript and Python that react to Dozer events.

To help you better understand the Dozer Lambda runtime, we will examine the following state diagram:

DCC: Data Change Condition

Data Source: Your application's data source, such as a database, sends data to the Dozer pipeline.
Dozer Pipeline: The data is processed through the Dozer pipeline, which can apply various transformations and aggregations based on the configuration you provide.
Lambda Runtime: The Dozer Lambda runtime monitors the pipeline for specific data change conditions that you define. If a data change condition is met, the corresponding Lambda function is triggered.
Action/Trigger: The triggered Lambda function performs the desired action, such as sending a Twilio notification or updating a database record.

The reponse of trigger can be used to write back top dozer, for example using gRPC service.

JavaScript and Python Bindings for Reading Dozer Logs

Dozer now provides bindings for both JavaScript and Python to read logs, making it easier for developers to work with the languages they are most comfortable with. These bindings are essential components of the Dozer Lambda runtime, as they enable developers to create Lambda functions that respond to
events.

React App Example: Room Temperature Monitoring

In this example, we will create a React app that generates a real-time graph based on room temperature data. We will use the pydozer_log module to react to data change conditions and trigger a Twilio message if the room temperature goes above 22.9°C.

Here are the steps to run the app:

Clone the Dozer Samples repository and install dependencies:

git clone https://github.com/getdozer/dozer-samples.git
cd dozer-samples/usecases/lambda/
pip install -r requirements.txt

Remove any existing Dozer configuration and start the Dozer:

rm -rf ./.dozer && dozer -c config/dozer-config.yaml

Run the Python scripts to read data from the pipeline:

Note: make sure to replace the Twilio configuration in reader.py, with your account id and auth token. To learn more about Twilio SMS setup checkout this getting started guide.

python reader.py
python main.py

Navigate to the React app folder, install the necessary packages, and start the development server:

cd react
yarn install && yarn start

Once the app is running, you should see a real-time graph of the room temperature data. If the temperature exceeds 22.9°C, a Twilio message will be triggered, notifying the relevant parties of the situation.

Conclusion 🎯

By using the Dozer Lambda runtime, you can create event-driven applications that automatically respond to changes in your data, making your applications more efficient, scalable, and responsive to user needs. By leveraging the Python and JavaScript libraries provided by Dozer, developers can easily build applications that respond to real-time data changes, making it an invaluable resource for modern applications.

Happy coding, Happy Data APIng! 🚀👩‍💻👨‍💻

🚀 Getting Started with Dozer with Docker: A Step-by-Step Tutorial

Abhishek Mishra — Mon, 24 Apr 2023 14:00:00 +0000

In the world of data-driven applications, accessing and manipulating data in real-time is crucial. Dozer is a powerful tool that helps you create low-latency data APIs (gRPC and REST) from any data source, enabling seamless integration with frontend applications. In this tutorial, we'll walk you through setting up Dozer with Docker and connecting it to PostgreSQL using Docker.

🌟 Why Dozer?

Dozer addresses the challenges of accessing and manipulating data in real-time by providing an easy-to-use solution to create efficient data APIs. With Dozer, you can quickly set up APIs that can be easily integrated with your frontend applications. Furthermore, using Docker allows you to containerize your application, making it portable and easy to manage.

In this step-by-step tutorial, we'll guide you through connecting PostgreSQL with Dozer using Docker. Let's get started!

🔧 Prerequisites:

Docker/Docker desktop installed on your machine
Basic knowledge of Docker and PostgreSQL

📝 Overall steps involved:

Create a docker-compose file for PostgreSQL
Create a Dozer config with PostgreSQL connector and API configuration
Run PostgreSQL (via Docker)
Run Dozer (via Docker)
Query your API

🔹 Step 1: Create a `docker-compose.yml` file

First, create a docker-compose.yml file and copy the following configuration to it or download the file here:

version: '3.9'
services:
  postgres:
    platform: linux/amd64
    container_name: quick-start-postgres
    image: public.ecr.aws/getdozer/dozer-samples-pg-stocks:latest
    command: postgres -c hba_file=/var/lib/stock-sample/pg_hba.conf
    environment:
      POSTGRES_DB: stocks
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      ALLOW_IP_RANGE: 0.0.0.0/0
    ports:
      - '5434:5432'
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -U postgres  -h 0.0.0.0 -d stocks" ]
      interval: 5s
      timeout: 5s
      retries: 5

volumes:
  sample-data:

🔹 Step 2: Create a Dozer config file

Next, create a dozer-config.yaml file with the following content:

app_name: 1-hypercharge-postgres-sample
connections:
  - config: !Postgres
      user: postgres
      password: postgres
      host: host.docker.internal
      port: 5434
      database: stocks
    name: stocks
sources:
  - name: stocks
    table_name: stocks
    columns:
      - id
      - ticker
      - date
      - open
      - high
      - low
      - close
      - adj_close
      - volume
    connection: !Ref stocks
  - name: stocks_meta
    table_name: stocks_meta
    columns:
      - symbol
      - security_name
    connection: !Ref stocks
endpoints:
  - name: stocks
    path: /stocks
    table_name: stocks
    index:
      primary_key:
        - id
  - id: null
    name: stocks_meta
    path: /stocks-meta
    table_name: stocks_meta
    index:
      primary_key:
        - symbol

🔹 Step 3: Run PostgreSQL with sample stock data

Open a terminal or command prompt and navigate to the directory containing the docker-compose.yml file. Run the following command to start PostgreSQL with sample stock data:

docker-compose up

🔹 Step 4: Run Dozer with PostgreSQL connector (via Docker)

Open another terminal or command prompt and navigate to the directory containing the dozer-config.yaml file. To run Dozer as a Docker container, execute the following command:

docker run -it \
       -v "$PWD":/usr/dozer \
       -p 8080:8080 \
       -p 50051:50051 \
       --platform linux/amd64 \
       public.ecr.aws/k7k6x1d4/dozer \
       dozer

Congratulations 🥳 , you should see Dozer up and running!

🎉 Query your API

Dozer automatically generates APIs in both REST and gRPC formats, along with documentation, OpenAPI, and proto files. In the following sections, we'll showcase some possible queries. Refer to the APIs section for more details.

🔸 Query Using gRPC:

You can use gRPCurl or Postman to interact with gRPC APIs.

gRPC query example:

grpcurl -plaintext localhost:50051 dozer.generated.stocks_meta.StocksMetas/query

🔸 Query Using REST:

You can use curl or Postman to interact with REST APIs.

REST query example:

curl --location --request GET 'localhost:8080/stocks-meta'

That's it! You've successfully set up Dozer with Docker and connected it to PostgreSQL. Now you can easily create APIs for your data and integrate them into your frontend applications.

If you're interested in learning more about how to use Dozer with applications, we recommend reading our previous blog post: "Building a Real-Time Data App with Dozer, React, and PostgreSQL". In this blog, you'll find a detailed tutorial that demonstrates how to integrate Dozer with a React frontend application and PostgreSQL as a data source. This comprehensive guide will give you a better understanding of how to build powerful, real-time data-driven applications using Dozer.

Happy coding, Happy Data APIng! 🚀👩‍💻👨‍💻

Dozer Goes Open Source: Empowering the Community to Build Real-time Data Apps

Abhishek Mishra — Fri, 21 Apr 2023 15:56:07 +0000

We are excited to announce that Dozer, a powerful platform for building low-latency data APIs (gRPC and REST) from any data source, is now open source! The Apache 2.0 license 🎉. With this move, we aim to empower the community to build and scale real-time data applications more effectively.

Dozer simplifies the process of connecting applications to various data sources, such as PostgreSQL, Kafka, or other databases & sources, enabling developers to easily create and manage real-time data APIs. By making Dozer open source, we are inviting developers to contribute to its growth and help shape the future of real-time data applications.

What is Dozer?

Dozer is a platform that enables developers to build low-latency data APIs (gRPC and REST) from any data source. It simplifies the process of connecting applications to data sources like databases, APIs, or other services, and automatically manages data streaming and synchronization. This allows developers to focus on building powerful applications that utilize real-time data without worrying about the complexities of managing data connections and APIs.
With a few lines of SQL and a simple YAML configuration, you can build, deploy and maintain full data backends. It also has support for client libraries in JS and React to easily integrate with front-end applications.

Why Open Source?

There are several reasons why we decided to make Dozer open source:

Community-driven innovation: We want developers from all over the world to contribute their ideas,and provide valuable insights, improvements, and fixes, leading to a more innovative and robust platform, and faster development cycles.
Transparency and trust: We want to enable users to view and understand the underlying code, fostering trust in the platform and ensuring that it meets their needs and expectations. Especially how data is being stored and processed.
Collaboration and learning: We encourage developers to collaborate, share ideas, and learn from one another. We want to foster a strong community that helps developers grow their skills and expertise in the real-time data space.

Getting Started with Dozer

To help you get started with Dozer, we've created documentation and tutorial blogs that guide you through the process of setting up, configuring, and using Dozer for your data applications. You can find these resources on our official documentation site and can also follow Dozer on dev.to or additional resources on getting to know & use Dozer better!

How to Contribute?

We welcome contributions from the community! If you're interested in contributing to Dozer, please check out our GitHub repository for the guidelines. Whether you want to submit a bug report, suggest a new feature, or contribute code, we appreciate your help in making Dozer even better.

We also encourage you to check out the following resources to know other ways of contributions:

Show and tell GitHub discussion forum: We have setup a GitHub discussion forum, where you can share your Dozer projects, and experience, ask questions, do a feature request, and connect with fellow Dozer community members. Feel free to join the conversation and share your own projects & ideas! Additionally, we have also created a discord channel to give developers an opportunity to showcase and talk about their projects. Join our discord channel, If you’ve built something using Dozer, we’d love to see it!
Dozer Samples Repository: We have created a Dozer Samples repository on GitHub, which contains sample applications built using Dozer. This repository can be a good resource for learning how to use Dozer and getting inspiration for your projects. You can also contribute to the dozer-sample repository by submitting a pull request with your projects. This not only helps showcase your work but also provides good examples for other developers to learn from and take inspiration to build their own projects.

You can also join our community discord to see what we are cooking at Dozer :female-cook: :male-cook:

Conclusion

We believe that by making Dozer open source, we are empowering the developer community to build amazing real-time data applications. We look forward to seeing the project and ideas that arise from this collaborative effort, and we're excited to work together to shape the future of real-time data!
Together, we can shape the future of real-time data and contribute to the ongoing success of open source software. 🚀

Happy coding, Happy Data APIng! 🚀👩‍💻👨‍💻

Building a Real-time Data App with Dozer, React, and PostgreSQL

Abhishek Mishra — Wed, 19 Apr 2023 08:13:29 +0000

In today's data-driven world, having access to real-time data is crucial and gaining a value from the data is more cumbersome. Another challenge developers face when working with real-time data is efficiently and quickly fetching, processing, and displaying the insights from the data.

Enter Dozer, it simplifies the process of creating low-latency data APIs (gRPC and REST) from any data source easily with OpenAPI specification. Dozer is a real-time data engineering tool built with Rust, which enables seamless integration of data streams(sources), efficient data processing, and lightning-fast response times. This allows developers to focus on building applications without worrying about the complexity of managing real-time data.

To know more about it's capabilities and features you can checkout Dozer repository.

In this blog, we'll walk you through the process of creating a real-time flight data application using Dozer, React.
For this particular blog, we are using PostgreSQL as our data source, but it's important to note that there can be multiple data sources that can be connected to Dozer and subsequently used by your React application. By using Dozer, you can quickly create efficient data APIs that enable seamless integration with your frontend applications.
PostgreSQL is connected to Dozer via PostgreSQL connector and fetches data from multiple Postgres tables that has a flight data and combines them in real time based on the queries and produces fast READ APIs.

On the client side, React app is being used to display the flight data on map coming from Dozer. The dozerjs and dozer-react libraries are used to make it easy to interact with Dozer and access real-time data without having to hardcode API endpoints in the code.
The dozerjs library manages gRPC requests between the frontend and the Dozer app, while the dozer-react library provides React hooks for easy integration with React components.

So let's dive in and learn how to harness the power of Dozer to create amazing real-time data-driven applications! 💪✨

🐘 Setting Up PostgreSQL:

Clone the sample repository and navigate to the appropriate directory:

git clone https://github.com/getdozer/dozer-samples.git
cd dozer-samples/usecases/pg-flights

Checkout the react branch:

git checkout react

In the usecases/pg-flights/ directory, run the following command to start PostgreSQL using Docker Compose:

docker-compose up

This command will create a PostgreSQL container with the necessary configuration and sample flight data as shown below:

To test the connection to the PostgreSQL container, you can use psql or any PostgreSQL client. Here's an example using psql:

psql -h localhost -p 5437 -U postgres -d flights

This command connects to the flights database using the postgres user on the default port 5437. Once connected, you can run SQL queries to explore the sample flight data.

With PostgreSQL set up and running, we can now proceed to build the React application and configure Dozer.

⚛️ Building & Running the React App 🚀:

Open a new termincal and navigate to the /usecases/react directory in the cloned dozer-samples repository:

cd dozer-samples/usecases/react

This directory contains a sample React app that displays flight data on a map component.

Install the required packages for the app using yarn install:

yarn install

In the project directory, run the following command to start the development server:

yarn start

This command runs the app in development mode.

Open http://localhost:3000 to view it in your browser. The page will automatically reload when you make changes, and you may also see any lint errors in the console. On this page, you will the Dozer logo, now navigate to http://localhost:3000/airports to see the map UI for respective ariport and it's flight booking data as follow:

This React app uses the @dozerjs/dozer and @dozerjs/dozer-react libraries to interact with Dozer, enabling seamless integration and real-time data updates.

Since, we have not configured the Dozer with PostgreSQL yet, there is not data on the map.
Let' quickly setup the Dozer and see the magic happening!

🔌 Configuring Dozer and PostgreSQL Connector:

Start by installing Dozer. In for this blog, we'll use Homebrew for the installation. To learn about other installation methods, visit the official Dozer installation documentation.

open a new terminal, ensuring you're in the same directory as your React project, and run:

brew tap getdozer/dozer
brew install dozer

Once the installation is complete, create a dozer-config.yaml file in your project directory and paste the following configuration into it:

# dozer-config.yaml content

app_name: flight-microservices
connections:
  - name: flights_conn
    config: !Postgres
      user: postgres
      password: postgres
      host: 0.0.0.0
      port: 5437
      database: flights

sql: |
  select f.arrival_airport as airport, a.coordinates as coordinates, COUNT(t.ticket_no) as tickets
  INTO airports_count
  from tickets t
  join ticket_flights tf on t.ticket_no = tf.ticket_no
  join flights f on tf.flight_id = f.flight_id
  join airports a on f.arrival_airport = a.airport_code
  group by f.arrival_airport, a.coordinates;

  select extract(HOUR FROM f.window_start) as start, count(f.window_start) as dep_count
  INTO departures_count
  from TUMBLE(flights, scheduled_departure, '4 HOURS') f
  group by extract(HOUR FROM f.window_start)

sources:
  - name: tickets
    table_name: tickets
    columns:
    connection: !Ref flights_conn

  - name: flights
    table_name: flights
    columns:
    connection: !Ref flights_conn

  - name: ticket_flights
    table_name: ticket_flights
    columns:
    connection: !Ref flights_conn

  - name: airports
    table_name: airports
    columns:
    connection: !Ref flights_conn

  - name: airports_flights_schema
    table_name: airports
    columns:
    schema: flights_schema
    connection: !Ref flights_conn

endpoints:

  - name: tickets
    path: /bookings/tickets
    table_name: tickets
    index:
      primary_key:
        - ticket_no

  - name: flights
    path: /bookings/flights
    table_name: flights
    index:
      primary_key:
        - flight_id

  - name: airports
    path: /bookings/airports
    table_name: airports
    index:
      primary_key:
        - airport_code


  - name: ticket_flights
    path: /bookings/ticket_flights
    table_name: ticket_flights
    index:
      primary_key:
        - ticket_no
        - flight_id

  - name: airports_count
    path: /airports_count
    table_name: airports_count
    index:
      primary_key:
        - airport
        - coordinates

  - name: departures_count
    path: /departures_count
    table_name: departures_count
    index:
      primary_key:
        - start

This configuration defines the PostgreSQL connector and sets up the API endpoints that our React application will use to fetch data.
For more information about the PostgreSQL connector, refer to documentation.
Dozer supports multiple connectors, allowing you to integrate various data sources. Check out the list of available connectors for more options.

With the configuration in place, run Dozer in action by executing the following command:

dozer -c dozer-config.yaml

Now, you should see Dozer logs indicating that the endpoints is ready to be queried by the React app.

You can now access your application at http://localhost:3000/airports to see the real-time flight data populating on the map component as shown below:

Additionally, You can also query the API using gRPC and REST endpoints to test the data. In this case, you can use the REST endpoint using curl or Postman http://localhost:8080/airports/

Additionally, you can check the OpenAPI specification documentation that Dozer generates for endpoints by requesting http://localhost:<port>/<endpoint_path>/oapi as shown below.

🌟 Conclusion:

Congratulations! You've successfully built a real-time flight data app using Dozer, React, and PostgreSQL. This powerful combination enables you to create interactive and data-driven applications with ease.
Feel free to expand on this example and explore other data sources and connectors to enhance your app's capabilities. Enjoy exploring the application and customizing it further to suit your needs!

Happy coding, Happy Data APIng! 🚀👩‍💻👨‍💻

This week at Dozer #5

Abhishek Mishra — Tue, 18 Apr 2023 04:43:51 +0000

🌟 Welcome to this week's update on dozer! Dozer is excited to announce the release of version v0.1.17, which brings a host of new features and improvements. Our team has been working tirelessly to enhance the overall functionality, and we can't wait for you to experience all the new features. Here are the updates for this week.

Key enhancement of v0.1.17 🚀

Switched to Apache License #1404 📜:

In this release, Dozer has switched to the Apache License (The 2.0 version), making is 100% Open Source. This change provides a more permissive and flexible licensing option, encouraging collaboration and adoption by a wider range of developers and organizations.

Asyncify Connectors #1409 ⚡:

In this release, connectors have been refactored to work asynchronously, leading to performance improvements and smoother execution.

SQL Logic Test Framework #1326 🧪:

A new SQL Logic Test Framework has been introduced, allowing for more comprehensive testing of Dozer's SQL capabilities.

APIs for Monitoring and Status Updates #1329 📈:

We've added APIs for monitoring and status updates, enabling better insight into Dozer's performance and health.

Graceful API Thread Shutdown #1416 🛠️:

The API thread now shuts down gracefully, preventing potential issues with abrupt termination.

Dozer Lambda Functions: A Glimpse into the Future 🚀🔮 #1424, #1426 🐍🌐:

We're excited to share that we've released JavaScript and Python bindings for reading Dozer logs. These bindings will be part of our Dozer Lambda runtime, allowing you to create Lambda functions in JS and Python that react to Dozer events. Just imagine triggering a Twilio notification based on a SQL statement (data change condition)! This feature is currently experimental, and we'll publish a complete article with use cases and samples soon.

Note: DCC stands for Data Change Condition

Here's an example for JavaScript:

const dozer = require('..');

async function main() {
    runtime = dozer.Runtime();
    reader = await runtime.create_reader('../.dozer/pipeline', 'trips');

    for (let i = 0; i < 10; ++i) {
        let op = await reader.next_op();
        console.log(op);
    }

    process.exit(0);
}

main();

And for Python:

import dozer_log

async def main():
    reader = await dozer_log.LogReader.new('../.dozer/pipeline', 'trips')
    for _ in range(10):
        data = await reader.next_op()
        print(data)

if __name__ == '__main__':
    import asyncio
    asyncio.run(main())

In these examples, the console.log() and print() statements can be replaced with user-defined code.

Other Improvements & Fixes 🔧

Fixed e2e tests writing to directories that are not removed in #1407
Fixed type validation in #1405
Replaced std::thread::sleep with tokio::time::sleep in #1413
Consolidated to a single Tokio runtime for everything in #1417
Used the correct runtime for internal pipeline thread in #1418
Refactored as dozer-log crate in #1420
Used proper error message for PostgreSQL replication slot creation in #1422
Stored schema of sinks separately in #1423
Added deb generation to release process in #1432
Removed unnecessary workspace.exclude in Cargo.toml in #1430
Bumped h2 from 0.3.16 to 0.3.17 in #1427

Dozer v0.1.17 is available now. Check out the release notes here.

Looking Forward 🌈

As we continue to make strides in improving Dozer, we remain committed to enhancing its features and functionality. Your feedback is invaluable to us, so please don't hesitate to reach out with any suggestions or ideas. Together, we can make Dozer the best it can be!

Full Changelog:

v0.1.16...v0.1.17

Contact us 📬

GitHub: https://github.com/getdozer/dozer
Discord: https://discord.com/invite/3eWXBgJaEQ
Twitter: https://twitter.com/GetDozer
LinkedIn: https://www.linkedin.com/company/getdozer/

Celebrating Dozer v0.1.16 Release and Exiting Stealth Mode 🎉

Abhishek Mishra — Wed, 12 Apr 2023 05:12:30 +0000

Dozer is a powerful, plug-and-play data infrastructure back end designed to help developers build real-time,low-latency data APIs (gRPC and REST) from any data source data apps quickly and efficiently. Dozer streamlines the process of accessing and managing real-time data updates. Key features include support for various data types, optimized join indexes structure, high-performance cache and seamless authorization implementation for gRPC/REST methods.

Key Features:

Here is a summary of all key features released till date.

Open Source: Dozer has transitioned to a fully open-source model and adopted the Apache 2.0 license. We value the contributions and input from our community.
Built with Rust: We care about performance a lot and Dozer is built end to end in Rust
Client Libraries: Dozer has libraries in JS, React and Python for easy integration.
Reactive SQL Engine: Dozer stays in sync by transforming data in real-time, providing a real time view of your data.
Out of the box Data APIs with Protobuf and OpenAPI support: Dozer automatically generates Open API documentation for REST and protobuf definitions for gRPC.
Caching Layer: Dozer stores data in an embedded cache and automatically optimses for efficient queries and is horizontally scalable.
Wide Variety of Connectors: Dozer connects with databases such as Postgres and MySQL, data warehouses such as Snowflake, data lakes and even the Ethereum ledger.

Here is a short video of Dozer in action.

We are Trending on Github 🥳 :

We got featured on TechCrunch 🥳 :

We were recently featured in TechCrunch with the title "Dozer exits stealth to help any developer build real-time data apps 'in minutes'". The article highlights how Dozer is addressing the challenges of stream data processing, by powering fast, read-only APIs directly from any source via a plug-and-play data infrastructure back end.

Founded by Vivek Gudapuri and Matteo Pelati, Dozer has been in testing with select design partners and has now emerged from stealth for any developer to access. The company also revealed it has raised $3 million in seed funding from Sequoia Capital's Indian arm (via its Surge program), Google's Gradient Ventures, and January Capital.

Dozer's unique approach provides developers with an efficient way to access real-time data updates and APIs within a single product. As a result, developers can save time and money by avoiding the need to build and maintain multiple integrations.

We are extremely proud of our achievements and grateful for the support of our investors and design partners. As we continue to grow, we remain committed to delivering the best possible experience for developers and helping them build real-time data apps quickly and efficiently.

Read the full TechCrunch article here.

Community achievements & other citations:

Exciting News: Dozer is now fully open source!

Based on community feedback, Dozer has transitioned to a fully open-source model and adopted the Apache 2.0 license. We value the contributions and input from our community, and we look forward to continuing to grow and evolve together. Stay tuned for a detailed blog post on this topic in the coming week.

Our Reddit Community:

We are delighted to see the love and support from our community on Reddit. The enthusiasm and engagement from developers and users have been invaluable in shaping the future of Dozer. Check out the Reddit thread here: https://www.reddit.com/r/rust/comments/12hb818/dozer_a_scalable_realtime_data_apis_backend/

Key Enhancements of v0.1.16:

Welcome to this week's update on Dozer! We are excited to announce the release of version v0.1.16, which brings a host of new features and improvements. Alongside this update, we are thrilled to share our recent feature in TechCrunch, announcing that Dozer has exited stealth mode!

1. Fixes and Documentation:
We have fixed the README Ubuntu instructions, ensuring a smoother user experience for Ubuntu users. Additionally, we've implemented integration tests on the Github runner, improving the stability and reliability of our codebase.

2. Performance Improvements:
Our team has been hard at work optimizing performance, including using MDB_APPEND when possible and reading object store files in parallel. These enhancements contribute to a more efficient and faster user experience.

3. Feature Additions:
We have implemented several new features, such as authorization in gRPC methods, support for u128 and i128 Field and FieldType, and snapshotting started messages in the ingestion process. These features will provide more functionality and improve the overall user experience.

4. Code Refactoring and Optimization:
Our team has been focusing on optimizing code, including refactoring the Join indexes structure and removing Record::version. This will contribute to a more robust and easier-to-manage codebase.

5. Duration Field and Aggregation Support:
We have added Duration, Field & FieldType support, as well as Duration for SQL representation and aggregation support. These additions will provide users with more flexibility when working with time-based data.

6. Bug Fixes:
We have addressed several issues, such as fixing minor errors while running with duration, updating resolution atomicity and stream operations, and resolving gRPC errors caused by duration. These bug fixes will contribute to a more stable and reliable product.

Read the full release notes here.

Other Citations and Press Releases:

Yahoo News Singapore: "Dozer exits stealth to help any developer build real-time data apps ‘in minutes’" - https://sg.news.yahoo.com/dozer-exits-stealth-help-developer-060042513.html
Surge leads $3m round of SG API firm - https://www.techinasia.com/surge-leads-3m-round-of-sg-api-firm
Singapore's Dozer raises $3M funding led by Sequoia Capital India and Southeast Asia’s Surge - https://technode.global/2023/04/04/singapores-dozer-raises-3m-funding-led-by-sequoia-capital-india-and-southeast-asias-surge/
Sequoia Capital, Others Invest $3 M In Dozer's Seed Round - https://bwdisrupt.businessworld.in/article/Sequoia-Capital-Others-Invest-3-M-In-Dozer-s-Seed-Round/04-04-2023-471607/
Dozer raises $3 Mn led by Sequoia Capital India and Surge - https://entrackr.com/2023/04/dozer-raises-3-mn-led-by-sequoia-capital-india-and-surge/
Early-Stage Startups Prozeal Infra, Zyod, Others Raise Funding - https://www.vccircle.com/earlystage-startups-prozeal-infra-zyod-others-raise-funding
Dozer raises $3 million in funding, emerges from stealth mode - https://economictimes.indiatimes.com/tech/funding/dozer-raises-3-million-in-funding-emerges-from-stealth-mode/articleshow/99237814.cms
Dozer Raises $3 Million In Seed Funding - https://www.entrepreneur.com/en-in/news-and-trends/dozer-raises-3-million-in-seed-funding/448966
Dozer raises US$3M to help engineers instantly build real-time data APIs - https://www.cxotoday.com/press-release/dozer-raises-us3m-to-help-engineers-instantly-build-real-time-data-apis/
B2B apparel startup Zyod, tech startup Dozer raise funding - https://www.zeebiz.com/startups/news-b2b-apparel-startup-zyod-tech-startup-dozer-raise-funding-229028
Data infrastructure platform Dozer raises $3 million in seed funding - https://www.thehindubusinessline.com/info-tech/data-infrastructure-platform-dozer-raises-3-million-in-seed-funding/article66698503.ece
Spendflo, Dozee, Blacksoil, and others raise funds - https://yourstory.com/2023/04/blacksoil-spendflo-dozee-and-others-raise-funds
Dozer Secures US$ 3M in Seed Funding - https://digitalterminal.in/startup/dozer-secures-us-3m-in-seed-funding

Join us in celebrating these milestones as we continue to work towards our mission of empowering developers to build real-time data applications with ease and efficiency!

Stay tuned for more updates, and join our community on GitHub, Discord, Twitter, and LinkedIn.

Changelog

https://github.com/getdozer/dozer/compare/v0.1.15...v0.1.16

Contact us

GitHub: https://github.com/getdozer/dozer
Discord: https://discord.com/invite/3eWXBgJaEQ
Twitter: https://twitter.com/GetDozer
LinkedIn: https://www.linkedin.com/company/getdozer/

DEV Community: Dozer

Creating a Low-Latency Cache with Dozer and PostgreSQL

Why Do We Need a Cache on Top of PostgreSQL?

What's the Problem with Not Having a Cache?

Why Not PostgreSQL Read Replica?

Why Dozer?

How to Use Dozer as a Cache

Architecture Diagram

Conclusion

Secure Your Data APIs with Dozer: Adding Authorization to APIs

TL;DR

Introduction

Server-Side Setup (Express.js)

Database Connection for Server-Side

Admin Login Endpoint

Public New User Endpoint

Public Book Movie Endpoint

Error Handling Middleware

Database Setup For Movie data (PostgreSQL)

Front-End Setup (React)

Login Component

Admin Dashboard Component

Movie List Component

Movies Component

Integrating Dozer JWT in the Application

1. Master JWT Token Generation

2. Custom Access Token Generation

3. Incorporating Tokens in Client-Side API Calls

Wrapping Up:

Announcing the Dozer Appathon!

Dozer Appathon Hacker Guide 👩‍💻👨‍💻

Introduction 🚀

Sample use cases📝

Some sample use cases are

Some wonderful project ideas💡

Prizes 🎁

Resources 📚

Judging Criteria 🧐

Submission Method 📩

Stargazers’ Raffle✨

Unleashing the Power of Dozer Lambda Runtime for Real-time and Event-driven Data Apps

Dozer Lambda Runtime: A Glimpse

JavaScript and Python Bindings for Reading Dozer Logs

React App Example: Room Temperature Monitoring

Conclusion 🎯

🚀 Getting Started with Dozer with Docker: A Step-by-Step Tutorial

🌟 Why Dozer?

🔧 Prerequisites:

🔹 Step 1: Create a docker-compose.yml file

🔹 Step 2: Create a Dozer config file

🔹 Step 3: Run PostgreSQL with sample stock data

🔹 Step 4: Run Dozer with PostgreSQL connector (via Docker)

🎉 Query your API

Dozer Goes Open Source: Empowering the Community to Build Real-time Data Apps

What is Dozer?

Why Open Source?

Getting Started with Dozer

How to Contribute?

Conclusion

Building a Real-time Data App with Dozer, React, and PostgreSQL

🐘 Setting Up PostgreSQL:

⚛️ Building & Running the React App 🚀:

🔌 Configuring Dozer and PostgreSQL Connector:

🌟 Conclusion:

This week at Dozer #5

Key enhancement of v0.1.17 🚀

Switched to Apache License #1404 📜:

Asyncify Connectors #1409 ⚡:

SQL Logic Test Framework #1326 🧪:

APIs for Monitoring and Status Updates #1329 📈:

Graceful API Thread Shutdown #1416 🛠️:

Dozer Lambda Functions: A Glimpse into the Future 🚀🔮 #1424, #1426 🐍🌐:

Other Improvements & Fixes 🔧

Full Changelog:

Contact us 📬

Celebrating Dozer v0.1.16 Release and Exiting Stealth Mode 🎉

Key Features:

We are Trending on Github 🥳 :

We got featured on TechCrunch 🥳 :

Community achievements & other citations:

🔹 Step 1: Create a `docker-compose.yml` file