DEV Community: Gyandeep Singh

Understanding Kafka Keys: A Comprehensive Guide

Gyandeep Singh — Wed, 29 Jan 2025 02:17:23 +0000

Apache Kafka is a robust distributed event-streaming platform widely used for building real-time data pipelines and applications. One of its core features is the Kafka message key, which plays a critical role in message partitioning, ordering, and routing. This blog post explores the concept of Kafka keys, their importance, and practical examples of when and how to use them effectively.

What Are Kafka Keys?

In Kafka, each message consists of two main components:

Key: Determines the partition to which a message will be sent.
Value: The actual data payload of the message.

The Kafka producer uses the key to compute a hash value, which determines the specific partition for the message. If no key is provided, messages are distributed across partitions in a round-robin manner.

Why Use Kafka Keys?

Kafka keys offer several advantages that make them essential in certain scenarios:

Message Ordering:
- Messages with the same key are always routed to the same partition. This ensures that their order is preserved within that partition.
- Example: In an e-commerce system, using an order_id as the key ensures that all events related to a specific order (e.g., "Order Placed," "Order Shipped") are processed in sequence.
Logical Grouping:
- Keys enable grouping related messages together in the same partition.
- Example: For IoT systems, using a sensor_id as the key ensures that data from the same sensor is processed together.
Efficient Data Processing:
- Consumers can process messages from specific partitions efficiently by leveraging keys.
- Example: In a user activity tracking system, using user_id as the key ensures all actions by a user are grouped together for personalized analytics.
Log Compaction:
- Kafka supports log compaction for topics where only the latest value for each key is retained. This is useful for maintaining stateful data like configurations or user profiles.

When Should You Use Keys?

Keys should be used when:

Order matters: For workflows requiring strict ordering of events (e.g., financial transactions or state machines).
Logical grouping is needed: To group related messages (e.g., logs from the same server or events from a specific customer).
Log compaction is enabled: To maintain only the latest state for each key.

However, avoid using keys if:

Order and grouping are not required.
Uniform distribution across partitions is more important (e.g., high-throughput systems).

Examples of Using Kafka Keys (Python)

Below are Python examples using the confluent-kafka library to demonstrate how to use keys effectively when producing messages.

Example 1: User Activity Tracking

Suppose you want to track user activity on a website. Use user_id as the key to ensure all actions by a single user are routed to the same partition.

from confluent_kafka import Producer

producer = Producer({'bootstrap.servers': 'localhost:9092'})

# Send a message with user_id as the key
key = "user123"
value = "page_viewed"
producer.produce(topic="user-activity", key=key, value=value)
producer.flush()

Here, all messages with user123 as the key will go to the same partition, preserving their order.

Example 2: IoT Sensor Data

For an IoT system where each sensor sends temperature readings, use sensor_id as the key.

from confluent_kafka import Producer

producer = Producer({'bootstrap.servers': 'localhost:9092'})

# Send a message with sensor_id as the key
key = "sensor42"
value = "temperature=75"
producer.produce(topic="sensor-data", key=key, value=value)
producer.flush()

This ensures that all readings from sensor42 are grouped together.

Example 3: Order Processing

In an order processing system, use order_id as the key to maintain event order for each order.

from confluent_kafka import Producer

producer = Producer({'bootstrap.servers': 'localhost:9092'})

# Send a message with order_id as the key
key = "order789"
value = "Order Placed"
producer.produce(topic="orders", key=key, value=value)
producer.flush()

Best Practices for Using Kafka Keys

Design Keys Carefully:
- Ensure keys distribute messages evenly across partitions to avoid hotspots.
- Example: Avoid using highly skewed fields like geographic location if most users are concentrated in one area.
Monitor Partition Distribution:
- Regularly analyze partition loads to ensure balanced distribution when using keys.
Use Serialization:
- Serialize keys properly (e.g., JSON or Avro) for compatibility and consistency with consumers.

Conclusion

Kafka keys are a powerful feature that enables ordered processing and logical grouping of messages within partitions. By carefully designing and using keys based on your application's requirements, you can optimize Kafka's performance and ensure data consistency. Whether you're building an IoT platform, an e-commerce application, or a real-time analytics system, understanding and leveraging Kafka keys will significantly enhance your data streaming architecture.

macOS: Universally Mute teams mic with keyboard

Gyandeep Singh — Tue, 26 Oct 2021 20:19:56 +0000

Issue

A lot of folks are tired of managing microphone mute/unmute when on teams calls. Since you have to go to teams call and than use a keyboard or hit the microphone icon. That a distraction if you are doing multitasking (which almost everyone is doing).

Solution

Use a universal keyboard shortcut to mute/unmute microphone in teams without going to teams app. Here how we do it:

Step 1

Open up Automator app on your macOS
- Keyboard shortcut: CMD + Space and than search for Automator
Hot quick action document
Hit Choose

Step 2

Let design the automation so that we can run it using a keyboard shortcut.

Change the Workflow receive current option to no input
Leave everything else to its default.
On the left panel, look for the action Run Applescript
- Double click on it so it moves to the right on your new action panel
Replace the code in it with the below code

on run {input, parameters}

    tell application "Microsoft Teams"
        activate
        tell application "System Events"
            keystroke "m" using {shift down, command down}
        end tell
    end tell

    return input
end run

Go to file on the top-left menu and hit save.
Give it some name (example teams-mute)

Step 3

Let's get to the keyboard shortcut

Go to System preferences -> keyboard -> shortcuts
Click services in the left panel and scroll down to look for your service. It should be under the general section.
Assign a shortcut to it.
- I used F1

Troubleshooting

When you execute the shortcut it might ask you for permission to change the system.
- Provide that permission in System preferences -> security and privacy
- select Accessibility in left panel and make sure Automator is present and select in the right right panel.
If your action is not working than go to Accessibility (as mentioned in previous step) section and remove Automator and than add it back again using the +/- buttons

Hope this helps and share your thoughts.

React - useRef used in conjunction with useEffect

Gyandeep Singh — Thu, 02 Jul 2020 00:00:00 +0000

Problem

Let's say you have to call an external API to submit a name change and API count number. Every time the name changes you have to call the remove name API and then call the add name API. Alongside this you need to count how many times the API was called regardless of which API you call and also send the count number to the API as well.

import React, { useEffect, useState } from "react";

export default function RefTest() {
  const [text, setText] = useState("");
  const [name, setName] = useState("");
  const [cnt, setCnt] = useState(0);

  // DOM handlers
  const inputChangeHandler = ({ target }) => setText(target.value);
  const sendHandler = () => setName(text);

  // HOOK
  useEffect(() => {
    console.log(`API - Add name: ${name} cnt: ${cnt + 1}`);
    setCnt(cnt + 1);

    return () => {
      console.log(`API - Remove name: ${name} cnt: ${cnt + 1}`);
      setCnt(cnt + 1);
    };
  }, [name, setCnt]);

  return (
    <div>
      <input type="text" value={text} onChange={inputChangeHandler} />
      <button onClick={sendHandler}>Send</button>
      <div>Name: {name}</div>
      <div>Count: {cnt}</div>
    </div>
  );
}

Note: All these examples can be better coded but I am trying to demonstrate a scenario.

There are couple of issues in the code above:

ESLint issue where we have not added cnt as a dependency.
If you run the code the cnt is not correct because of closure it maintains an older value of cnt before it can increment.

Adding `cnt` as a dependency

Note: Please do not add cnt as dependency as it will cause an infinite render. But if you want to try, do it on a page which you can kill easily.

The main issue with this approach apart from the infinte render is that it's going to start calling the API even when the cnt changes. Which we don't want as we only want to call the API when name changes.

Solution

Maintain the cnt as a ref so that it can be updated and mutated without impacting the useEffect hook execution cycle.

import React, { useEffect, useState, useRef } from "react";

export default function RefTest() {
  const [text, setText] = useState("");
  const [name, setName] = useState("");
  const [cnt, setCnt] = useState(0);
  const cntRef = useRef(cnt);

  // DOM handlers
  const inputChangeHandler = ({ target }) => setText(target.value);
  const sendHandler = () => setName(text);

  // HOOKS
  useEffect(() => {
    console.log(`API - Add name: ${name} cnt: ${cntRef.current++}`);
    setCnt(cntRef.current);

    return () => {
      console.log(`API - Remove name: ${name} cnt: ${cntRef.current++}`);
      setCnt(cntRef.current);
    };
  }, [name, setCnt]);

  return (
    <div>
      <input type="text" value={text} onChange={inputChangeHandler} />
      <button onClick={sendHandler}>Send</button>
      <div>Name: {name}</div>
      <div>Count: {cnt}</div>
    </div>
  );
}

At this point I am using cnt in the state as well so that I can display it on UI otherwise it's not needed.

Conclusion

Anytime you want the useEffect to execute for state S1 but you want to use other state values inside it but don't want other states to trigger the useEffect for those states than use useRef hook to store the other states.
This is particularly helpful if you subscribe to an API and in your handler you want to do something with the incoming data combined with other state data (not S1) before handing it over to some other operation.

Garage door operations using Raspberry Pi

Gyandeep Singh — Tue, 28 Apr 2020 02:57:10 +0000

Expectation

In this post I am going to explain how I automated certain garage door operation using raspberry pi. If you find it useful then you can follow this guide and build your own too.

Open/Close garage door from your phone
Know the status (open or close) of the garage door.
Auto close garage door in 5 mins.
Override auto close in 5 mins.
Know the time it was last closed/opened.

Note: This is my first take at this and I am still learning.

Parts needed

Raspberry Pi - ~$35
1. You can use any kind 3 or 4. I used 4 because I got it for $5 price difference.
2. Buy link - Microcenter
Breadboard jumper wires - Amazon
Relay module - Amazon
20 Gauge 2 pin wire - Amazon
1. You can buy a lower gauge wire also.
Magnetic switch - Amazon

Setup

Raspberry Pi

I am not going to go into detail on how to setup Raspberry Pi as most of the setup instructions are available out in the wild.

Here is what I used:

Youtube video - https://www.youtube.com/watch?v=BpJCAafw2qE
Docs - https://crosstalksolutions.com/getting-started-with-raspberry-pi-4/

Thanks to @crosstalksol for the awesome setup guide.

Note: Please read up on raspberry pi GPIO pins as it will help you when you work on this project.

gpio module on raspberry pi

Inside raspberry pi, open a terminal and write gpio -v. We need atleast 2.52 version.

If an update is needed, then follow this: http://wiringpi.com/wiringpi-updated-to-2-52-for-the-raspberry-pi-4b/

IP address

First, note down the current IP address of the Pi. Second. make sure you make the ip of the Pi static because we don't want to keep changing it after restarts. This is not a necessary step, but it helps while you work on this.

In my setup i have google fiber, so I logged into the router setup page and in the client list select raspberry pi. Activated the reserved flag on pi client so that it keeps the same ip address. Feel free to google on how to do this on your router.

Computer

I installed Putty on my windows machine to ssh into raspberry pi from my windows laptop.
Feel free to use any other ssh client which works best for you.

Let us get to work

Hardware setup

Let's setup all the different parts to get door working first. At this point don't worry about all these connection as we will go in detail when we start programming them.

We will use BCM conventions for GPIO pin numbers. When using pin numbers those will denote physical pin numbers. Run gpio readall inside bash session on your raspberry pi to get more details.

Find the manual button wire connection into your garage door

I have a Chamberlain garage door opener, I was able to trace the button (from the wall) wires going into one terminal in your garage door opener. It looks like a thin looking wire as compared to regular electric wires in your house. Do not confuse the wires coming from the safety sensors hooked up to your garage door sides. In my case red was positive and white was negative.

Connect wires from your garage opener sockets to relay module

Make sure you connect the negative (black in pic) to center terminal of the relay module and hot (red in pic) wire to left terminal on the relay module.
Concept is that when we connect these two wires, it completes the circuit which in turn trigger the garage open/close operation.

Connect relay to raspberry pi

Connect GND to pin 6 (other orange looking wire)
Connect VCC to pin 2 ie 5v supply (red wire in pic)
Connect IN2 to GPIO 4 (orange wire in pic)

IN2 pin is what will be used to trigger the changes. Other 2 wires is to provide power (5v) to the relay.

Connect magnetic switch to raspberry pi

Connect COM to ground on pin 6
- Black wire from the switch to orange looking wire
Connect NO to GPIO 2 (red wire in pic)
- This is pin is an pull-up type which means it outputs 3.3v
- I am still trying to better understand this area.
- I think you can use other pins as pull-up by configuration but i did not try it.

Mechanical switch connections

Connection to the raspberry pi

Mechanical switch to the garage door

Hook up the mechanical switch on the side of the garage door panel as show in the picture above.
Hook up the other piece on the movable garage door so that when its closed it comes very close to the other side.
Basically, it detects close when they are close to each other and open when they are not close to each other.

Software

Complete code: Github

Using `onoff` npm package

This package only works on Linux so make sure you code on the raspberry pi
- You can use vscode via ssh to code inside raspberry pi (Google is your friend here)
Lets setup GPIO 4 to out since we are going to use that pin to trigger the open close.
Configure GPIO 2 to in as we are going to read high (1) and low (0) based on the magnetic switch changes.
- Configure this pin to call out function in both direction (high and low)
- This library allows to read in one direction also.

const buttonTrigger = new Gpio(4, "out", undefined, {
    reconfigureDirection: false
});
const doorSensor = new Gpio(2, "in", "both", {
    reconfigureDirection: false,
    debounceTimeout: 1000
});

Open close action

When we want to close/open the garage door we are going to output a low on GPIO 4 then wait for 1 second and then output high.
The reason we flip flop is that we want to just output a pulse and then get back to its original state.
Think of this as a button click, press and release action kind of a deal.
When you do this at this point it sends the signal to the relay where its output state is disconnected.
When the trigger happens, it completes the relay switch circuit on the other side.
That complete circuit allows the current to flow from inside the garage opener and that's why its start to work.
Sorry I am not an electrician so that's the best I can explain.

export const openCloseGarage = async () => {
    buttonTrigger.write(Gpio.LOW);
    await sleep();
    buttonTrigger.write(Gpio.HIGH);
};

Bonus stuff

If you use the garage module, I have setup then it should get you all started. Follow these steps inside your raspberry pi:

git clone https://github.com/gyandeeps/garage.git
npm install (This is going to take a long time)
sudo sh prod.sh

Phone

On your phone you should be able to hit http://<ip address to pi>:3000
Then open close the garage.

Closing thoughts

This is my first project and i am still trying to figure a lot of things out.
Please provide feedback or anything which can be improved.
Any ideas on what else can be done.

Overall picture (needs to find a good place for this but they work for now)

References

Code Ownership

Gyandeep Singh — Thu, 16 Apr 2020 21:40:04 +0000

What are you thoughts around code ownership?
What would you pick in below list and why?

From an ownership standpoint, which one would you pick?#coding @JavaScriptDaily #softwaredevelopment
— Gyandeep Singh (@gyandeeps) April 16, 2020

Is React a dev or prod npm dependency?

Gyandeep Singh — Fri, 03 Apr 2020 20:20:03 +0000

Do you add react in your dependencies or devDependencies inside package.json and explain why you choose that?

My thoughts

dependencies

It can be here since its part of the application which gets shipped. Or you can say its used by the application/library at runtime.

devDependencies

It is still part of the dependency but its needed during release/compile time but for consumers they don't need it since bundled code already has that in it.

React - How to use keys to avoid using getDerivedStateFromProps

Gyandeep Singh — Sat, 15 Sep 2018 00:00:00 +0000

Background

Until today I used key in react only when react dev threw warnings inside the console. Other than that I never cared about keys and never invested time to understand them. This is me being brutally honest to you all readers.

Problem

Have a component which display person name form based on different name and id passed to it. So using that form user can edit person details.

import React, { Component } from "react";
import ReactDOM from "react-dom";

class NameForm extends Component {
    constructor(props) {
        super(props);
        this.state = {
            userId: props.userId || 0,
            name: props.name || ""
        };
    }

    handleChange = event => {
        this.setState({ name: event.target.value });
    };

    handleSubmit = event => {
        console.log("A name was submitted: " + this.state.name);
        event.preventDefault();
    };

    render() {
        return (
            <form onSubmit={this.handleSubmit}>
                <label>
                    Name:
                    <input
                        type="text"
                        value={this.state.name}
                        onChange={this.handleChange}
                    />
                </label>
                <input type="submit" value="Submit" />
            </form>
        );
    }
}

class App extends Component {
    constructor(props) {
        super(props);
        this.state = {
            users: {
                1: "A",
                2: "B",
                3: "C"
            },
            editUserId: "new"
        };
    }

    onChange = event => {
        const value = event.target.value;

        this.setState(() => ({
            editUserId: value
        }));
    };

    render() {
        const { editUserId, users } = this.state;

        return (
            <div>
                <span>Select Edit userId: </span>
                <select onChange={this.onChange}>
                    <option value="new">New User</option>
                    {Object.entries(users).map(([userId, name]) => (
                        <option value={userId}>{name}</option>
                    ))}
                </select>
                <NameForm userId={editUserId} name={users[editUserId] || ""} />
            </div>
        );
    }
}

const rootElement = document.getElementById("root");
ReactDOM.render(<App />, rootElement);

Observations

Form name field displays no data since by default its set to new user.
Now when you select user A from the select list, it still shows name as empty.
Since react thinks its the same component so it will not call the constructor again.
At this point you will be thinking that I need to change state of NameForm component based on prop changes.
- This is called syncing state with props.
And now you will be convinced that you need to use getDerivedStateFromProps static method because you need to change state based on prop change.
This is where you went wrong.

Solution

Let me quote a line from React documentation:

If you want to “reset” some state when a prop changes, consider either making a component fully controlled or fully uncontrolled with a key instead.

Make NameForm component controlled by using key for React elements.

That's it.

import React, { Component } from "react";
import ReactDOM from "react-dom";

class NameForm extends Component {
    constructor(props) {
        super(props);
        this.state = {
            userId: props.userId || 0,
            name: props.name || ""
        };
    }

    handleChange = event => {
        this.setState({ name: event.target.value });
    };

    handleSubmit = event => {
        console.log("A name was submitted: " + this.state.name);
        event.preventDefault();
    };

    render() {
        return (
            <form onSubmit={this.handleSubmit}>
                <label>
                    Name:
                    <input
                        type="text"
                        value={this.state.name}
                        onChange={this.handleChange}
                    />
                </label>
                <input type="submit" value="Submit" />
            </form>
        );
    }
}

class App extends Component {
    constructor(props) {
        super(props);
        this.state = {
            users: {
                1: "A",
                2: "B",
                3: "C"
            },
            editUserId: "new"
        };
    }

    onChange = event => {
        const value = event.target.value;

        this.setState(() => ({
            editUserId: value
        }));
    };

    render() {
        const { editUserId, users } = this.state;

        return (
            <div>
                <span>Select Edit userId: </span>
                <select onChange={this.onChange}>
                    <option value="new">New User</option>
                    {Object.entries(users).map(([userId, name]) => (
                        <option value={userId}>{name}</option>
                    ))}
                </select>
                <NameForm key={editUserId} userId={editUserId} name={users[editUserId] || ""} />
            </div>
        );
    }
}

const rootElement = document.getElementById("root");
ReactDOM.render(<App />, rootElement);

Observations

We added key to the NameForm component.
- This tells react to create a new instance of NameForm based on the key.
- With that, component constructor will get called and the state is maintained.
Now when you change values from the dropdown the form values will change.

Conclusion

When you think about using getDerivedStateFromProps, take a step back and think through the problem and see if you can use key.
That's why you will see people say that you should avoid or use getDerivedStateFromProps rarely.
This was an eye opener for me when I actually used it. While reading docs I didn’t get the whole point.

// Detect dark theme var iframe = document.getElementById('tweet-1041147850745372672-956'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1041147850745372672&theme=dark" }

AWS AppSync App with React and Apollo

Gyandeep Singh — Mon, 11 Jun 2018 00:00:00 +0000

Introduction

Walk through on how to build an application using AWS AppSync and what all things are needed on the front end side to get the whole system working. We will not be covering any basics as there are so many blog posts out there for that.

Prerequisites

Basic understanding of:

React
React router
AWS AppSync setup inside AWS console
Different Apollo libraries
GraphQL terminologies - Query, Mutations and Subscriptions

Resources

Building Serverless React GraphQL Applications with AWS AppSync by Nader Dabit

If you don’t have basic understanding on any of these topics then I would highly recommended learning about that before proceeding ahead.

What are we going to learn

Wire up AWS AppSync with all the config
Authentication using AWS Cognito integration
Manage mutation and resolvers
Manage client state and server side state using apollo

Motivations

Not able to find any docs which explains how all these pieces work together.
Not much info for advanced level stuff vs basic getting started.
Spent so much time trying to figure out all these pieces and how they work together.

Setup

Libraries

aws-amplify, aws-amplify-react - For Auth and other AWS calls
aws-appsync, aws-appsync-react - Wrapper around apollo client which manages communication to graphql api endpoint
react-apollo - Apollo client side library for React framework
Other apollo libraries like apollo-link-http, apollo-link-state and apollo-cache-inmemory

Note: Most of the blogs out their doesn’t cover all the aspects of an app and that why they only use sub-set of these libraries.

Building blocks

Basic app loading

I assume you already have something loading just using react (create using create-react-app). We are going to build on top of that. So let’s say you have a div rendering with hello world for now.

import * as React from "react";
import * as ReactDOM from "react-dom";
import { BrowserRouter } from "react-router-dom";

class App extends React.Component {
    render() {
        return <div>Hello World</div>;
    }
}

ReactDOM.render(
    <BrowserRouter>
        <App />
    </BrowserRouter>,
    document.getElementById("root")
);

Add AWS AppSync layer to connect to backend

We are going to use aws-amplify library provided by AWS AppSync team. This would take care of talking directly to AWS resources like cognito for Auth, analytics api, pubsub, API calls, etc. For more detailed info, please use the readme.

import * as React from "react";
import * as ReactDOM from "react-dom";
import { BrowserRouter } from "react-router-dom";
import Amplify, { Auth } from "aws-amplify";

Amplify.configure({
    Auth: {
        // REQUIRED - Amazon Cognito Identity Pool ID
        identityPoolId: "XX-XXXX-X:XXXXXXXX-XXXX-1234-abcd-1234567890ab",
        // REQUIRED - Amazon Cognito Region
        region: "XX-XXXX-X",
        // OPTIONAL - Amazon Cognito User Pool ID
        userPoolId: "XX-XXXX-X_abcd1234",
        // OPTIONAL - Amazon Cognito Web Client ID
        userPoolWebClientId: "XX-XXXX-X_abcd1234"
    }
});

class App extends React.Component {
    render() {
        return <div>Hello World</div>;
    }
}

ReactDOM.render(
    <BrowserRouter>
        <App />
    </BrowserRouter>,
    document.getElementById("root")
);

Once you have configured Amplify then in any other module you can import a particular artifact and use it.
- Docs - https://aws.github.io/aws-amplify/media/developer\_guide
In this case we are only using Auth so we have configured that portion only. But for API or any other modules, you can add a block and its configurations.
As per the above setup you should be good to go and use Auth module anywhere in your app.

Add cache and state management

We are going to use aws-appsync and aws-appsync-react libraries to create local cache where the data from graphql and your local state will be saved. The concept is more like Redux but here behind the scene aws-appsync uses Apollo cache and its libraries to do the heavy lifting.

import * as React from "react";
import * as ReactDOM from "react-dom";
import { BrowserRouter } from "react-router-dom";
import Amplify, { Auth } from "aws-amplify";
import { ApolloProvider } from "react-apollo";
import { ApolloLink } from "apollo-link";
import { InMemoryCache } from "apollo-cache-inmemory";
import AWSAppSyncClient, { createAppSyncLink } from "aws-appsync";
import { withClientState } from "apollo-link-state";

Amplify.configure({
    Auth: {
        // REQUIRED - Amazon Cognito Identity Pool ID
        identityPoolId: "XX-XXXX-X:XXXXXXXX-XXXX-1234-abcd-1234567890ab",
        // REQUIRED - Amazon Cognito Region
        region: "XX-XXXX-X",
        // OPTIONAL - Amazon Cognito User Pool ID
        userPoolId: "XX-XXXX-X_abcd1234",
        // OPTIONAL - Amazon Cognito Web Client ID
        userPoolWebClientId: "XX-XXXX-X_abcd1234"
    }
});

const cache = new InMemoryCache();
let nextTodoId = 1;
const stateLink = withClientState({
    cache,
    defaults: {
        todos: []
    },
    resolvers: {
        Mutation: {
            addTodo: (_, { text }, { cache }) => {
                const query = gql`
                    query GetTodos {
                        todos @client {
                            id
                            text
                            completed
                        }
                    }
                `;
                const previous = cache.readQuery({ query });
                const newTodo = {
                    id: nextTodoId++,
                    text,
                    completed: false,
                    __typename: "TodoItem"
                };
                const data = {
                    todos: previous.todos.concat([newTodo])
                };
                cache.writeData({ data });
                return newTodo;
            }
        }
    }
});

const authConfig = {
    type: appSyncAtrributes.authenticationType,
    jwtToken: async () =>
        (await Auth.currentSession()).getAccessToken().getJwtToken()
};

const client = new AWSAppSyncClient(
    {
        disableOffline: true,
        url: appSyncAtrributes.graphqlEndpoint,
        region: appSyncAtrributes.region,
        auth: authConfig,
        complexObjectsCredentials: () => Auth.currentCredentials()
    },
    {
        cache,
        link: ApolloLink.from([
            stateLink,
            createAppSyncLink({
                url: appSyncAtrributes.graphqlEndpoint,
                region: appSyncAtrributes.region,
                auth: authConfig,
                complexObjectsCredentials: () => Auth.currentCredentials()
            })
        ])
    }
);

class App extends React.Component {
    render() {
        return <div>Hello World</div>;
    }
}

ReactDOM.render(
    <BrowserRouter>
        <ApolloProvider client={client}>
            <App />
        </ApolloProvider>
    </BrowserRouter>,
    document.getElementById("root")
);

A lot of stuff happened in the code above. Lets discuss the important pieces here.

Cache/State

Based on all the articles out their, when you use AWSAppSyncClient directly by providing the first parameter you automagically start maintaining local cache of your remote calls. i.e. when you fetch data from your graphql api then its stored inside cache. But we want to maintain some client side local state as well, which is not coming from the grapghql api. We do that using apollo-link-state.

Most important pieces to understand here are:

createAppSyncLink - Default state setup done by the aws-appsync library.
withClientState - we create the local state maintained by the app on the front-end. In this case it’s the todo state.
ApolloLink.from - Using this we combine the output from above 2 commands to get a single entry point into the state. Think of this as merging the remote and local state into a single state.
ApolloProvider - It’s like react-redux provider which exposes the client downstream into other components.

Using state in component

Here I am just going to focus on the App component as all the other code around it remains the same. Ideally you should create the App component as a separate file and import it.

import * as React from "react";
import { Mutation, Query } from "react-apollo";
import gql from "graphql-tag";

const GET_TODOS = gql`
    {
        todos @client {
            id
            completed
            text
        }
    }
`;

const GET_ORDERS = gql`
    {
        listOrders {
            items {
                id
                name
            }
        }
    }
`;

const ADD_TODO = gql`
    mutation addTodo($text: String!) {
        addTodo(text: $text) @client {
            id
        }
    }
`;

class App extends React.Component {
    render() {
        return (
            <Query query={GET_ORDERS}>
                {({ data }) => (
                    <Mutation mutation={ADD_TODO}>
                        {(addTodo) => (
                            <>
                                <div>{JSON.stringify(data)}</div>
                                <button
                                    onClick={() =>
                                        addTodo({
                                            variables: { text: "gyandeep" }
                                        })
                                    }
                                >
                                    add
                                </button>
                            </>
                        )}
                    </Mutation>
                )}
            </Query>
        );
    }
}

Most important pieces to understand here are:

Query and Mutation - Components for graphql stuff.
onClick addTodo call - Add todo calls a mutation on the client side using the directive @client inside the mutation definition. This tells the underlying apollo infrastructure that this graphql command is for local changes only.
- In the main file in stateLink using withClientState, we defined the mutation resolver for add todo that basically writes to the local cache and then the components refreshes to read the values.
- Think of this as redux actions and reducers.
GET_ORDERS - This graphql query doesn’t use the @client so it hit’s the graphql interface over the network and then when data comes back it updates the cache automagically.

Authentication

After all the setup done, if you want your route in this case App to be authenticated before procedding then you can use the helper to achieve that.

import { withAuthenticator } from "aws-amplify-react";

// App class definition here as seen above

export default withAuthenticator(App);

So before this components gets rendered it will route to login page. For more details on this I would recommend using the guide here. It explains everything very nicely.

Conclusion

You should not use Redux as it doesn’t work nicely with apollo cache. I know its hard but once you move over to apollo you will be fine.
- It took me some time to understand.
Think of your resolvers as redux reducers, I would recommend writing them separately.
Think of all gql queries as actions, I would recommend writing them separately as well.

Notes

Wrote this because I struggled with all these pieces (different libraries) a lot and complained a lot to people and twitter.
I am just 2-3 weeks old in AppSync world, so this post is like a brain dump.

Coding with TypeScript, Prettier and ?

Gyandeep Singh — Fri, 09 Mar 2018 20:09:12 +0000

Recently I started working on a new web app project and we used TypeScript on both front-end and the server. TypeScript was a game changer just from the types.

We need static code analysis and code formatting to make the code readable, correct and <whatever>. Here is how I solved this in my code setup:

Code analysis -> TypeScript compiler
Code formatting -> Prettier

That's it. Using pre-commit hooks it made my life so easy that i don't even have to think about code formatting as its done for me and consistent across the board. All I have to think about is the actual code.

Question: Do I need ESlint/TSLint to check my code for non-style issues?

Because typescript compilers already does that for you.

I am just asking because I have not felt the need for it and I know everybody uses it. So i am thinking what am I missing?

Please share your thoughts :)

Understanding async/await with Array reduce method

Gyandeep Singh — Sun, 10 Dec 2017 00:00:00 +0000

I was working on a project when I ran into a use case where I needed to use Array’s reduce method with some async/await (Promise based) network calls. That’s when things actually became confusing and complicated. Let’s learn about how to use async/await with array reduce by running through an example.

Problem

Get all the pull request objects using GitHub API from a repo who have a commit message not starting with Fix: or Docs:.

We will use a lot of helper methods to talk to GitHub API and do some other operation but will not discuss those in this post as we want to focus async/await with array reduce.

Solution

Let create a function named getNonSemverPatchPRs which is going to return all the PR objects from a repo who doesn’t not qualify as semver patch PR based on a the commit message of the PR. Expectation here will be that when we call the getNonSemverPatchPRs it should return a Promise and that promise should resolve with a array of PR objects.

const getNonSemverPatchPRs = async () => {
    const allOpenPrs = await getAllOpenPRs();

    return allOpenPrs.reduce((collection, pr) => {
        // 1. This is where we want to get all the commits of the PR in context
        // 2. Then we want to see if the commit message of the first commit message starts with `Fix:` or `Docs:`
        // 3. If yes then ignore it otherwise add it to the collection.
    }, []);
};

Fetch all the commits for a PR

To complete step 1 we need to perform a network call to fetch all the commit for a PR. Now this call will be promised based. Since we have to make await the call we need to make the reduce handler function async.

const getNonSemverPatchPRs = async () => {
    const allOpenPrs = await getAllOpenPRs();

    return allOpenPrs.reduce(async (collection, pr) => {
        const allCommits = await getAllCommitsForaPR(pr.number);

        // 2. Then we want to see if the commit message of the first commit message starts with `Fix:` or `Docs:`
        // 3. If yes then ignore it otherwise add it to the collection.
    }, []);
};

Check the commit message from the first commit

Now we are going to check the commit message of the first commit to see if it starts with Fix: or Docs:. This call is synchronous call to a helper function.

const getNonSemverPatchPRs = async () => {
    const allOpenPrs = await getAllOpenPRs();

    return allOpenPrs.reduce(async (collection, pr) => {
        const allCommits = await getAllCommitsForaPR(pr.number);

        const isNotSemverPatchPR = checkCommitMessageForPatch(allCommits[0]);
        // 3. If yes then ignore it otherwise add it to the collection.
    }, []);
};

Add to collection if PR is not semver patch PR

Now we are going to check if it’s not a semver patch PR then add to collection of reduce otherwise ignore it.

const getNonSemverPatchPRs = async () => {
    const allOpenPrs = await getAllOpenPRs();

    return allOpenPrs.reduce(async (collection, pr) => {
        const allCommits = await getAllCommitsForaPR(pr.number);

        const isNotSemverPatchPR = checkCommitMessageForPatch(allCommits[0]);

        if (isNotSemverPatchPR) {
            collection.push(pr);
        }

        return collection;
    }, []);
};

Problem inside reduce with async function handler

Thought: Based on your knowledge of async/await and array reduce, you would think that it will keep pushing the pr objects to the collection and return the collection so that next iteration of the reduce can use it and keep adding stuff to the collection.
Reality: The reduce callback function is an async function so it will always returns a Promise. Since it returns a Promise the value of the collection parameter is not an array but its a Promise from the previous execution.
Solution: Since collection always contains a Promise then we need to resolve that promise to get the response which will finally become our collection and then we can keep pushing stuff to it and then return that as part of the function.
- Make the initial value of reduce to be a dummy resolved Promise and then we can keep resolving the promises returned by every call.
- Make a collection inside the function which can be extracted by resolving the passed in Promise.

const getNonSemverPatchPRs = async () => {
    const allOpenPrs = await getAllOpenPRs();

    return allOpenPrs.reduce(async (previousPromise, pr) => {
        const collection = await previousPromise;
        const allCommits = await getAllCommitsForaPR(pr.number);

        const isNotSemverPatchPR = checkCommitMessageForPatch(allCommits[0]);

        if (isNotSemverPatchPR) {
            collection.push(pr);
        }

        return collection;
    }, Promise.resolve([]));
};

Conclusion

Would recommend running through the above example and try to put breakpoints to better understand the flow. Feel free to use the JSBin to play around.
async function always return a Promise that is why the reduce function starts accumulating previous Promise. When this promise resolves it gives you the real collection retuned inside the function.

I am still working through this but I wanted to write something about this so that I can help others who run into this. Feel free to leave feedback in the comments below.

Git helpers - Simplify your git workflow

Gyandeep Singh — Sat, 12 Aug 2017 20:05:09 +0000

This was originally posted here

Being a git lover, I got tired of typing same commands over and over again. This motivated me to build some alias for my git workflow I use every day. Most of the alias was inherited from Nicholas C. Zakas's gist. I took that a step further by creating more alias which I use every time I contribute to open source projects like Eslint.

Commands

Actual code for all the commands is here

Work on an issue

When you plan to work on an issue on an open source project, you will want to create a new branch on your project and then start work on it. For that you have to go through couple of steps. To stream line that process now on top of your project you can just run the command:

ws 34

This command will update the master branch with the updates from the remote and then create a new branch on it with name issue34.

Remove a branch after work is done

Once you are done with your work on the branch. Then you can simply run the below command to delete the branch from local and remote git.

wd 34

Update your branch (all changes commited) with master

let’s say after working on your branch, you want to update your branch with latest content from the master. This is important in scenarios where you want to rebase your branch with latest content from master branch.

update

Update your branch (uncommited changes) with master

let’s say after working on your branch, you want to update your branch with latest content from the master. But at this time you have not commited your changes. Using updateD, it will stash your changes then update your branch from master and then pick the changes from stash and apply it back on the current branch.

updateD

Squash your commits

I always prefer to keep my changes history clean by squashing my commits. Once you are working on a branch and you want to squash your commits, simple run the following command to rebase. You can pass in a parameter of how many commits you want to include in your rebase from the HEAD. By default it will run rebase on last 2 commits.

rebase 4 # This will rebase on last 4 commits, default is 2

Push your changes to remote

After you have finished your work, you want to push your changes to remote. Here mostly you will be doing a regular push but if you have done some rebases on your branch then your only option is to force push. We use --force-with-lease flag for force ppush just to be little more safe.

Regular push:

push # This will push your branch changes but will not push on master

Force push:

fpush # This will push your branch changes but will not push on master

Both commands will not allow you to push it to master because it considered a good practice to always branch off your changes and then merge it to master using a PR. This is completely my view.

Pull changes from origin

Running this command will pull thhe changes from upstream and merge them into thhe current branch. It uses --rebase --prune flags.

pull

Update fork

Quick way to update your fork from the upstream repo. It will go to the master branch, get thhe upstream master, merge that upstream master into local master and then push thhe master to origin master. Its a complete cycle.

uf

It expects you to setup upstream before running this command.

Quick commit

Quick way to commit your changes by supplying the commit message. Itt will add the currently changed files using the -a flag.

c "<message>"

Log commits

Quick and easy way to view your commit log in oneline. Its formatted in a way where you can see the commit message, date and author along with some other info.

log

Git status

Get the current status of the branch.

Setup

To use all these command, follow the steps below to setup:

Go to your user directory, for windows user it will be C:\Users\<your user>.
Create a .bashrc file, if you do not have that file.
Add/append the content from the below gist to that file.
Open bash window. If already open then close it and then reopen bash window.
Run alias command and you should see the list of all the alias defined.

Note: Feel free to change the function based on your need.

Gist for all the above commands

Click here to see the gist

I hope this helps people to streamline their git workflows. Please share your suggestions.

Simple way to write JSON to file

Gyandeep Singh — Mon, 24 Jul 2017 19:15:27 +0000

Simple way to write JSON to file

DEV Community: Gyandeep Singh

Understanding Kafka Keys: A Comprehensive Guide

What Are Kafka Keys?

Why Use Kafka Keys?

When Should You Use Keys?

Examples of Using Kafka Keys (Python)

Example 1: User Activity Tracking

Example 2: IoT Sensor Data

Example 3: Order Processing

Best Practices for Using Kafka Keys

Conclusion

macOS: Universally Mute teams mic with keyboard

Issue

Solution

Step 1

Step 2

Step 3

Troubleshooting

React - useRef used in conjunction with useEffect

Problem

Adding cnt as a dependency

Solution

Conclusion

Garage door operations using Raspberry Pi

Expectation

Parts needed

Setup

Raspberry Pi

gpio module on raspberry pi

IP address

Computer

Let us get to work

Hardware setup

Find the manual button wire connection into your garage door

Connect wires from your garage opener sockets to relay module

Connect relay to raspberry pi

Connect magnetic switch to raspberry pi

Mechanical switch to the garage door

Software

Using onoff npm package

Open close action

Bonus stuff

Phone

Closing thoughts

References

Code Ownership

Is React a dev or prod npm dependency?

My thoughts

dependencies

devDependencies

React - How to use keys to avoid using getDerivedStateFromProps

Background

Problem

Observations

Solution

Observations

Conclusion

AWS AppSync App with React and Apollo

Introduction

Prerequisites

Resources

What are we going to learn

Motivations

Setup

Libraries

Building blocks

Basic app loading

Add AWS AppSync layer to connect to backend

Add cache and state management

Cache/State

Using state in component

Authentication

Conclusion

Notes

Coding with TypeScript, Prettier and ?

Understanding async/await with Array reduce method

Problem

Solution

Fetch all the commits for a PR

Check the commit message from the first commit

Adding `cnt` as a dependency

Using `onoff` npm package