DEV Community: Shubham Naik

Let's learn about protocol buffers

Shubham Naik — Thu, 09 Jul 2020 16:16:50 +0000

This post is a mirror of a post I wrote on my own blog.

Let's learn about Protocol Buffers

Protocol Buffers or “Protobufs” is a term often thrown around the rooms of big tech companies when designing application systems. Application systems can contain hundreds of thousands of machines all communicating with each other. At that scale, many companies try to optimize in any way possible—Protocol Buffers is tool you can use to send data between your applications at high speeds.

In this article, I’ll be shedding some light on protocol buffers and showing you how to use it!

Protobufs are often paired with gRPCs (Remote Procedure Calls), which are a topic of its own. I’ll try to cover it in a few weeks.

The Gist

Protobufs is an interface definition language and communication protocol used to build applications and transport data between them. Protobufs accomplishes this by enforcing a common data structure in the sections of code where data will be transmitted between applications. These data structures are defined in .proto files. A commandline tool, protoc, uses those .proto files to generate class files that are used to write your applications.

These classes come with a few helper functions that can convert data defined in a class to binaries--which then is used to transmit data between two servers.

Protobufs can be compared to JSON, the two differences are:

You need to pre-define how your structure looks like in .proto files
The data stored in protobufs are modified by helper functions provided by the autogenerated classes from those .proto files

Any time you transmit JSON between two servers; you could replace that with a protobuf binary instead. Sending data via protobuf binaries can offer performance improvements in faster download times between 4 to 78% depending on the situation (I discuss more in Tradeoffs and Benefits).

In my mind, there are two processes when developing with protobufs: the development process and the implementation process. The development process deals with creating and managing protobufs. The implementation process is the use of protobuf classes to build our applications/servers/services.

Let's look at these processes by example. Let's say we're developing an application that returns us a list of customers our company has.

Our development process looks like the following:

A developer writes some data structures called CustomerList and Customer in a customerlist.proto file
A command line tool that comes with the protobuf library, called protoc, reads .proto files and generates classes in the programming langauge of the developer's choice.
The developer commits the .proto and generated code into their codebase
If any changes are needed to that datastructure, we start again at step one.

The generated code in our case is the classes CustomerList and Customer. We can now use these classes to build out application.

When the time comes to send data between two systems, we can invoke a helper function that's attached to these classes to convert our Class data into a string. An invoked REST/gRPC/etc call passes this data to another service. Our listener on our other service can then use the same classes to deserialize the string back into language readable data.

Implementing protobufs

Let’s build a system that transports a list of customers from our python application server to a Node.js webserver and shows us that list on a table.

This application is a bit complicated, so I have provided a Github link below for you to follow along:

4shub / protobufs-example

The file structure of our application should look like the following:

// @language-override:Our folder
application_root
|_src
   |_ generated
   |_ protos

First let’s build a customerlist.proto in src/protos:

// @language-override:proto3
syntax = "proto3";

message Customer {
  required string name = 1;
  required int32 id = 2;
  required string email = 3; 
  required bool isNewCustomer = 4;
}

message CustomerList {
  repeated Customer customer = 1;
}

Above I created our data structure following the proto3 language.

Then we need to run following command in our application root:

// @language-override:Terminal
protoc --python_out=src/generated --js_out=import_style=commonjs,binary:src/generated src/protos/customerlist.proto -I src/protos

This command will generate our classes in files named customerlist_pb.py and customerlist_pb.js in a folder called generated.

Now let’s build our python server

# @language-override:Python + Flask
import flask
from generated import customerlist_pb2

app = flask.Flask(__name__)

# creating our "database"
customer1 = customerlist_pb2.Customer(name='Shubham', id=0, email='shub@shub.club')
customer2 = customerlist_pb2.Customer(name='Rui', id=1, email='rui@too.com', isNewCustomer=True)

customer_list = customerlist_pb2.CustomerList()
customer_list.customer.append(customer1)
customer_list.customer.append(customer2)


@app.route('/customer-list')
def get_customer_list():
    # `SerializeToString` is a helper function that serializes customer_list to a binary format
    return customer_list.SerializeToString()

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=3001)

In the code above, I instantiate the class CustomerList and populate it with some customer data. Then I convert that data into a protobuf binary and pass it anyone who requests /customer-list.

Our node server will act as our receiving server, it will host a html page that would contain a button that requests us the customer list stored on the python server. The node.js server will make the request on behalf of the client to get that data.

// @language-override:Node.js + Express
const path = require('path');
const axios = require('axios');
const express = require('express');
const app = express();
const port = 3000;

const { CustomerList } = require('./generated/customerlist_pb');
const PYTHON_SERVER_URL = 'http://localhost:3001';

app.get('/customers', async (req, res) => {
    try {
        const binaryData = await axios.get(`${PYTHON_SERVER_URL}/customer-list`);

        // convert string to base64 to be read by `deserializeBinary`
        const base64data = Buffer.from(binaryData.data).toString('base64')

        const customerList = CustomerList.deserializeBinary(base64data)

        // convert to json
        res.send(customerList.toObject());
    } catch (e) {
        console.log(e)
        res.send(404);
    }
});

app.get('/', (req, res) => res.sendFile(path.join(__dirname, './index.html')));

app.listen(port, () => console.log(`Example app listening at http://localhost:${port}`))

We see CustomerList's helper function deserializeBinary converting our binary string into a workable CustomerList class object. We use toObject to convert our class data into a JSON. We finally pass the JSON to the client.

Tradeoffs and Benefits

Not everything you build requires protobufs!

Sometimes it’s easier and more efficient to not deal with sophisticated methods over sending data. In a study by Auth0 [0], where they compared JSON vs protobuf binary performance, Protobufs significantly improved data transmission rates from java server to java server communication (78% download time reduction), while java server to client communication had only a 4% download time reduction.

Auth0 also did a second test from a java server to the client in an “uncompressed” environment. Download time was improved by 21%. Using this information, if your goal is just to enhance performance, it's much better just to compress your JSON data and forget implementing protobufs.

Outside optimizations, protobufs provides a method of documenting and enforcing a data structure. This is super useful with keeping data consistent across multiple programming languages and multiple teams.

What do tradeoffs and benefits mean for you, the developer? It means that sometimes a tool you could use in one part of your application system might not be useful elsewhere. Or it could mean that maybe the additional development time to enforce protobufs on your whole application is worth it. In the end, it's up to you as a developer to see if a solution is viable for your product or use-case.

Conclusion

Building an application ecosystem can be daunting, but with protobufs in your toolkit you can optimize your networking capacity to its full potential. Companies like Square, Google and Netflix use it every day in their systems. Maybe you can try and build something cool with it too. As always, let me know what you’ve built with protobufs.

[0] https://auth0.com/blog/beating-json-performance-with-protobuf/

Printer Hacks for Fun

Shubham Naik — Wed, 01 Jul 2020 03:52:29 +0000

Earlier today I was playing around with the web usb api.

// Detect dark theme var iframe = document.getElementById('tweet-1278158963255545856-341'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1278158963255545856&theme=dark" }

Ended up making a typewriter-like tool!

// Detect dark theme var iframe = document.getElementById('tweet-1278171656045752320-32'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1278171656045752320&theme=dark" }

If you have an epson printer lying around, the code below will work. Though I've only tested this on Chrome.



<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8">
    <title>Garbage Typewriter</title>
</head>
<body>
    <div>Requires an epson printer to work</div>
    <form id="form-container">
        <br/>
        <label>
            Type in a sentence <br/>
            <input type="text" id="typewriter-input"/>
        </label>
        <br/><br/>
        <div>
            <button type="button" id="find-printer">Find Printers</button>
            <button  type="button" id="connect-printer">Connect Printer</button>
            <button  type="button" id="cut-printer">Cut Page</button>
        </div>
    </form>


    <script>
        // Epson printer specific codes
        const INIT_PRINT = '\x1B' + '\x40';
        const LINE_BREAK = '\x0A';
        const CUT_PAPER = '\x1B' + '\x69';

        // USB devices can have 1-16 possible endpoints that can be connected to
        // we don't know what endpoint we need to connect to to send a successful print command
        // so we just start at endpoint 1, and we shuffle until 16 to see which endpoint the printer is using
        let endpointNumber = 1;

        const getDevices = async () => navigator.usb.getDevices().then(([printer]) => Promise.resolve(printer));

        const sendMessageToPrinter = (device, content) => {
            const encoder = new TextEncoder();

            const data = encoder.encode(content);

            console.log('trying ', endpointNumber)

            return device.transferOut(endpointNumber, data);
        }

        const startPrint = async  (device, content) => sendMessageToPrinter(device, content.join('')).catch(e => {
            if (e.message.includes('The specified endpoint is not part of a claimed and selected alternate interface')) {
                if (endpointNumber < 15) {
                    endpointNumber = endpointNumber + 1;
                    startPrint(device, content);
                    return;
                }

                console.log('failed!');
                return;
            }

            console.error('Send Error:', e);
        }).then(e => console.log(e))


        let device;

        const requestPrinter = () => {
            navigator.usb.requestDevice({
                filters: [],
            });
        }

        const initPrinter = async () => {
            device = await getDevices();

            await device.open();
            await device.selectConfiguration(1);

            device.claimInterface(
                device.configuration.interfaces[0].interfaceNumber
            );

            endpointNumber = 1;
        };

        const cutPrinter = () => {
            startPrint(device, [INIT_PRINT, LINE_BREAK, LINE_BREAK, LINE_BREAK, LINE_BREAK, CUT_PAPER]);
        }

        document.getElementById('form-container').onsubmit = (e) => e.preventDefault();

        let buffer = '';
        document.getElementById('typewriter-input').addEventListener('keydown', (e) => {
            const code = (e.keyCode ? e.keyCode : e.which);

            if  (code === 13) { //Enter keycode
                startPrint(device, [INIT_PRINT, buffer, LINE_BREAK]);
                document.getElementById('form-container').reset();
                setTimeout(() => document.getElementById('typewriter-input').focus(), 100)
                buffer = '';

                return;
            }

            buffer += String.fromCharCode(code);
        });

        document.getElementById('find-printer').onclick = requestPrinter;
        document.getElementById('connect-printer').onclick = initPrinter;
        document.getElementById('cut-printer').onclick = cutPrinter;
    </script>
</body>

</html>

Conditional GETs explained (with Node.js)

Shubham Naik — Wed, 24 Jun 2020 20:24:47 +0000

This post is a mirror of a post I wrote on my own blog. Feel free to check it out here! I release articles on my website 2 weeks earlier than here.

Let's imagine one day you've been poking around the network usage section of your phone--trying to see what apps are killing your allotted 10GB of mobile data.

You scroll down and notice the usual suspects, YouTube, TikTok, whatnot. Then out of the blue, you start to see a bunch of applications that seem out of place: Newspaper apps, stock apps, even some banking apps! These apps can sometimes use more bandwidth than what you think.

How could that be? It turns out that many applications from the New York Times to Robinhood will often re-poll for the latest information from every few minutes to every second. These constant GET requests, while small, can add up.

In this article, I'll be explaining to you a method many of these Apps (hopefully) use to reduce the amount of bandwidth they take up, Conditional GETs. Conditional GETs can help prevent your apps from getting the same 20kb response every time you ping your server.

The gist

Conditional GETs are used in asset caching to prevent a browser from receiving the same javascript/image/CSS payload if a browser cached the latest copy. We should try to use conditional GETs in any request to the server when we poll for cachable content.

Let's look at a typical flow for the conditonal request:

The browser requests some content from a website.
The server returns the content with one or both of these headers:
- Last-Modified: some-date - The time (usually a timestamp) that this content was last modified
- Etag: some-generated-value - A unique id referencing a resource to a particular state in time
  - An ETag could be a hash of the content, an id assigned whenever the content is updated, or a unique string representing the content
The browser requests the same content later time; the browser can pass some conditional request headers:
- If-Modified-Since: some-date - The last timestamp saved on the browser
- If-None-Match: some-generated-value - The previous ETag saved on the browser
The server will check if any of those two values satisfy these conditions:
- If the content is the same, the server will return a 304 status
- If the content is different, the server will return new data with a new Last-Modified and or Etag.

In Practice

In the example below, I am creating a server that allows a user to update and retrieve their user information. The application would allow us to fetch a user's social media information on request.

We use the attribute updatedAt of someUser to validate the "newness" of response and return it as Last-Modified. We will work with ETags later.

Going Deeper

More headers!

The conditional request specification gives us a few different conditional header tags we can work with besides If-None-Match and If-Modified-Since. Those are listed below:

If-Match: If the ETag on the server matches the ETag passed in, the server should send us new data.
If-Unmodified-Since: If the timestamp on the server is older than the timestamp we pass in, the server should send us new data.
If-Range: If an ETag or timestamp on a server matches a range of timestamps of ETags we pass in, the server should send us new data.

Strong and Weak Validation

The ETag HTML specification provides us two methodologies we can implement for validating our Etags:

Strong validation must ensure that the content requested is byte-by-byte the same as the previously requested content for a client to receive a 304 response. An example could be a dataset containing all your banking information. If anything has changed on the server, we should always send the most recent data.

Weak validation means that the server's content could be different from what already is on the client, but the change is not significant enough for the server to pass back new data. Let's go back to that banking information example. Let's say the banking information also contains some metadata information on an A/B test going on. This information is not essential and probably doesn't need to be updated on the client if we are performing live updates on the browser.

To ask a server to perform weak validation, you would prepend your Etag with W/.

Let's build a server that can perform both strong and weak Etag validation.

const express = require('express');
const md5 = require('md5');

const server = express();
const port = 3000;

const article = {
  content: 'Hello there! this is an article there!',
  meta: 'Meta content for user',
  adInfo: '349243'
}

// gets an article from "our database"
const getArticle = () => Promise.resolve(article);

const generateETag = (article) => {
  const contentHash = md5(article.content);
  const metaHash = md5(article.meta + article.adInfo);

  return `${contentHash}_${metaHash}`;
}

const validateETag = (etag, article) => {
  const useWeakValidation = etag.includes('W/');
  const parsedTag = etag.replace('W/', '');

  if (useWeakValidation) {
    const weakCompare = md5(article.content);

    return weakCompare === parsedTag.split('_')[0];
  }

  const strongCompare = generateETag(article);

  return strongCompare === parsedTag;
}

server.get('/article', async (req, res) => {
  const etag = req.headers['if-none-match'];

  const article = await getArticle();

  if (!etag || validateETag(etag, article)) {
    res.sendStatus(304);
    return;
  }

  const nextEtag = generateETag(article);
  res.setHeader('ETag', nextEtag);
  res.send({ article });
})

server.listen(port, () => console.log(`App listening at 
    http://localhost:${port}`));

Above, we created a function called generateTag that creates an ETag composed of two parts, a contentHash and metaHash. The contentHash is an md5 hash of only the article's content. The metaHash is an md5 hash of all the non-content parts of this article.

We also created a validation function that will:

If weak validation is requested: we return a new payload if the article's content's md5 hash is different than what is on the server. We will send a 304 if any other data has changed.
If strong validation: we will return content if anything in our article object has changed, we check both parts of the md5 hash group.

Weak validation is a little more complicated to implement then just checking if any byte has changed. Still, the benefit of building weak validation can help reduce unnecessary GETs when doing repetitive polls.

Conclusion

Conditional GETs are a straightforward way to reduce the bandwidth handled through your application. The bandwidth savings can directly reduce your networking costs and also help your customers reduce their networking costs (if they pay for their bandwidth).

Try this solution out alongside client-side caching, and you can have even more savings as users who return to your website or app don't need to redownload content that hasn't changed since their last visit. Anyway, give it a go--let me know what you make!

Fun with details

Shubham Naik — Fri, 12 Jun 2020 23:22:06 +0000

Happy Friday! Here's a little article to end your week. It is a mirror of an article on my site. Feel free to visit it if you want to see live examples of what I show here!

Let's learn about `<details />`

<details /> is a very spiffy, yet relatively unadopted component in the HTML catalog. It's not super dynamic, but pretty useful if you ever want to build accordions like below:

Hacking with Details

The cool thing about <details /> is that it can act as a state machine in pure HTML. This means when we add some useful CSS selectors, we can make the <details /> element act like a full-fledged toggle button.

Let's look at this example below:

As you can see above, I just created a "button" entirely in CSS that increases the font-size of .open-me-target on click of the summary.

Take note of the sibling combinator ~ and the attribute selector https://developer.mozilla.org/en-US/docs/Learn/CSS/Building_blocks/Selectors/Attribute_selectors. [open] tell us to look at instances when the class .open-me has the attribute open, and (~) tells us to look for instances when .open-me-target follows .open-me within the same parent element. This lets us only apply the blue, 50px style to instances that meet both of these criteria.

Polishing it up

While functional, the UI above looks kind of janky. Let's make it look a little better.

Pretty cool I think.

Notes

Other HTML elements can be used to create a toggle-like state, the most significant being <input type="checkbox" />. I focused on <details /> because keyboard users can toggle using the return — which I find more natural — while checkboxes are toggled with the spacebar.

I probably wouldn't use this hack in production. Changing HTML components' core functionality, even if done carefully, hampers accessibility, as not all browsers can understand the wackiness we just wrote down.

Building with server-sent events with React and Node.js

Shubham Naik — Tue, 09 Jun 2020 04:17:03 +0000

This post is a mirror of a post I wrote on my own blog. If you would like python or native javascript examples of the code I presented below, feel free to check it out here

Building real-time applications on the web has never been easier. In this post, I'll explain how you can use server-sent events, or SSE for short, to get real-time data for your web applications.

At the end of this article you should know:

What a server-sent event is
How to listen to server-sent events on the browser
How to send server-sent events from your server

This tutorial is for those who have some familiarity with developing on the web as well as some knowledge in either python or nodejs.

The gist

Server-sent events (SSE) are a client initiated, unidirectional, server controlled messages. When you visit a website that queries an SSE-enabled endpoint, the server can send your browser unlimited amounts of information until you leave that page. SSE urls are always accessed via an asynchronous request from your browser. You can visit a url that serves an SSE endpoint from your browser but there is no standard on what you will experience.

const source = new EventSource('/an-endpoint');

source.onmessage = function logEvents(event) {
   console.log(JSON.parse(data));
}

In this code snippet, I create a new EventSource object that listens on the url /an-endpoint. EventSource is a helper class that does the heavy lifting of listening to server-sent-events for us. All we need to do now is to attach a function, in this case logEvents, to the onmessage handler.

Any time our server sends us a message, source.onmessage will be fired.

Let's look at a more realistic example. The code below listens on a server at the url https://ds.shub.dev/e/temperatures. Every 5 seconds, the server returns a server-sent event with the temperature of my living room.


// @codepen-link:https://codepen.io/4shub/pen/QWjorRp
import React, { useState, useEffect } from 'react';
import { render } from "react-dom";

const useEventSource = (url) => {
    const [data, updateData] = useState(null);

    useEffect(() => {
        const source = new EventSource(url);

        source.onmessage = function logEvents(event) {      
            updateData(JSON.parse(event.data));     
        }
    }, [])

    return data;
}

function App() {
  const data = useEventSource('https://ds.shub.dev/e/temperatures');
  if (!data) {
    return <div />;
  }

  return <div>The current temperature in my living room is {data.temperature} as of {data.updatedAt}</div>;
}

render(<App />, document.getElementById("root"));

What's happening behind the scenes?

Let's look at these two properties of EventSource:

url - The url that we want to listen on for changes
readyState - The state of the connection. This can be (0) CONNECTING, (1) OPEN and (2) CLOSED. Initially this value is CONNECTING.

When EventSource is invoked, the browser creates a request with the header Accept: text/event-stream to the url that was passed through.

The browser will then verify if the request returns a 200 OK response and a header containing Content-Type: text/event-stream. If successful, our readyState will be set to OPEN and trigger the method onopen.

The data from that response will then be parsed and an event will be fired that triggers onmessage.

Finally, the server we pinged can send us an unlimited amount of event-stream content until:

We close our page
We fire the close() method on event source
The server sends us an invalid response

When we finally close our connection, the EventSource object's readyState will fire a task that sets readyState to CLOSED and trigger the onclose event.

In case of a network interruption, the browser will try to reconnect until the effort is deemed "futile," as determined by the browser (unfortunately, there are no standards on what constitutes "futile").

Sending events on the server

Sending server-sent events is just as easy as listening to them. Below, I've written a few different implementations of sending server-sent events to your client.

// @repl-it-link:https://repl.it/@4shub/server-sent-events-node
const express = require('express');

const server = express();
const port = 3000;

// create helper middleware so we can reuse server-sent events
const useServerSentEventsMiddleware = (req, res, next) => {
    res.setHeader('Content-Type', 'text/event-stream');
    res.setHeader('Cache-Control', 'no-cache');

    // only if you want anyone to access this endpoint
    res.setHeader('Access-Control-Allow-Origin', '*');

    res.flushHeaders();

    const sendEventStreamData = (data) => {
        const sseFormattedResponse = `data: ${JSON.stringify(data)}\n\n`;
        res.write(sseFormattedResponse);
    }

    // we are attaching sendEventStreamData to res, so we can use it later
    Object.assign(res, {
        sendEventStreamData
    });

    next();
}

const streamRandomNumbers = (req, res) => {
    // We are sending anyone who connects to /stream-random-numbers
    // a random number that's encapsulated in an object
    let interval = setInterval(function generateAndSendRandomNumber(){
        const data = {
            value: Math.random(),
        };

        res.sendEventStreamData(data);
    }, 1000);

    // close
    res.on('close', () => {
        clearInterval(interval);
        res.end();
    });
}

server.get('/stream-random-numbers', useServerSentEventsMiddleware, 
    streamRandomNumbers)


server.listen(port, () => console.log(`Example app listening at 
    http://localhost:${port}`));

In the example above, I created a server with an event-stream that sends users a random number every second.

Conclusion

Many companies use server-sent events to pipe data to their users in real time. LinkedIn uses server sent events for their messaging service, Mapbox uses SSE to display live map data, and many analytics tools use SSE to show real-time user reports. SSE will only become more prominent as monitoring tools and real-time events become more relevant to users.

Let me know if you try it out — I'd love to see what you come up with!

DEV Community: Shubham Naik

Let's learn about protocol buffers

Let's learn about Protocol Buffers

The Gist

Implementing protobufs

4shub / protobufs-example

Tradeoffs and Benefits

Conclusion

Printer Hacks for Fun

Conditional GETs explained (with Node.js)

The gist

In Practice

Going Deeper

More headers!

Strong and Weak Validation

Conclusion

Fun with details

Let's learn about <details />

Hacking with Details

Polishing it up

Notes

Building with server-sent events with React and Node.js

The gist

What's happening behind the scenes?

Sending events on the server

Conclusion

Let's learn about `<details />`