DEV Community: Denis Anisimov

Lighter font weight in code editors - yay or nay?

Denis Anisimov — Tue, 29 Sep 2020 13:52:34 +0000

Many code-oriented monospaced fonts looks 'bold' by default compared to your regular fonts. I've recently switched to Cascadia Mono SemiLight and kind of dig it, it feels easier to pick up different code tokens than with the regular weight.

Anyone else is using lighter versions of this or other fonts?

Share a thing you've been working on lately

Denis Anisimov — Sun, 23 Feb 2020 19:19:07 +0000

What's something you've been building, researching or tinkering with lately?

It doesn't have to be prime-time ready, just share it here to get some encouragement from the community🤗

Hiring without interviews - is it possible?

Denis Anisimov — Thu, 20 Feb 2020 23:14:51 +0000

Hatred towards an interviewing process in the tech field is a common topic, but if you were to design the hiring process from scratch how would you do it?

Should we keep doing the standardized repeatable whiteboard-style interviews?

Or should it be a take-home assignment? Paid or not?

Or maybe we should move to the model of already vetted candidates where you just pick and hire?

I'm interested in the perspective from both sides - the companies and the candidates.

Building Image CDN with Firebase

Denis Anisimov — Sun, 10 Nov 2019 23:58:26 +0000

Delivering a perfect image on the web is not that easy of a task as it seems. Long gone the days where you can just put 'image.jpg' in your '/var/www/dist' and call it a day. You want your website to load images quickly, in the right format and with a perfect size based on the user's device. And you want that without blowing up your cloud storage bills and ideally without tedious work of resizing and tweaking your images manually.

Luckily for us Image CDNs exist. These are specialized types of Content Delivery Networks (CDN) that can take a source image, perform the conversions and transformations on the fly and cache the result to be delivered globally in a matter of milliseconds. Some examples are Cloudinary and imgix.

In this post we will build our own image CDN - efficient, fast, customizable, living on our own domain, and free using Firebase.

What we are going to build

In a nutshell we want to:

Upload an image image.jpg to a storage bucket.
Use a specially formatted URL to retrieve that image in the right size, e.g /cdn/image/width=100,height=50/image.jpg.
That URL should work fast.
We want to get the image in the webp format if our browser supports that.

Overall architecture

Btw, this image is served from Cloudinary Image CDN

Some important details about the proposed design:

Firebase Hosting

The Firebase Hosting part is where the automatic caching magic happens. By design Hosting caches all the responses from Functions that have Cache-control HTTP header set to public with an appropriate expiration time. That means that subsequent requests to the same image size, even from different browsers, will be served super fast and without firing another Function execution which saves us money.

Note that Firebase Hosting rewrite rules allow us to co-host our image CDN on the same domain as our main website.

Firebase Function

The Firebase Function contains the image processing logic. It is also responsible for setting the appropriate response HTTP headers for Firebase Hosting to cache processed images correctly.

Firebase Storage

The main storage for original images is Firebase Storage and we identify images by their key in the bucket. Using Firebase Storage is good for the speed of access and works on the free tier since we aren't making outbound requests from our functions.

Browser

Modern browsers that support webp format send a special Accept HTTP header with a value image/webp (at least Firefox and Chrome do that). We will use that to automatically detect the support and convert the image.

Note that since the desired image response depends on the HTTP header we also need to include it in the Function's response Vary header. This ensures that Firebase Hosting cache won't result in serving webp images to the browsers that don't support it.

The image transformation function

For image processing we are going to use sharp JavaScript library. This is super handy for us as we don't need to spawn any additional processes. Also sharp is faster than ImageMagic, so win-win.

You can find the full source code here. But without a single listing that wouldn't be a serious DEV post, right? So here is the snippet of the function implementation:

// Full code is here: https://github.com/dbanisimov/firebase-image-cdn

// Run the image transformation on Http requests.
// To modify memory and CPU allowance use .runWith({...}) method
export const imageTransform = functions.https.onRequest((request, response) => {
  let sourceUrl;
  let options;
  try {
    const [optionsStr, sourceUrlStr] = tokenizeUrl(request.url);
    sourceUrl = new URL(sourceUrlStr);
    options = parseOptions(optionsStr);
  } catch (error) {
    response.status(400).send();
    return;
  }

  // Modern browsers that support WebP format will send an appropriate Accept header
  const acceptHeader = request.header('Accept');
  const webpAccepted =
    !!acceptHeader && acceptHeader.indexOf('image/webp') !== -1;

  // If one of the dimensions is undefined the automatic sizing
  // preserving the aspect ratio will be applied
  const transform = sharp()
    .resize(
      options.width ? Number(options.width) : undefined,
      options.height ? Number(options.height) : undefined,
      {
        fit: 'cover'
      }
    )
    .webp({ force: webpAccepted, lossless: !!options.lossless });

  // Set cache control headers. This lets Firebase Hosting CDN to cache
  // the converted image and serve it from cache on subsequent requests.
  // We need to Vary on Accept header to correctly handle WebP support detection.
  const responsePipe = response
    .set('Cache-Control', `public, max-age=${cacheMaxAge}`)
    .set('Vary', 'Accept');

  // The built-in node https works here
  https.get(sourceUrl, res => res.pipe(transform).pipe(responsePipe));
});

And also firebase.json rewrites section, which transparently forwards requests to the Function:

{
...
  "rewrites": [
    {
      "source": "/cdn/image/**",
      "function": "imageTransform"
    },
    {
      "source": "**",
      "destination": "/index.html"
    }
  ]
...
}

The end result

You can see the live demo here: https://fir-image-cdn.web.app/.

The cat's photo is resized on-the-fly and cached, you can check that by opening it again in a private browsing mode - the result will be noticeably fast. Another way to verify that we are seeing content served by the CDN cache is to check for the x-cache: HIT header in the response.

How fast is it?

Cached responses - low 10ms for small images which should stay pretty consistent globally.

Cache miss - 500ms+ for small images, up to 1s for medium images and seconds (!) for large images. Whom to blame here is an open question - quick experiment with changing the Firebase Function type to a faster one hasn't shown noticeable difference, so it may not be due to the compute power, but rather Functions outbound networking or CDN cache fill performance.

Caveats

Cold starts. The first Function execution takes significantly longer than subsequent executions, which may affect the user's experience. One way to avoid that is to use Cloud Run containers, which allow concurrent requests to the same instance. The great thing is that Cloud Run is supported by Firebase Hosting as the request forwarding destination.

Low cache hit ratio. In our approach we vary the cached response based on the Accept header in order to correctly support webp detection. Every browser may send different value for that header which may result in very low cache hit ratio. We may avoid that by disabling the automatic detection and request webp version explicitly. Modern browsers allow to do that with a combination of <picture> and <source type="image/webp"> HTML elements.

How much does it cost?

Great news everyone! The combo described above works perfectly under the Firebase free Spark plan. You roughly get 5GB of stored source images and 100K image transformations per month.

It also seems like the cached responses served by CDN count towards the total Hosting download allowance of 10GB per month.

Next steps

If you want to take the self-hosted Image CDN idea to the next level then take a look at Thumbor. It has a ton of features and could be easily run inside of a Cloud Run container.

You may get better overall performance (and cost of operations) by manually combining Cloud CDN, Load Balancer, multi-regional Cloud Storage and Cloud Functions deployed in multiple regions. But should you?

What else to read

Stay sharp!

Implementing manual OAuth sign-in flow for Facebook and Google

Denis Anisimov — Sat, 09 Nov 2019 19:53:16 +0000

Our web app uses Facebook and Google social logins. We've implemented those using the recommended approach with JS SDKs.

That was easy to do and works fine most of the time, but I've started to see many users facing issues with that when they access the web app using a private browsing mode or have other privacy-related settings turned on. Namely some browsers block the Facebook JS SDKs, and many block third-party cookies which breaks the default Google sign-in.

Given that I've started thinking about implementing the OAuth flow manually completely skipping the JS SDKs. Seems like this will give the most robust results, it is better for user's privacy and can save some bandwidth on the initial page load.

Have anyone followed the same route? Are there any concerns with this approach, e.g. about the security of SDK way vs the manual way?

Firestore Dataflow Illustrated and Measured

Denis Anisimov — Mon, 25 Mar 2019 03:55:35 +0000

Firestore is more sophisticated than just a managed DBaaS - it comes with a real-time updates support, client caching, offline persistence and background triggered cloud functions. All of these pieces can be used to create very dynamic and scalable apps with complex dataflows without the headache of managing the infrastructure.

But how all these pieces fit together and how can they be used to achieve different user experiences? In this blog post I'm presenting my view of a typical app data flow and perform a little testing.

No time to read? Play with the interactive version here: Firestore Dataflow Test

A typical Firestore app dataflow

For this part I'm assuming that we are building an app (web or mobile) that uses Firestore directly through official SDKs with local caching enabled. Our app may also want to perform some background data processing of Firestore documents and write the results back to the Firestore to be displayed on the client. All communication is realtime, which means that we use snapshot listeners on the client and Firestore triggers for functions.

An example may be a chat app where we fetch link previews in the background.

Take a look at this diagram and read below for the description

TC ~15ms
From client write to client read from the cache

The shortest dataflow path is through the local cache. The data written by the client is saved in the cache and immediately triggers a snapshot listener.

You should care about this data path if you rely on the local cache to provide optimistic UI updates.

TDb ~200ms
From client write to client read from the database

The default path is from the client to the database backend. All writes eventually go there producing the final document result that is streamed back to the client and all other clients listening to affected queries. This path performance is reliant on the quality of a client network connection.

This is the data path of client-to-client realtime communication. It's also the one for transactions, as those needs to be committed to the backend.

TTr ~400ms
From firestore write to firestore update with the triggered function

Once the data is written to Firestore a background function is triggered. This path is heavily affected by the cold start time and also by the regional locations of your functions and Firestore.

This path is important if you're making data changes in response to document creation or update.

TU ~600ms
From client write to client read of the data updated with the triggered function

Finally your updated data needs to reach back the client. Technically it is the same snapshot listening to document updates. For the user experience it is the longest path.

Despite the path length it may be a good choice to do background data processing.

A special case with a callable function

Sometimes talking directly to Firestore is not an option - the operation is too computationally heavy, or it needs to access some sensitive data, or requires complex validation that cannot be done with security rules. A common approach is to move the logic from the client to a callable function. The obvious downside of this case is that you loose the offline capabilities and caching and need to implement optimistic UI separately.

Let's look at this special case

TCall ~200ms
From client request to response received

The direct path is a callable cloud function request. The response may be anything, but it's a good idea to include the document being written to the database so it can be used for optimistic update on the client. This path is heavily affected by the cold start time and network latencies.

TDb ~250ms
From client request to client read from the database

The indirect path through Firestore. The data written to Firestore by the function can be read back at the client and other client listening to affected queries. This is the simplest way to get the result back as you don't need to process the callable function response separately.

Measuring the paths lengths

The times above are the results of measurements I've made and present here to show the relative lengths of different paths. They are on the optimistic side of the spectrum, especially if you factor in cold start times and proximity to Google Cloud datacenters. All times are for round-trip data paths.

I've written a simple web-based testbed to get those numbers:

Github dbanisimov / firestore-dataflow-test

I encourage you to clone the code and deploy your own project to see the performance in your case. Your geographical location and regions of Firestore and Cloud Functions will change the numbers significantly. It also shows very vividly cold start delays, time to establish connection to Firestore for the first time, and some other artifacts. It uses HTM + Preact (look ma, no build!) for the web client.

Feel free to play around with the live version of the testbed:

Firestore Dataflow Test

Both Firestore and Functions are in us-central.

Conclusion

Of course, the two cases above don't cover all the possible combinations of data flow primitives. More complex systems can be built with inclusion of PubSub and Cloud Storage, as both of them are well integrated with Cloud Functions. And drawing some high-level pictures and running experiments before building the whole system may save from some surprises later.

Happy flowing! :)

Automatic Firestore Backups

Denis Anisimov — Sat, 09 Mar 2019 21:55:30 +0000

Cloud Firestore has an export/import functionality using CLI and REST APIs that allows you to make simple backups to a Cloud Storage bucket. It basically looks like this:

gcloud beta firestore export gs://BUCKET_NAME --project PROJECT_ID

If you are as clunky as me and prone to messing up with database you'd probably want to have it backed up somewhat often. But unfortunately, even with Firestore in GA, there is no managed automatic backups option yet. Here is what my solution looked like for a while:

Not very DRY, is it? Here is a better way using only Google Cloud Platform services and a tiny bit of coding.

Overall idea

Cloud Scheduler issues a message to a Cloud PubSub topic periodically based on a cron schedule.
Cloud Function is triggered by the message and executes a call to the Firestore REST API
Firestore export API starts a long-running operation that saves a backup to a specified bucket
Cloud Storage stores backups organized by timestamps

Prerequisites

This post assumes that you have your Firebase project set up, have gcloud command-line tool installed and know your way around Cloud Console.

Cloud Scheduler

Cloud Scheduler is a very simple service by GCP, but nevertheless very useful and much awaited by the server-less crowds. With it it's finally possible to define simple cron-like jobs that can trigger HTTP functions or emit a PubSub message. For use with Cloud Functions a PubSub way is preferred, as HTTP jobs don't have any authentication.

The following command creates a PubSub job scheduled at midnight everyday. The message body is not used in our case, so can be arbitrary.

gcloud scheduler jobs create         \
    pubsub firestore-backup          \
    --schedule "0 0 * * *"           \
    --topic "firestore-backup-daily" \
    --message-body "scheduled"

Once you execute this command your job will run at the specified time.

Cloud Function

The cloud function is triggered on PubSub message and makes a request to Firestore REST API. To make a request with proper authentication we need to have an OAuth access token. Luckily for us Cloud Functions have a default service account that can be easily used to generate a token.

import * as functions from 'firebase-functions';
import * as firebase from 'firebase-admin';
import * as request from 'request-promise';

firebase.initializeApp();

export const backupOnPubSub = functions.pubsub.topic('firestore-backup-daily').onPublish(async () => {
  // Get bucket name from cloud functions config
  // Should be in the format 'gs://BUCKET_NAME'
  //
  // Can be set with the following command:
  //   firebase functions:config:set backup.bucket="gs://BUCKET_NAME" 
  const bucket = functions.config().backup.bucket;

  // Firebase/GCP project ID is available as an env variable
  const projectId = process.env.GCLOUD_PROJECT;

  console.info(`Exporting firestore database in project ${projectId} to bucket ${bucket}`);

  // Use default service account to request OAuth access token to authenticate with REST API
  // Default service account must have an appropriate role assigned or
  // the request authentication will fail
  const { access_token: accessToken } = await firebase.credential.applicationDefault().getAccessToken();

  const uri = `https://firestore.googleapis.com/v1/projects/${projectId}/databases/(default):exportDocuments`;
  const result = await request({
    method: 'POST',
    uri,
    auth: { bearer: accessToken },
    body: {
      outputUriPrefix: bucket,
    },
    json: true,
  });

  // The returned operation name can be used to track the result of the long-running operation
  //   gcloud beta firestore operations describe "OPERATION_NAME"
  const { name } = result;
  console.info(`Export operation started ${name}`);
});

Don't forget to set the config and deploy

firebase functions:config:set backup.bucket="gs://BUCKET_NAME"
firebase deploy --only functions:backupOnPubSub

Firestore REST API permissions

Import/Export Rest API requires appropriate OAuth scopes to be enabled. You can achieve that by adding Cloud Datastore Import Export Admin role to the default service account in your project.

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member serviceAccount:PROJECT_ID@appspot.gserviceaccount.com \
    --role roles/datastore.importExportAdmin

Cloud Storage

In case you don't have a bucket yet you should create one. I've found that Firestore import/export requires at least regional storage class, coldline or nearline won't work.

gsutil mb gs://BUCKET_NAME/

Testing everything together

Once everything is set up you can trigger the whole process from the Cloud Scheduler console by clicking "Run now" button

Check logs in the Firebase Console or Stackdriver

Check export operation status

gcloud beta firestore operations describe "projects/PROJECT_ID/databases/(default)/operations/ASA3NDEwOTg0NjExChp0bHVhZmVkBxJsYXJ0bmVjc3Utc2Jvai1uaW1kYRQKLRI"
done: true
metadata:
  '@type': type.googleapis.com/google.firestore.admin.v1.ExportDocumentsMetadata
  endTime: '2019-03-09T21:04:39.263534Z'
  operationState: SUCCESSFUL
  outputUriPrefix: gs://PROJECT_ID.appspot.com/2019-03-09T21:04:32_85602
  progressBytes:
    completedWork: '5360'
    estimatedWork: '4160'
  progressDocuments:
    completedWork: '40'
    estimatedWork: '40'
  startTime: '2019-03-09T21:04:32.862729Z'

And finally check the backup in the bucket

Next steps

Automatic backup is only useful if it's working reliably. It is a good idea to have monitoring and alerting for your backup operations.

One way to do this is to save the last export operation to Firestore and schedule a job some time after to check the result of the long-running operation. If the results is not successful it can send an email or log an error that will trigger an alert through Stackdriver Error Reporting.

Another improvement is enabling of automatic deletion of old backups after a certain time. This could be achieved with lifecycle management.

Closing remarks

Would love to hear you thoughts on this approach and how it could be improved.

Happy coding! Make backups :)

DEV Community: Denis Anisimov

Lighter font weight in code editors - yay or nay?

Share a thing you've been working on lately

Hiring without interviews - is it possible?

Building Image CDN with Firebase

What we are going to build

Overall architecture

Firebase Hosting

Firebase Function

Firebase Storage

Browser

The image transformation function

The end result

How fast is it?

Caveats

How much does it cost?

Next steps

What else to read

Implementing manual OAuth sign-in flow for Facebook and Google

Firestore Dataflow Illustrated and Measured

A typical Firestore app dataflow

TC ~15ms From client write to client read from the cache

TDb ~200ms From client write to client read from the database

TTr ~400ms From firestore write to firestore update with the triggered function

TU ~600ms From client write to client read of the data updated with the triggered function

A special case with a callable function

TCall ~200ms From client request to response received

TDb ~250ms From client request to client read from the database

Measuring the paths lengths

Conclusion

Automatic Firestore Backups

Overall idea

Prerequisites

Cloud Scheduler

Cloud Function

Firestore REST API permissions

Cloud Storage

Testing everything together

Next steps

Closing remarks

TC ~15ms
From client write to client read from the cache

TDb ~200ms
From client write to client read from the database

TTr ~400ms
From firestore write to firestore update with the triggered function

TU ~600ms
From client write to client read of the data updated with the triggered function

TCall ~200ms
From client request to response received

TDb ~250ms
From client request to client read from the database