DEV Community: Arif Amirani

A JSON Based Serverless Quasi-Static Platform

Arif Amirani — Tue, 10 Aug 2021 13:16:28 +0000

Introduction

I've been working with large NGOs to architect their multi-faceted systems. These systems are responsible for information dissemination, data collection, analysis, and sources & sinks to other systems. Our near term goal was to build an information platform (IP). The MDP was narrowed down to the following feature set.

Target persona

The IP was intended for 3 main personas.

Villagers
- Direct consumers
- Low tech capability
- Language barriers
- Varied devices - sizes and capabilities
- Irregular bandwidth
Ops team
- Responsible for training using the content
- Submitting feedback for content from users
- Updates to content (limited)
Content team
- Primarily responsible for content
- Regular content updates

Requirements

Media support

The portal has to support different media types which includes but not limited to; video, digital books, images, audio (podcast). The content is generated by a marketing and education team which is then uploaded to a public repository for downloads. All content is in the public domain.

Multilingual and region support

The content itself is versatile. The information and instructions change based on the local language, diet, and availability of resources. The portal has to support reuse of content as well as specific content for a particular region. Ease of management of the portal data by the content team was paramount.

Interval based updates

The content team updates the data several times a day. However there was no need to do real-time updates. New content can show up within the hour.

Analytics

Measurement is core to any successful deployment, especially for large and diverse ones. We factored in the need of granular measurement for clicks, bounces, playback, skips right from day one.

3rd Party API

The content for the portal has to be made available to 3rd party applications for their internal consumption. The interesting part here is that the content over the API is the same as the website, however the 3rd party applications must be throttled to avoid proxy or overuse of our endpoint.

Final Architecture

Apart from the requirements above, we also were tasked to ensure cost-effectiveness along with speed of delivery. The obvious choice was a typical three tier architecture that would achieve most of the objectives. The team also had the right experience. However I decided to go a different route. In the recent past I had deployed a JSON based architecture that had scaled well with up to 20 million visitors but nowhere close to the complexity.

I made a few changes and architected the solution loosely on CQRS - Command Query Responsibility Segregation pattern. All the data that was needed to be displayed (read path) were based on JSON files that were continously refreshed via a fleet of Lambda functions. On the other hand the write path were served by API Gateway (HTTP).

Origin Database

These are the primary sources of information. They can be a Google Sheet, Airtable, RDBMS data sources. They provide the actual content metadata and rules of transformation.

Pull Lambda Fleet

PLF run either on a reactive or schedule basis. They fetch data from the data sources and merge them based on rules. The output is split based on language, region, content, and use case. They generate JSON files at the end of the process and upload them to an S3 bucket using a convention that the webapp can understand.

Information Portal (IP)

Built on React and Tailwind CSS, the IP delivers the content to users. It is a lightweight, responsive PWA. Can work with limited bandwidth and works across all device display sizes. Once it loads, it pulls the appropriate JSON from the server via CloudFront, depending on the region and language setting. All UI actions such as filtering, and search are done on the client side. The size of each JSON is manageable and uses aggressive compression to deliver data quickly. The auto reload mechanism in React (ReactSWR), ensures that the client refetches the JSON every 15 mins or on reload of page.

API Clients

For the content 3rd party API, we use the main API Gateway (APIGW). Using the APIGW method, we can connect directly the GET method to an S3 resource. This requires no glue code or handling. It is seamlessly handled by APIGW.

Using API Keys & Usage Plans, we ensured only authenticated clients got access to data and also rate limited their API calls.

Analytics & Read

All of the write paths use the main API Gateway (APIGW) and Lambda functions to write to DynamoDB. We capture all granular events such as playback, download, visit, and batch them up to the server. Partial data loss is ok with the volume we expected.

We created an internal portal to create a feedback loop of usage which was based on the data in DynamoDB. The web app used the HTTP API endpoints within the same APIGW to fetch data from DynamoDB.

Aggregator Lambda

To monitor and analyse the content changes, we deployed a cron based lambda functions that pulled the current JSON in the data bucket and created a snapshot of it. The snapshots were aggregated over a time interval and uploaded to reporting S3 bucket. A webapp fetched the latest aggregate data and charted it for the admin to review.

Results and Conclusion

The entire setup took about 1.5 months to build. Once deployed, we never breached the free tier of many services for processing. Our biggest cost center was data transfer. We had also enabled all edge nodes for Cloudfront which cost us a bit more. The latency issues were non existent. Our DR was pretty self solved due to S3 becoming our primary data store and DynamoDB was highly available but not in the critical path.

Final notes

We had a specific use case that was served quite well with our design choices. This may not be the most optimal architecture for more demanding and real-time applications.

How I passed the AWS Certified Solutions Architect Professional 2021 Exam - SAP-C01

Arif Amirani — Mon, 09 Aug 2021 10:39:01 +0000

Introduction

I recently passed my AWS Solutions Architect Professional Certification on the first attempt. I wanted to share my experience of preparing for the exam as well as the actual exam day tips and tricks I learnt. I'll also talk about what to expect and approaches to navigate the challenges.

Getting an AWS certification is important for your cloud ecosystem career. Any role including DevOps, Data Engineer, Software Engineer, Architect would immensely benefit from this certification. This certificate demonstrates your ability to analyse and evaluate architectures.

The certificate

Credly Page

Background

I've been a full stack developer for 15 years. Worked in small and large organizations across domains. I started my cloud journey with AWS, working on the very basic services such as EC2, VPC, ELB, R53, S3; 8 years ago. My primary focus has always been on AWS with 20% time spent on evaluating and working with cloud vendors like Google Cloud Platform (GCP). I've built large scale, distributed, real-time systems in various domains and seen them through growth and chaos.

I passed my AWS DevOps certification a few years ago. I am an AWS Community Builder, which is an amazing platform for AWS enthusiasts.

AWS Community Builders

If you want to learn about AWS and contribute with brilliant minds from across the globe, you should definitely signup at AWS Community Builders. With Jason Dunn steering the effort it has become engaging and vibrant.

What does the exam test you on

The AWS SA Pro is one of the toughest and sought after certifications in the cloud ecosystem. The primary reason is, the certification not only tests your knowledge but more importantly your experience.

AWS tries very hard to ensure that candidates DO NOT clear this certification by sheer repetition or memorization. Unless you can demonstrate experience and the ability to evaluate situations this is a hard exam to pass.

Does that I mean I cannot pass without experience?

Of course you can, but according to me, cramming for a professional certification and passing it is a waste of your time. You're better off studying the associate certification. Follow that up with real world experience to get to the professional level. You and your peers will appreciate it more.

Understanding the exams

To clear the exam, it is important to understand the mechanics. They will help you be aware of what AWS expects out of each question. AWS goes into painstaking details to make the exam fair, clear and reliable. Their focus is to test your knowledge and experience at every step.

My notes on the research:

Questions

There are no trick questions. No one is trying to confuse you
Several options (keys) are distractors. Distractors are answers which may be wrong but seem perfectly plausible
Very rarely will previous questions/answers aid your next ones. Stop wasting time to find clues
You get about 2.5 minutes per question. Each question is verbose and takes time to read. Learn to skim over (more on this later)
Many answers are right, you have to pick the most appropriate one given the requirement
At the Professional level, the questions require you to Analyze and Evaluate. This means you need to understand the scenario and pick one based on several factors. Even the most obvious and straightforward answer will be wrong because it is not cost-effective
Questions will only test you on one area at a time even with multiple systems
Do not make assumptions about the scenario. If it isn't written, then you have to ignore that factor
Questions will not test you on UI, numbers, math, etc. You do however need to know the limits of services to select the right answer. For e.g. API Gateway has a timeout of 29 secs and an answer that uses APIGW but runs for 35 secs will be wrong
Scores are scaled. Each question does not have the same weight. This will impact your total score.
There is no penalty for wrong answers

My Preparation

I spent over two weeks of rigorous study time prior to the exam. I continously work on projects that require me to interact with AWS (most common services), this helped. Some of the questions are recall level, which means they test your ability to remember services and their characteristics, and continously working with AWS will help you answer them.

Give yourself at least 1 month to prepare even if you are an AWS expert

Focus

Like any certification exam, passing the SA Pro, requires a bit of everything:

Knowledge - You absolutely must be well versed with AWS and all the services it has to offer
Experience - You must know how to solve problems using a combination of AWS services and have solved at least a few in the past
Scope - Understand what the certification will test you on, before you start studying. Go to at least more than one source to get latest information on what to cover
Skill - The questions are long, designed to test your focus and need time to read and comprehend. Picking the right approach is key to solving all questions. Come up with your own strategy using the practice exams
Time Management - Learn to manage your time. I cannot stress enough how easy it is to run out of time. This happens primarily because we cannot let go of a question

There are many courses and prep tools out there for the professional exam. I did a cursory search on the internet and asked experts around for the recommended ones. You should do your own research and pick a few.

The two I ended up using were:

AWS Certified Solutions Architect Professional Practice Exams 2021 by Tutorials Dojo

These exams are the holy grail of practice. They are continously updated and Jon Bonso (a fellow AWS CB) is looped into the AWS ecosystem. The team keeps the questions updated and very relevant to the exam. If there is one service you pay for to pass the exam, this is it.

Jon and his team are great and will help you every step of the way.

How to use Tutorials Dojo

Do these exams at the end of your learning schedule
You only get 4 or 5 exams. Do not waste them by taking them unprepared
Go thru your study material and then follow up with the exam
Prepare yourself for 3 hours each and take one exam at a time in its entirety. If you try and take the exam in breaks you will lose a significant advantage
Do this as close to the exam date as your schedule allows. Doing them close to your exam date will keep the rigour
Use the review mode repeatedly. Even if you answer the question correctly, go over the others and read why they are wrong. The reasoning reinforces your choices

Click here to visit Tutorials Dojo

Ultimate AWS Certified Solutions Architect Professional 2021 by Stephane Maarek (Udemy)

Stephane Maarek is one of the best fast paced trainers out there for AWS certifications. If you have significant experience with AWS, then his courses will walk you through the exam oriented aspects of the services.

How to use Stephane's course

This is a slides only course. No UI or demos
Follow up with self guided practical sessions to understand the details which Stephane skips over
His course runs about 500+ slides. Go thru all of them
Review the slides before the exams. It'll take you up to 2 days to review. Plan your schedule
This is a long course, playback at 1.25x or 1.5x to save time

Visit Stephane Maarek's course on Udemy (Aff Link)

Visit Stephane Maarek's course on Udemy (Direct Link)

Study notes

Go through all the services whether you have used them or not
Write down characteristics and limitations of every service (e.g. EBS is 3IOPS/GB, Lambda can run for max 15 mins, Lambda vCPU increases with RAM)
Map services to keywords. e.g. NLB = expensive, fast, millions of connections, SQS = decouple services. Questions revolve around these keywords and if you use process of elimination as a last resort, these keywords will see you through
Take notes, lots and lots of notes. I filled an entire book with simple one liner notes as the course was going on
There are many services with similar names but different purposes. Clearly write their attributes to distinguish e.g. Different types of storage gateway options (file vs volume vs tape)

How to read the questions

Almost all questions are scenario based and require you to read them quickly. I devised a strategy that worked for me.

After the first 120 minutes, I had gone through the entire set of questions, completely skipped 5 cause they were too long, and had marked 12 of them for review. This gave me a full hour to review the ones I wasn't sure of.

The strategy is as follows:

You get about 2.5 minutes per question. Do not watch your time while reading the question. Focus only on the question especially the keywords & requirements
If the question is long or has answers that are long or involve multiple services - Mark it for review and skip it
Get through all the small ones first. Small means you can read it quickly and do not have several services involved
With each question:
- Read the scenario and requirement two times. Pick up keywords and pivot your answer around them. Read aloud to yourself if you have to. I can't count the number of times I ignored the main requirement (cost-effective, low overhead, speed, on-premises)
- Eliminate the wrong ones immediately. Wrong ones have clear issues e.g. Archive data to ephemeral storage
- Questions can have single word differences, read each word carefully e.g. instance-data vs user-data
- Once an answer is chosen, read it and map each requirement to the scenario and requirement. Sometimes the obvious answer will miss the requirement
- If you don't know the answer at all, work through process of elimination. Remove the obviously wrong ones and pick the one closest. There will almost always be clues that will lead you to the most probable answer

Scheduling the exam

I personally recommend you choose the morning slot. This exam unlike the associate ones is not about memory, it is about being alert. Stick to morning timings depending on the commute you have to the center. Try and wrap up the exam before lunch so you won't get hunger pangs at the fag end of the exam. I chose the 10:30AM slot which gave me ample time to do a mock test, get ready and travel.

Exam day - Game time!

I woke up early on exam day, after a small run, I took the final timed exam from Tutorials Dojo.

Notes:

Have a good breakfast. You'll be in the exam for 3 hours, you need the energy
Have a quick walk/run
Use maps to know the traffic and location of the center. Call the center if you can't find the way on maps. Ask about parking before hand
Try and reach at least 20 minutes earlier
Relax yourself, with deep breathing
There is no set exam time/batch. Once you arrive, they log you in and off you go
You cannot carry anything in the exam center (no watches, wallets, keys, etc). They'll probably give you a locker for your valuables
Don't forget your ID cards and other prerequisites as mentioned in the email
Bathroom breaks are allowed but the exam timer does not pause or stop. Best to take care of things before the exam starts
Exam centers generally have smaller monitors and uncomfortable chairs & keyboards. Be mentally prepared to not be physically comfortable
Ask for a pen & paper to write notes. You can only mark the question for review, not the choices. I avoided using the comments feature of the exam. Instead I wrote down the choices I was confused between, on paper and revisited them after I got done with other questions. The questions I skipped I wrote their numbers down in big bold letters
Remain calm. The first few questions and minutes can be overwhelming. Once you start answering your pace picks up (TD exams will also help pace you)
There is no penalty for wrong answers, so make sure you attempt all of them
If you finish early, review only the most ambigous questions. I tried reviewing the majority and wasted a lot of time
One of the issues I had was, if I had to review a question, I had to go thru it again, losing precious minutes. Train yourself with practice exams

And that's it, if you've practiced and read enough this should be a breeze. You'll see a PASS grade immediately.

Best of luck!

The perfect dev environment using AWS for large databases

Arif Amirani — Tue, 16 Mar 2021 07:23:12 +0000

During the start of a product, the database is quite small, mostly empty or populated with dummy data. Developers prefer having a local database instance running their favorite database like PostgreSQL or MySQL. This frees them from dependencies, and has the added advantage of having near zero latency.

As the product grows, inevitably the database also increases in size. In some cases, replicating production issues also requires that the code be run on a copy of the production database. Which eventually leads to often databases for development environments being created by copying a database dump from production and then importing that database dump. And since database dumps are text, they can be highly compressed, which can result in a relatively small file to copy over. But the import of the dump can still take lots of time and cause high load on the dev computer as it rebuilds tables and indexes. As long as your data is relatively small, this process may be perfectly acceptable.

Our team also went through the journey of setting up an acceptable strategy to work with our ever-growing database on Cassandra.

Early days

As we started out, the database was small. Feature set was growing faster than usage, which meant we had to resort to dummy data. We wrote a script to generate dummy data based on several parameters and usecases. It worked fairly well and kept everything on the developer's device.

For production issues, as the customers were beta and friendly, with consent, we seldom restored a copy of our daily backup to reproduce issues in our development environment.

Zero to one

MetroLeads data strategy is built on a schemaless model. Although we know the shape of the data, we rarely can count on it being complete or consistent. As the data passes through the data pipeline, it gets normalized for consumption by various stakeholders. When the product included features such as 3rd party integration, bring-your-vendor models, the situation was exacerbated. Data grew exponentially to be housed on a developer's laptop. The requirement of a consistent database for multiple microservices to run increased further.

To combat this situation we introduced the "shrinking process". Shrinking was a way to run a particular backup through a processing pipeline that:

Removed all customer data
Anonymize or scrub remaining data to remove traces of any PII (Personally Identifiable Information)
Leaves testing sandboxes intact
Reduced the number of events by time e.g. only keep events of last 7 days

Developers have their own production accounts which are connected to dummy vendors and QA communication stacks. For e.g. we use fake data generators such as Mockaroo which is my personal favorite and a combination of excel functions to generate large import payloads.

MetroLeads provides a sandbox for each organization. This makes it easy for us to remove all customer organization data in one-go during the shrinking process.

Over a period of time we extended the shrinking the process to target a specific organization. This allowed us to run the same scrubbing process on a customer account without compromising security or data policies.

Scaling up

We ran this process on our tools server which went from being a m3.medium to m4.2xlarge to handle simultaneous requests. We ended up timing them so that the load did not overwhelm the server. This was not scalable although worked for quite a long time.

Subsequently we hit an upper limit and decided to run the process as a daily cron with the developer accounts being refreshed every night. To avoid this being another tool that developers had to learn, we hooked it up as a Slack bot. Developers could simply run a Slack bot command and within a few hours the database would be made available on S3.

The shared solution

With the advent of Mumbai region the latency was no longer a problem. Our development team is mainly in Pune. Latency to the nearest Mumbai region was now below 40ms. Quite alright for our use case because our earlier architecture decisions lent themselves to handle this latency.

MetroLeads uses a combination of source of truth DB (Cassandra), a search engine (ElasticSearch), a message bus (RabbitMQ) and local caching (Redis). We decided to setup a shared database for all developers in Mumbai region. The idea was to setup all of our database in AWS and only keep Redis local to the developer laptop. As expected this worked really well for 80% of our UI based scenarios. The event processing flow was always meant to handle delays so that was never a problem.

We quickly changed our onboarding documentation to not require any database installations. Only setup your python servers with Redis and connect to the shared AWS server that housed all of the remaining databases.

The non-shared solution

At times several developers would work on the data set and would inadvertently overwrite changes. Since we already had a blueprint for setting up a remote database, we tweaked the blueprint to restore a shrunk database to any server of choice. This lead to an interesting setup; developers would launch their own servers in the nearest region and restore the blueprint on it. With large databases, this had two problems:

Increased cost of running servers
Stale data as developers would restore their copy less frequently

After a brainstorming session, we resorted to:

Launch only spot instances for non-shared servers
Use local port forwarding to switch between shared and non-shared servers
Kill non-shared servers as early as possible

Port forwarding was a great idea. It let us switch between development servers without changing config every time. We've seen local.env, local.dev1.env, local.dev2.env way too many times. With a port forward, you always point to localhost:9160 but depending on which server you've connected to, it will route it appropriately. On a Mac, we recommend using Core Tunnel.

QA and other environments

This post only focuses on developer's setup, however, the QA environments are a subset of that problem. Spinning up a new instance with dummy, test or production data is a sinch using the above techniques.

Conclusion

We built these approaches over a number of years, refining and tweaking them for our use case. There is a lot of room to improve and we are constantly striving to learn from others and get better at it.

Hope you enjoyed the article.

Project Sentiment Tracker Using AWS Comprehend + Serverless in 1 hour

Arif Amirani — Mon, 08 Mar 2021 16:39:07 +0000

Project is available at Sentiment Tracker

Github is a great place to contribute and create amazing projects. As the project scales, so does the team and the issues from users. Most are clear, concise and contructive to the project. Developers leave comments and answer these questions. For a large team, maintaining a positive sentiment around the project needs to be tracked. The goal of this project was to provide a fast way to keep a signboard about the overall project sentiment.

AWS Comprehend, provides a NLP service to provide insights on contextual text including sentiment analysis. I wanted to experiment how quickly we can go from idea to deployment for a trivial service without having to manage any resources. Below is my experience to solve the project sentiment problem using AWS at the heart of the solution.

Project Sentiment Tracker

You can view it at: https://st.arif.work/

Processes the recent issues and comments in the project
Analysis the text for sentiment patterns
Provides a quick badge for project owners

Technology choices

Analyzer - AWS Comprehend

AWS Comprehend was the obvious choice for setting up this low volume requirement. The API is easy and clear with no surprises on the usage pricing. Although it can do a lot more, I used it purely for the sentiment requirement. In the future, I hope to use Keyphrase Extraction, Topic Modeling and Entity Recognition to automatically tag issues and comments and even possibly assign them automatically.

Platform - Serverless - AWS Lambda

The API endpoints are simple. Apart from a small Oauth workflow, everything can be achieved without any added complexity. A perfect recipe for a serverless solution using AWS Lambda. I used Zappa and Flask as my deployment framework.

Database - DynamoDB

I've never used DynamoDB before in production. There are some nuances to its usage but for this small usecase, it was the easiest to get going. I used boto3 to connect to AWS services which is a breeze.

Github Connector - PyGithub

PyGithub is the defacto wrapper for Github v3 API. It has a 1:1 mapping to all APIs that Github exposes.

Presentation - Shields.io

Shields.io is the gold standard in presenting data points for a Github project. Apart from the vast array of services it supports (I was really surprised how much comes out of the box!), it also allows to integrate your own endpoint. The tracker integrates with shields to give the repo owner a badge that shows the current sentitment of the project.

Design

The idea is simple:

Submit a Github repo name
Authenticate with Github
Tracker connects with Github and fetches the most recent issues and comments
Submit them to AWS Comprehend for analysis
Store response in cache for 24 hours
Shields.io makes badge requests which is served from sentiment cache

Source code

https://github.com/kontinuity/github-sentiment-tracker

TODO

Oauth support for private repos
Refresh repo sentiment on updates
Circumvent Github rate limits

Hope you enjoyed the article.

Pi-hole on Raspberry Pi with IPv6

Arif Amirani — Tue, 23 Feb 2021 03:35:23 +0000

I've had a Raspberry Pi 4B sitting in my cabinet for a few months now. I dusted it off and realized that the SD card was busted. Got a replacement 64GB U3 A2 card and got it up and running with Ubuntu server. The primary intended use was to run docker with DB containers that I use for my side projects such as Postgres/MySQL/MongoDB.

My current home network consists of several routers for WiFi reachability. I try to ensure 5Ghz coverage to all my devices. Due to the concrete walls a single router does not suffice, hence a home made mesh.

The earlier ISP was a IPv4 only provider. Easy to setup. Worked out of the box. To safeguard other users of the network, I use a privacy sensitive DNS provider, Adguard. The issue with Adguard DNS was no geo proximity. It caused large ping times, as the closest CDN was never picked.

When I moved to my new ISP which supports IPv6, I decided to also deploy Pi-hole alongwith my docker containers.

Installing Pi-hole

Pi-hole is quite easy to deploy, especially with the auto install script.

curl -sSL https://install.pi-hole.net | bash

Once I followed the steps I got Pi-hole running out of the box. As suggested you have to change your router settings to send the IP address of your Pi-hole server as your local DNS. Done and done.

The Problem

Ads still show!

To ensure that I wasn't fudging up DHCP/DNS discovery I used scutil:

scutil --dns

resolver #1
  search domain[0] : lan
  nameserver[0] : 2405:201:xxxx:xxxx:xxx:xxxx:4209:2091
  nameserver[1] : fe80::a204:60ff:fe43:3005%en0
  nameserver[2] : 192.168.2.50
  if_index : 6 (en0)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

IPv4 looked good. 192.168.2.50 is the IP of my pi-hole server.

However, the IPv6 didn't belong to pi-hole. The default ISP setting was to use stateless IPv6 config and DNS was being advertised by the router. It was using my upstream DNS (ISP) server.

To fix this, I made the following changes:

IPv6 on pi-hole

Get your Ipv6 address for pi-hole

ip addr show

Enable IPv6 support on pi-hole

Set the IPV6_ADDRESS add /etc/pihole/setupVars.conf

This should've already been done during the installation of pi-hole but if isn't, you can manually set it up using the address above

IPV4_ADDRESS=192.168.2.50/24
IPV6_ADDRESS=2405:xxx:xxxx:xxxx::50/64

A neat trick I learnt while assigning static IPv6 addresses is to the use the same suffix as IPv4. Easy to remember and deal with.

Restart FTL:

systemctl restart pihole-FTL

Enable AAAA query analysis for Pi-hole

Pi-hole by default will only analyse A queries, so we need to add support for AAAA.

DBINTERVAL=60
MAXDBDAYS=7
AAAA_QUERY_ANALYSIS=yes

PRIVACYLEVEL=0

I also added a throttling parameters to not wear out the SD Card.

Refresh Pi-hole

Reboot the server. Fetch gravity lists again

pihole -g

Stateful DHCPv6

My attempts at trying to get DHCP v4 and v6 from pi-hole failed. This is probably due to wrong assumptions of SLAAC + RA.

Debug Pi-hole

At the end I made sure everything is kosher by running diagnosis on Pi-hole

pihole -d

Setting up Unbound

I do not like using any of the open DNS servers. They have their place and provide a lot of value but it's not for me. I prefer a combo of ad-block and recursive DNS. Unbound works perfectly with Pi-hole. Setting it up is again very simple.

apt install unbound

Update the unbound config to support Pi-hole

/etc/unbound/unbound.conf.d/pi-hole.conf

Make a note of the port, the default port 53 is used by Pi-hole and must be changed.

server:
    interface: 0.0.0.0
    interface: ::0
    port: 5335

    access-control: 192.168.2.0/24 allow
    access-control: 127.0.0.0 allow
    access-control: 2001:db8:dead:beef::/48 allow

    # unbound optimisation
    num-threads: 4
    msg-cache-slabs: 16
    rrset-cache-slabs: 16
    infra-cache-slabs: 16
    key-cache-slabs: 16
    outgoing-range: 206
    so-rcvbuf: 4m
    so-sndbuf: 4m
    so-reuseport: yes
    rrset-cache-size: 100m
    msg-cache-size: 50m

    # unbound security
    do-ip4: yes
    do-ip6: yes
    do-udp: yes
    do-tcp: yes
    cache-max-ttl: 86400
    cache-min-ttl: 3600
    hide-identity: yes
    hide-version: yes
    minimal-responses: yes
    prefetch: yes
    use-caps-for-id: yes
    verbosity: 1
    harden-glue: yes
    harden-dnssec-stripped: yes

    # download from ftp://ftp.internic.net/domain/named.cache
    # root-hints: "/var/lib/unbound/root.hints"

    # Ensure privacy of local IP ranges
    private-domain: "lan"
    private-address: 192.168.0.0/16
    private-address: 169.254.0.0/16
    private-address: 172.16.0.0/12
    # private-address: 10.0.0.0/8
    private-address: fd00::/8
    private-address: fe80::/10

Tweak it as per your needs and restart Unbound

systemctl restart unbound

Check unbound

Will pass:

root@pi-desktop:~# dig sigok.verteiltesysteme.net @127.0.0.1 -p 5335

; <<>> DiG 9.16.6-Ubuntu <<>> sigok.verteiltesysteme.net @127.0.0.1 -p 5335
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48863
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;sigok.verteiltesysteme.net.    IN  A

;; ANSWER SECTION:
sigok.verteiltesysteme.net. 3600 IN A   134.91.78.139

;; Query time: 499 msec
;; SERVER: 127.0.0.1#5335(127.0.0.1)
;; WHEN: Tue Feb 23 08:53:17 IST 2021
;; MSG SIZE  rcvd: 71

Will fail:

root@pi-desktop:~# dig sigfail.verteiltesysteme.net @127.0.0.1 -p 5335

; <<>> DiG 9.16.6-Ubuntu <<>> sigfail.verteiltesysteme.net @127.0.0.1 -p 5335
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 56203
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;sigfail.verteiltesysteme.net.  IN  A

;; Query time: 1147 msec
;; SERVER: 127.0.0.1#5335(127.0.0.1)
;; WHEN: Tue Feb 23 08:53:43 IST 2021
;; MSG SIZE  rcvd: 57

DHCP Confirmation

At least for IPv4, you can run the debug tool to make sure DHCP is correct.

root@pi-desktop:~# pihole-FTL dhcp-discover
Scanning all your interfaces for DHCP servers
Timeout: 10 seconds

* Received 301 bytes from eth0:192.168.2.1
  Offered IP address: 192.168.2.113
  Server IP address: 192.168.2.1
  Relay-agent IP address: N/A
  BOOTP server: (empty)
  BOOTP file: (empty)
  DHCP options:
   Message type: DHCPOFFER (2)
   server-identifier: 192.168.2.1
   lease-time: 3600 ( 1h )
   renewal-time: 1800 ( 30m )
   rebinding-time: 3150 ( 52m 30s )
   netmask: 255.255.255.0
   broadcast: 192.168.2.255
   router: 192.168.2.1
   dns-server: 192.168.2.50
   dns-server: 192.168.2.50
   domain-name: "lan"
   --- end of options ---

Switching Pi-hole to use unbound

Disable all Upstream DNS servers and add custom DNS that you setup for Unbound.
Use the loopback addresses for Unbound:

IPv4 - 127.0.0.1#5335

IPv6 - ::1#5335

That should be it! Hope you enjoyed reading the article.

5 Underrated React Design Systems for 2021

Arif Amirani — Fri, 19 Feb 2021 07:33:45 +0000

Design systems give your budding project a jump start and more importantly a structure when the project continues to grow. These systems bring in a level of sophistication of thinking and uniformity. Their value lies beyond pre-made CSS/JS assets. Identifying the right design system in the initial phases is crucial for progress. I employ various metrics to pick one such as:

Community support & acceptance
Documentation
a11y/i18n/l10n
Component library
Commit rate
Backers

However when it is time to play aka. a throwaway project for ML/AI or Raspberry Pi, you should experiment with the budding ones. Below are 5 of my current experimental design systems; known and unknown. They may not check all the boxes above however have potential and are super fun. They are not Bootstrap, Ant Design, or Material.

Grommet

part design system, part framework, and all awesome

Grommet is a react-based framework that provides accessibility, modularity, responsiveness, and theming in a tidy package.

Demo
Github

Garden React Components - Zendesk

The source of truth for tools, standards, and best practices when building products at Zendesk.

Garden is a minimal and clean design system that provides a formidable base for react projects.

Demo
Github

U.S. Web Design System (USWDS)

The U.S. Web Design System helps the federal government build fast, accessible, mobile-friendly websites.

The United States Web Design System includes a library of open source UI components and a visual style guide for U.S. federal government websites.

Demo
Github

Priceline One

Priceline.com Design System

In order to create a consistently great experience for our users, the design system is meant to be the single source of truth for user interface standards for both designers and developers.

Built off of the work of previous efforts, this project intends to consolidate those ideas into a living, well-documented, and growing system.

Demo
Github

Ring UI - JetBrains

A collection of JetBrains Web UI components

This collection of UI components aims to provide all of the necessary building blocks for web-based products built inside JetBrains, as well as third-party plugins developed for JetBrains' products.

Demo
Github

Hope you had fun reading!

AWS Cost Optimizations - The Easy Parts

Arif Amirani — Sun, 14 Feb 2021 13:07:41 +0000

Introduction

Moving to or starting with AWS (or any cloud provider) comes with an implicit assumption that your business will pay what it uses. Although technically true, most businesses ignore the human aspect. More often than not, developers will make assumptions while allocating resources and end up with:

Overprovisioned resources
Unused resources

In this article, based on our experiences and AWS events/whitepapers, we will outline a few approaches to combat the ramifications of these decisions at any stage of product growth. The earlier we recognize how to manage them constantly, the more cost savings we can achieve.

Remember saving costs means nothing if either your are spending too much time on it or not achieving your business goals.

Cost optimization is not a one-time activity. You cannot throw people at the problem once and hope the results will outlive your business goals. Organizations and smaller teams within them are more fluid than ever. They are adapting to the external (Porter's Five Forces) and internal forces being applied to the business. Thus it is prudent on our part to deal with cost optimization as a continous process of optimization.

One such useful technique is the Deming Cycle. PDCA (Plan-Do-Check-Act) is an iterative four-step management method used in business for the control and continuous improvement of processes and products.

For our cost optimization problem the four steps map to various opportunities and tools provided by AWS.

Plan

When developers launch new services or environments, cost optimization should be one of the metrics that is planned and predicted. Without compromising on functionality, a budget should be set alongwith any predictions on the usage. This can help the team decide on the right instances and regions to use from the get-go.

In practice, a time horizon of a quarter works great for new services that have no historical data on usage patterns. AWS also provides auto scaling features that can help in quickly reacting to sudden spikes of usage.

Even if the service is already deployed, minor tweaks in auto scale capacity (ECS or EC2), will allow you to get started on the cost optimization journey.

Tools to use

Auto scaling
Load tests

Do

you cannot improve what you cannot measure

With your services, you cannot tweak costs unless you know how they are being utilized and your spend pattern. You must identify your biggest spends and focus on them.

Use Tags

First tool to help you measure, is tags. Tags are a crucial and prolific way of categorizing and inspecting utilization of your cloud resources at every level. Tags can be set on resources instances, volumes, EIPs, AMIs, LBs, etc.

Standardize on tags across your teams and BU. Add the bare minimum dimensions and with meaningful information for the org, using each tag such as:

environment - prod, staging
team - frontend, backend
geography - eu, apac

These tags will give you different vantage points of your current and forecasted costs.

Auto scaling

Auto scaling plays an important in mitigating unpredicted service behaviors. Although we may plan for them, a sudden spike needs to be handled and not compromise on service performance or user experience. Setup auto scaling where possible with EC2, Fargate or ECS containers.

Compute Optimizer

AWS Compute Optimizer recommends optimal AWS resources for your workloads to reduce costs and improve performance by using machine learning to analyze historical utilization metrics. Compute Optimizer is available to you at no additional charge

The best part about CO is it not only recommends instance types but also visualize a what-if scenario. This helps you understand how the recommended instance would have performed on the recommended instance type.

Apart from AWS CO suggestions, you can also consider optimizing on cheaper instance alternatives such as the AWS Graviton instances. If the workload can be run on ARM then you can get more performance at a cheaper monthly price.

Consolidate control planes

If you're running EKS or ECS, you've got a control plane that is orchestrating the data plane. With EKS, you pay $0.10 per hour for each Amazon EKS cluster that you create. You can use a single Amazon EKS cluster to run multiple applications by taking advantage of Kubernetes namespaces and IAM security policies.

Hence, it's a good idea to analyse all your clusters and consolidate them into one or few as your architecture permits. Maybe reduce that by a couple different teams or even application types if you have a lot of applications that are say machine learning jobs or web applications you can break those down into different types of clusters that use different types of compute in each cluster that's optimized for those workloads. The larger the clusters, the more you can share some resources and bin pack things onto ec2 instances.

On-demand vs spot

If you architecture is fault tolerant and can restart failed processed or jobs, you can try to move away from on-demand instances to spot instances. This has helped us personally shave off almost 60% of our primary costs. To ensure fault tolerance via uptime, we have a hybrid approach of 20% on-demand and 80% spot instances.

Enable alerts and healthchecks

Your SLAs need to be met while you're optimizing. To react within your SLA, you must enable alerts and healthchecks where auto scaling does not help. Using AWS Cloudwatch and others, you should create a notification system to handle failures and spikes.

Tools to use

Auto scaling
AWS Compute Optimizer

Check

Monitoring for both, cost analysis and performance variance is essential. For cost, you have two options, Cost Explorer and Usage Reports. Cost explorer is more realtime and gives you an insight on your current spend whereas with CUR (Cost and Usage Reports), the current month’s billing data will be delivered to an Amazon S3 bucket that you designate during set-up. You can receive hourly, daily or monthly reports that break out your costs by product or resource and by tags that you define yourself. AWS updates the report in your bucket at least once per day. After setting up a Cost and Usage Report, you will receive the current month’s billing data and daily updates in the same Amazon S3 bucket.

You can read an in-depth article on AWS Cost Allocation Tags and Cost Reduction

You should set alerts on your spend which will automatically alert your predicted costs are exceeded.

Tools to use

Cost Explorer
Usage Reports
AWS Cloudwatch

Act

Once your optimization efforts have yielded results, it is time to make them the norm and perform hygenic actions.

First off, after a cycle of activities (quarter, product release, Black Friday sale), clear your unused resources. Identify them by running reports or checking usage, such as ALBs with no target groups, unused EIPs, very old snapshots, S3 buckets that have no new objects being added, unmounted EBS volumes, etc. Remove unhealthy or unused instances that may be up due to target capacity in Spot Requests.

Standardize the clusters and ensure new services are launched within the same clusters. Use AWS CDK or Cloudformation to provide templates to developers.

Tools to use

Cloud Formation
AWS CDK
AWS Cloudwatch

Finance folks that look at the bill once and pay it do not understand the business value of each line item. It is up to the developers to understand the workloads and the compute environment to continously optimize and reduce costs.

Hope you enjoyed reading the article.

Top 10 tools for the full stack developer

Arif Amirani — Sun, 31 Jan 2021 18:11:05 +0000

Mac! I absolutely love MacOS for development. It gives me the power of Unix with a whole lot of convenience. Can it be replaced with Unix? YES! Do I want to? No. As a full stack developer and CTO of my company, I spend 80% of my day in three apps:

Shell/Terminal
IDE
Browser

These three are the same no matter which OS I use. I see MacOS simply as a shell for my Unix environment. On a fresh install, I'll setup iTerm, oh-my-zsh, and install my most used packages from brew. I prefer brew over DMG or other package installations.

Listed below are top 10 packages that I use with zsh and iTerm. Not all are brew based though. Allons-y mon ami!

awless

awless is a powerful, innovative and small surface command line interface (CLI) to manage Amazon Web Services.

awless is my go to tool to avoid using the clunky AWS console. Although awless does not cover all services, the most important ones are covered i.e. EC2 and RDS. I can start/stop my instances, check on health, etc across regions.

z - jump around

z tracks your most used directories, based on 'frecency'.

After a short learning phase, z will take you to the most 'frecent' directory that matches ALL of the regexes given on the command line, in order.

Working on several projects; front-end and backend requires moving around directories very rapidly. We base our project names on MCU characters which works very well for z.

z will take some time before it becomes really good at understanding where you want to jump. Once it does, though it saves a ton of time as you use shorter target names.

E.g. I'd start by using

z bifrost

But now it's down to

z bi

aria2

aria2 is a lightweight multi-protocol & multi-source command-line download utility. It supports HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink. aria2 can be manipulated via built-in JSON-RPC and XML-RPC interfaces.

Downloading large files such as backups or log files is a regular occurence. I prefer to use multiple segments and resume capability. Suprisingly, browsers can only resume and in some cases that does not work as intended. I use aria2 for all large files and have a zsh alias to allow for multiple segments.

alias a2c='aria2c --max-connection-per-server=4 --min-split-size=1M --file-allocation=none '

bat

bat is a cat clone that supports syntax highlighting for a large number of programming and markup languages. It is also pretty smart when it comes to showing binary files. Supports line numbering, pagination, etc. All of this is quite helpful while browsing through files.

I have cat aliased to bat and cat aliased to ccat for the times when I just need to echo and pbcopy it.

ripgrep/rg/ack

ripgrep is a line-oriented search tool that recursively searches your current directory for a regex pattern. By default, ripgrep will respect your .gitignore and automatically skip hidden files/directories and binary files. ripgrep has first class support on Windows, macOS and Linux, with binary downloads available for every release. ripgrep is similar to other popular search tools like The Silver Searcher, ack and grep.

I moved from standard grep to rg mostly for code search but it has now taken over all of my search use cases.

gr (mixu/gr)

gr is a multiple git repository management tool. Managing all the repos in one command is a time saving tool. Check status of all repos? One command. Update all repos? One command. It hasn't been updated in a while but it still works as intended.

With support for auto repo detection, tags and many other features, it is an indispensable tool in my shell kit.

httpie

HTTPie (aitch-tee-tee-pie) is a user-friendly command-line HTTP client for the API era. It comes with JSON support, syntax highlighting, persistent sessions, wget-like downloads, plugins, and more.

I use Postman only for complicated use cases. My go to tool for regular HTTP API calls is the command line httpie. Works amazingly well and does not have the GUI clunkiness of Postman.

parallel

GNU parallel is a command line tool for running jobs in parallel. It allows you to do a ton of stuff when working with multiple files. It goes beyond mere execution. The docs can be difficult to work with initially but once you get the hang of it, it becomes essential.

One of my use case, is to redeploy containers across multiple aws clusters. I could use a for loop to do it but parallel makes it much easier. Example:

parallel aws --region us-east-1 ecs update-service --service apps-svc-prod --force-new-deployment --cluster {} ::: prod-a prod-b prod-c

The above command will run the same command with 3 different arguments in parallel. You get the power of replacement with parallelism.

mtr

mtr is a powerful tool that enables administrators to diagnose and isolate networking errors and provide reports of network status to upstream providers. MTR represents an evolution of the traceroute command by providing a greater data sample as if augmenting traceroute with ping output.

exa

exa is a modern replacement for the venerable file-listing command-line program ls that ships with Unix and Linux operating systems, giving it more features and better defaults. It uses colours to distinguish file types and metadata. It knows about symlinks, extended attributes, and Git. And it’s small, fast, and just one single binary.

I've configured exa as a replacement with many options as aliases.

l='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store"'
la='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store" --extended --tree --level=1'
lac='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store" --sort=accessed --time=accessed'
lar='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store" --extended --tree --level=2'
lara='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store" --extended --tree'
larr='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store" --extended --tree --level=3'
larrr='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store" --extended --tree --level=4'
larrrr='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store" --extended --tree --level=5'
lb='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store" --sort=size'
lch='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store" --sort=changed --time=changed'
lcr='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store" --sort=created --time=created'
ll='exa -lhgF'
lm='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store" --sort=modified --time=modified'
lr='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store|node_modules" --git-ignore --tree --level=2'
lra='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store|node_modules" --git-ignore --tree'
lrr='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store|node_modules" --git-ignore --tree --level=3'
lrrr='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store|node_modules" --git-ignore --tree --level=4'
lrrrr='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store|node_modules" --git-ignore --tree --level=5'
lt='exa --long --all --header --classify --group --color-scale --ignore-glob=".git|.DS_Store" --modified --changed --created --accessed'

Hope you guys liked the post!

Service Mesh - Introduction

Arif Amirani — Sun, 31 Jan 2021 14:23:07 +0000

As a development team matures and moves across various stages of code organization and system design, the deployment layout also adapts. Change can be a function of product growth, team size changes, technology decisions, or a combination.

General progression is from a monolith to handful of homogenous microservices. As the product gets diverse and team size grows, these microservices become heterogeneous. They can use different languages, servers, API end points, etc.

An explosion in microservices requires each team to deal with the same set of issues and service meshes provides a non-intrusive albeit expensive way to solve them.

What is a service mesh

Wikipedia describes it as:

In software architecture, a service mesh is a dedicated infrastructure layer for facilitating service-to-service communications between microservices, often using a sidecar proxy.

A sidecar proxy is an application design pattern which abstracts certain features, such as inter-service communications, monitoring and security, away from the main architecture to ease the tracking and maintenance of the application as a whole.

A sidecar will sit on the same node (or pod) as the service instance and externalize certain network traffic related functionality.

In practice, it would look something like the architecture below:

Having such a dedicated communication layer can provide a number of benefits, such as providing observability into communications, providing secure connections, or automating retries and backoff for failed requests.

Key benefits

Service meshes provide 3 key benefits to the services they connect with

Telemetry

Microservices distribute the fabric of the application into manageable pieces. However, the service landscape behaves as a single unit. Data passes across the service landscape to serve a single request. The ability to see the data for the same request across services as a single unit is desirable for debugging, troubleshooting, and analysis. This is called observability.

Service mesh intercepts all traffic for the service and hence is in a unique position to provide telemetry data to a central server for each request. Visualization dashboards can be connected with the telemetry data to get an x-ray of the service performance at a granular level.

Full dashboard:

Traffic Control

Service mesh use proxies such as Envoy to intelligently route traffic. Traffic routing allows interesting use-cases such as:

A/B Testing
Canary releases
Load/Latency based traffic handling
Circuit breaking
Retries
Timeouts

Below is the traffic route architecture for Istio. It contains Istio specific components but gives a general idea about how traffic routing works with service mesh.

The most important thing to remember about service mesh vis-a-vis traffic control is that, service mesh features are destination oriented. In other words, service meshes are well suited to balance individual calls across a number of destination instances, but rather unsuitable to control traffic from a number of sources to an individual destination or to control traffic across an entire service landscape, for that matter.

Because service mesh control extends from Layer 4 into Layer 5 and above, some also offer development teams the ability to implement resiliency patterns like retries, timeouts and deadlines as well as more advanced patterns like circuit breaking, canary releases, and A/B releases.

Service mesh focuses on east-west network rather than north-south. This means external communication or providing endpoints for external consumption is still left to the traditional load balancers and API gateways.

Policy Enforcement

A service mesh also supports the implementation and enforcement of cross cutting security requirements, such as providing service identity (via x509 certificates), enabling application-level service/network segmentation (e.g. "service A" can communicate with "service B", but not "service C") ensuring all communication is encrypted (via TLS), and ensuring the presence of valid user-level identity tokens or "passports."

Challenges

Service meshes have their own challenges, both technical and business.

Latency and cost

Due to the added layers of management, performance can suffer in some cases. Case in point, Istio publishes its own latency benchmarks as opposed to non service mesh deployments. You can read more about it at Istio Performance and Scalability

Smaller setups

If you have a small team or set of microservices, service mesh might be an overkill. Alternatives include:

An in-process solution, a library such as Netflix Hystrix or Twitter Finagle might be a more appropriate and effective way to address your service communication pain points.
There’s little need to abstract the details away, hence a service mesh might be an overkill.
You might want to avoid the added complexity of sidecar proxies (costs, debugging, etc.) making introducing a service mesh too early in the process might be counterproductive.
Immaturity of the technology in general, lack of hands-on experience, or size of the community behind solutions.

Options

Popular service meshes include:

Istio
Istio is an extensible open-source service mesh built on Envoy, allowing teams to connect, secure, control, and observe services. Open-sourced in 2017, Istio is an ongoing collaboration between IBM and Google, which contributed the original components, as well as Lyft, which donated Envoy in 2017 to the Cloud Native Computing Foundation.
Linkerd
Linkerd is an "ultralight, security-first service mesh for Kubernetes," according to the website. It's a developer favorite, with incredibly easy setup (purportedly 60 seconds to install to a Kubernetes cluster). Instead of Envoy, Linkerd uses a fast and lean Rust proxy called linkerd2-proxy, which was built explicitly for Linkerd.
Consul Connect
Consul Connect, the service mesh from HashiCorp, focuses on routing and segmentation, providing service-to-service networking features through an application-level sidecar proxy. Consult Connect emphasizes application security, with proxies offering mutual Transport Layer Security (TLS) connections to applications for authorization and encryption.
Kuma
Kuma, from Kong, prides itself on being a usable service mesh alternative. Kuma is a platform-agnostic control plane built on Envoy. Kuma provides networking features to secure, observe, route, and enhance connectivity between services. Kuma supports Kubernetes in addition to virtual machines.
Maesh
Maesh, the container-native service mesh by Containous, bills itself as lightweight and more straightforward to use than other service meshes on the market. While other meshes build on top of Envoy, Maesh adopts Traefik, an open-source reverse proxy and load balancer.

Supporting technologies within this space include: Layer 7-aware proxies, such as Envoy, HAProxy, NGINX, and MOSN; and service mesh orchestration, visualization, and understandability tooling, such as SuperGloo, Kiali, and Dive.

Glossary

API gateway: Manages all ingress (north-south) traffic into a cluster, and provides additional. It acts as the single entry point into a system and enables multiple APIs or services to act cohesively and provide a uniform experience to the user.

Consul: A Go-based service mesh from HashiCorp.

Control plane: Takes all the individual instances of the data plane (proxies) and turns them into a distributed system that can be visualized and controlled by an operator.

Data plane: A proxy that conditionally translates, forwards, and observes every network packet that flows to and from a service network endpoint.

East-West traffic: Network traffic within a data center, network, or Kubernetes cluster. Traditional network diagrams were drawn with the service-to-service (inter-data center) traffic flowing from left to right (east to west) in the diagrams.

Envoy Proxy: An open-source edge and service proxy, designed for cloud-native applications. Envoy is often used as the data plane within a service mesh implementation.

Ingress traffic: Network traffic that originates from outside the data center, network, or Kubernetes cluster.

Istio: C++ (data plane) and Go (control plane)-based service mesh that was originally created by Google and IBM in partnership with the Envoy team from Lyft.

Kubernetes: A CNCF-hosted container orchestration and scheduling framework that originated from Google.

Kuma: A Go-based service mesh from Kong.

Linkerd: A Rust (data plane) and Go (control plane) powered service mesh that was derived from an early JVM-based communication framework at Twitter.

Maesh: A Go-based service mesh from Containous, the maintainers of the Traefik API gateway.

MOSN: A Go-based proxy from the Ant Financial team that implements the (Envoy) xDS APIs.

North-South traffic: Network traffic entering (or ingressing) into a data center, network, or Kubernetes cluster. Traditional network diagrams were drawn with the ingress traffic entering the data center at the top of the page and flowing down (north to south) into the network.

Proxy: A software system that acts as an intermediary between endpoint components.

Segmentation: Dividing a network or cluster into multiple sub-networks.

Service mesh: Manages all service-to-service (east-west) traffic within a distributed (potentially microservice-based) software system. It provides both functional operations, such as routing, and nonfunctional support, for example, enforcing security policies, quality of service, and rate limiting.

Service Mesh Interface (SMI): A work-in-progress standard interface for service meshes deployed onto Kubernetes.

Service mesh policy: A specification of how a collection of services/endpoints are allowed to communicate with each other and other network endpoints.

Sidecar: A deployment pattern, in which an additional process, service, or container is deployed alongside an existing service (think motorcycle sidecar).

Single pane of glass: A UI or management console that presents data from multiple sources in a unified display.

Traffic shaping: Modifying the flow of traffic across a network, for example, rate limiting or load shedding.

Traffic shifting: Migrating traffic from one location to another.

Dart null safety - Solving the billion-dollar mistake

Arif Amirani — Mon, 23 Nov 2020 18:29:37 +0000

In a 2009 talk, Tony Hoare traced the invention of the null pointer to his design of the Algol W language and called it a "mistake":

I call it my billion-dollar mistake. It was the invention of the null reference in 1965. At that time, I was designing the first comprehensive type system for references in an object oriented language (ALGOL W). My goal was to ensure that all use of references should be absolutely safe, with checking performed automatically by the compiler. But I couldn't resist the temptation to put in a null reference, simply because it was so easy to implement. This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years.

Source: https://en.wikipedia.org/wiki/Void_safety

What is null safety?

Null was devised as a special value to represent the intentional absence of any value. This is different from an empty string, a zero int or false. It may or may not exist.

Color favoriteColor;

favoriteColor can be either a Color object or null. This distinction is important understand. Developers may incorrectly refer to the null value as:

Color object which has a null value

On an average, 80 - 90% of variables in our code will have a value. An int will be initialized to a number or is set before being used. This is valid and no harm will fall upon your app. But as we all know, this happens more than 10% of the time:

To be clear, null-safety does not mean nulls are bad. On the contrary, a null value has valid use cases. For example,

class Person {
  String firstName;
  String middleName;
  String lastName;
}

In the above case, an application can assume firstName and lastName will be present and may perform no checks. middleName may not be present, in which case, it will either be a String or null.

Null safety is crucial to avoid human errors. Null safety provides a declarative mechanism to understand application logic.

Let's contrast a code block with and without null safety.

Without null safety, a typical code block would look like:

void middleAllCaps(Person p) {
  return p.middleName.toUpperCase();
}

It would work in most cases, and when middleName is null we would encounter the dreaded Null error:

Uncaught TypeError: Cannot read property 'toUpperCase$0' of nullError: 
TypeError: Cannot read property 'toUpperCase$0' of null

Show the above to a developer, and you would get a staunch reply, "No worries! I got this", and a null check would appear.

void middleAllCaps(Person p) {
  if (p.middleName == null) return '';
  return p.middleName.toUpperCase();
}

Again, this is a good answer but as developers we forget to do this before going to production, and that is a problem.
Why not let the language help us deal with potential null variables.

With null safety on, and we'll talk about how to enable it for dart, we have options of indicating "nullability".

Non nullable

By default, with null safety on, all variables are non nullable. This will be valid for 80-90% of your code.


class Person {
  String middleName;
}

void middleAllCaps(Person p) {
  return p.middleName.toUpperCase();
}

The code above will not compile because we have indicated middleName as non-nullable but not initialized it.

Error: Field 'middleName' should be initialized because its type 'String' doesn't allow null.
  String middleName;
           ^^^^^^^^^^
Error: Compilation failed.

One solution is to add the constructor with required parameter. The other is late keyword which we will talk about later.

class Person {
    String middleName;
    Person(this.middleName);
}

String middleAllCaps(Person p) {
    return p.middleName.toUpperCase();
}

In essence, Dart will perform flow analysis to ensure a non-nullable variable is initialized.

Nullable or the ?

You can indicate to Dart that a variable is nullable i.e. it can have a value or not. Nullable variables do have the potential for blowing up. Dart will use flow analysis to prove that no such error can occur. If it cannot it will ask the developer to handle it during compilation itself.

class Person {
  String? middleName;
}

void middleAllCaps(Person p) {
  return p.middleName.toUpperCase();
}

This would cause a compilation error.

Error: Method 'toUpperCase' cannot be called on 'String?' because it is potentially null.
  return p.middleName.toUpperCase();
                        ^^^^^^^^^^^
Error: Compilation failed.

Why? The ? indicates that the variable can have a null value. Dart's flow analysis will identify a potential null error and halt compilation. The developer is expected to explicitly handle the condition such that p.middleName is not accessed or read when the value is null. Or in other words, add a null check.

void middleAllCaps(Person p) {
  return p.middleName?.toUpperCase() ?? '';
}

So far so good! Dart's flow analysis will detect the null check and continue compilation.

Nullable force override or the !

Dart's flow analysis will not detect setters especially where classes are involved. What if you have a class with no constructor but you ensure initialization outside of it. Dart may not understand that flow in which case you want to force Dart to assume the value will always be available. You do that by adding the ! to the accessor.

class Person {
  String? middleName;
}

String middleAllCaps(Person p) {
  // The trailing ! symbol will skip null check
  return p.middleName!.toUpperCase();
}

void main() {
  Person p = Person();
  p.middleName = 'Fancy';
  print(middleAllCaps(p));
}

The developer is responsible for setting the value before accessing it. There may be a handful of cases where overriding null check is acceptable but generally considered a code smell.

required keyword

When creating classes using positional non-nullable fields, Dart's flow analysis may not detect setting of variables. E.g.

class Person {
  String middleName;
  Person({this.middleName});
}

This will yield a compilation error. middleName is non-nullable but the constructor will set it to null if value is not passed. This is not acceptable. You can use the required keyword to indicate that middleName is a required non-nullable positional parameter.

class Person {
  String middleName;
  Person({required this.middleName});
}

late keyword

Not all non-nullable fields may be initialized in the constructor. The analyzer cannot prove that the value will be set before access. Such as the following case:

class Person {
  String middleName;

  void assignRandomMiddleName() {
    middleName = 'Wilcox';
  }
}

String middleAllCaps(Person p) {
  return p.middleName.toUpperCase();
}

void main() {
  Person p = Person();
  p.assignRandomMiddleName();
  print(middleAllCaps(p));
}

In such cases, you should not force override using ! but rather tell Dart that the variable will be initialized before reading the value like so:

class Person {
  late String middleName;
}

If you set the late keyword and do not initialize it, you won't get a nullError but rather a LateInitializationError:

Uncaught Error: LateInitializationError: Field 'middleName' has not been initialized.

How to enable null safety

Null safety is still experimental. Library developers have already started pushing out null safe versions but you can continue run your apps without switching over.To opt-in to null safety for your project make sure:

You have dart version 2.10+
All dependencies are null safe

To run an app using null safety on:

dart --enable-experiment=non-nullable --no-sound-null-safety bin/myapp.dart

To enable your IDE to provide null safety errors you have to modify your analysis_options.yaml and add the following options:

# Defines a default set of lint rules enforced for
# projects at Google. For details and rationale,
# see https://github.com/dart-lang/pedantic#enabled-lints.
include: package:pedantic/analysis_options.yaml

# For lint rules and documentation, see http://dart-lang.github.io/linter/lints.
# Uncomment to specify additional rules.
# linter:
#   rules:
#     - camel_case_types

analyzer:
  enable-experiment:
      - non-nullable

How do I make my code null safe

dart comes with a migration tool built-in. More options are available on migration in the dart docs

dart migrate

How I got used to vim (neovim)

Arif Amirani — Sun, 22 Nov 2020 06:47:27 +0000

TLDR; I got faster at writing (code and articles) once I stopped fighting the editor. My journey from a mouse dependent editor to a keyboard one.

As an active developer, I constantly strive to optimize my workflows. Anything that is repetitive must be automated to the nth degree. All actions that are pure, that is to say, yield the same result if the inputs are same are easy to do.

I have an elaborate zshrc (based on oh-my-zsh), Mac Automator actions, Alfred workflows to generate UUIDs or query data from my production database. These little gems save a ton of time and ensure no human errors are made.

Most of my time is spent in an editor writing code, articles, reports, etc. In all these formats, a significant chunk is navigation. I seldom write new code and I will review articles or rewrite them a bit. Any developer will tell you that code writing is jumping to locations (jump to definition, symbol and back) and then adding more code.

Since the past decade, I've been using IDEs for code writing and text editors like Sublime Text for articles and standalone scripts. Keeping aside the issue of resource usage, IDEs do a good job of allowing you to write code faster. When it comes refactoring IDEs are a boon. Having said that, they fundamentally do not solve the problem of speed/efficiency. In other words, I was still performing more keystrokes to do the same repetitive task. I saw myself constantly shuffling between the keyboard and mouse to get navigation or editing done. Shortcuts are good but they don't work across different editors. I spent a lot of time trying to configure each editor the same way.

My goal was to write faster irrespective of the content type.

Line or terminal based editors have always been a holy grail of efficiency for me. For the simple reason that they do not assume a pointer device, all actions are fine tuned to be done from a keyboard.

Source catonmat.net

catonmat explains the design of keys in this article

Emacs and vim both offer the same promise, get efficient at editing via keyboard. I tried using Emacs for almost 3 months a few years ago. A half-hearted attempt that did not result in anything worth mentioning.

Vim however, has been my go to editor when logging into terminals. Whenever a config needed tweaking, I'd fire up vim. Vim ubiquity made it so that I never lost touch. Full time usage is a whole different thing though.

TLDR; Vim (or Emacs) improves your navigation speed.

At some point, I started off with gusto to move lock, stock and barrel to vim. That attempt sorely failed. I was expecting workflows to work. The basic hjkl keys I had learnt, I thought would be enough. My productivity plummeted and I blamed on how difficult vim is.

My mistake was that I conflated the two problems:

I want write faster
I want to replace my IDE

Several months ago, I revisited my editor needs. I spoke to people who do use vim and started to focus on the first problem. Focusing on speed removed the friction and mental block of comparing output. I started doing non code activities on vim like articles or JSON formatting, etc.

Changes since my last attempt also made life easier:

Neovim got LSP support and with coc I got basic JSON editing working better than Sublime
ThePrimeagen Vim coding sessions gave me insight into practical keystrokes that help in efficient writing
GoNeovim - a GUI based neovim instance cause I hate having a tab in iterm for my editor (personal quirk)

How did Vim increase my writing speed

My hands stay on the keyboard 90% of the time
I do minimal wrist position changes to get anything out of the current window, be it manipulating text, navigating, performing terminal actions
Vim motions change the way you view existing text on the screen

Vim motions

Vim motions need their own section. Motions take some time getting used to. Once you understand what motions want you to view text as they become significantly simpler to work with. With motions I started seeing text as markers. I won't get into the motions tutorial, the documentation does a great job of explaining each one.

Motions in itself reduced time spent in micro context switches of navigation. A simple motion of Shift + [ meant moving up a object block. In 90% of the cases I'd land on the right place.

The curve below was accurate for me. My output decreased but for non critical tasks. That helped me stay on course to keep trying to learn.

Keymaps

Vim comes with a usable set of maps that you don't need to change. To make vim work for you though I've seen every single true vim developer add keymaps. They are shortcuts that you create which work only for you. I've tried so many times copying keymaps from a 3k+ stars github repo only to delete it with prejudice.

Keymaps make vim even better. My experience has been to keep trying different options. Some of my milestones have been:

Moved leaderkey twice - comma, space
Not using Meta keys (Alt/Cmd) - Now I do everywhere
Tried to use plugin defaults - Now I disable them

Moving on to code editing

With LSP and neovim, there was enough to move smaller projects into vim. I still don't use vim for large/complex projects that require IDE capabilities. I find the required plumbing an absolute overkill.

But that's not it!

I use Intellij which comes with IdeaVim. It gets me 80% of the way. All motions work, keymaps work. Whatever I use CoC for, is built into Idea. I had to tweak Idea to get the keymaps working identically to nvim. IdeaVim is also constantly improving; providing access to Idea IDE actions via keymaps which makes my life a lot easier.

Neovim is embeddable and I'm waiting for it to be incorporated for true vim experience. VSCode already does that with VSCodeVim

How do I improve with Vim

Read blogs and watch videos (at 1.5x) to find better ways
- ThePrimeagen
- ChrisAtMachine
Github dotfiles - Scour github.com for new nvim configs
Vimawesome - a great resource to find the latest and greatest packages
Coc plugins - I love CoC. They keep dishing out new plugins for usecases

What I still do wrong (as others may say it)

Use arrow keys - Some blogs make me feel guilty but I understand why
Limit nvim to only some usecases
Run nvim in a GUI window - So many people have told me to use terminal

Conclusion

It's been a great journey so far. Using vim on a regular basis for 90% of my writing activities. I have seen visible improvement in my output and now editing is not the biggest time drain of my day.

Nova Editor by Panic - A Quick Review

Arif Amirani — Sun, 22 Nov 2020 06:45:35 +0000

Over the years the choices of IDEs and code editors have reduced significantly. We earlier had quite a few options across OSes from BBEdit, Notepad++, Eclipse, etc. However now with the need for advanced IDEs there are only a handful picks that work across all OSes, and that is great! We have contenders in both the open source (VS Code, Vim, etc) and commercial (Intellij, Sublime Text, etc) spaces. These are all amazing editors and IDEs but when it comes to native ones that are fast and native the choices are quite limited.

I've been a fan of Panic since I started using a Mac professionally about a decade ago. Transmit has been a god send when dealing with file transfers and they've improved it over the years to work seamlessly with cloud platforms. That's not the best part, Panic does an amazing job adhering to and most of the times improving the Apple Human Interface guidelines. If a product is coming out of the Panic stable, it's virtually guaranteed to be aesthetically pleasing and conforming to the Mac experience.

So when Nova was announced, I was unsurprisingly excited to try it out. Unfortunately I did not get a chance at the beta and eventually my enthusiasm died out as I went back into the traditional IDE world.

Recently though I had a chance to get my hands on Nova and I was thoroughly excited to try it out. The below review points are based on my usage for a few days using it for a relatively small Typescript project.

To set the context straight, I work on a Macbook Pro 2019 with 16GB RAM and a touch bar (Almost never use it). I have two code editors that I use daily:

Intellij Idea Ultimate as my IDE for Python, Flutter, Web
Neovim (goneovim) as my code editor for Web, Go, articles

The distinction is important because I crave performance, speed and efficiency when coding as much as possible, but also need the expansive IDE experience for refactoring, wiring, etc which is too cumbersome with a deeper integration.

With the advent of LSPs (https://microsoft.github.io/language-server-protocol/), even editors can now provide features of an IDE out-of-the-box. The trick is presenting those features in a seamless manner.

VIM has LSP support but there is a lot of fumbling needed to get it to work correctly although coc (which I use) does a pretty decent job.

Coming back to Nova, my expectations from Nova are that of an IDE. Something that goes beyond the typical editor experience whilst retaining all of the goodness of a native app from Panic (native look and feel, speed, chock full of features) and Nova does not disappoint.

Initial setup

Installation of Nova is pretty straightforward, download the dmg from https://nova.app/ and install it. The default configuration works really well however I was quick to change the font to Fira Code and higher point size.

Nova is geared more towards web development out-of-the-box so you'll get support for CSS, HTML, Javascript languages. Being an Applesque app, it shares commonality with Xcode and gives you options to start projects or simply open directories

You can open single files and get editing right away.

Configuration

Configuring Nova can be a bit jarring if you come from an IDE. The nomenclature is a bit different alongwith the organization of options (more Xcode). There is no search provided for so many settings which I am used to with Intellij/VSCodium (yay! no telemetry). It does take time to get things setup the way one likes and it does not follow the dark theme.

Compare this to Intellij settings

Extensions

Nova supports a ton of extensions and alongwith a rich API that will allow you to work with a myriad of languages, themes, etc. I believe the choice of available extensions is based on popularity. For e.g. all the popular themes on Vim, Idea are available but none of the key bindings. If your language is supported good for you, mine wasn't (Dart).

The experience of working with extensions is great. A simple search and install feature and off you go. More extensions are being made available regularly and the ecosystem will get richer. How much of this is Panic's push? No idea.

Code Editor & Experience

You got one job Nova! Code editing is generally good with lots of panes and information floating around to work with. The command palette similar to Sublime Text, Idea, VSCode gives you quick access to functionality. The default keybindings are bit off putting especially coming from Idea/Vim/VSCode.

You get the standard code folding, structure navigation, etc similar to others.

Even with support for LSP, the code actions are limited and does not allow a lot of refactoring.

~~Another gripe is that basic string manipulation is not available out-of-the-box such as duplicate line (WAT!). We need an extension to get that working (DOUBLE WAT!)~~

This is not correct, nova has included this since v1, the problem is that they decided to name it "Copy lines Down and Copy Lines Up" the guy who did that extension did not find it either. - imustknowsomething

Workflows

Workflows is similar to Run actions that others provide. Honestly I didn't get time to explore it much and I don't need it a lot too. I prefer to execute workflows via the terminal. I am sure there is good stuff in there.

The Cool Parts

Built-in Git support

Publish/Remote servers

This is where Nova shines. It has Transmit kinda built-in. It allows you to push content to remote servers which they support a lot of.

Pricing

Nova has a steep price point of $99 USD with each additional year at $49 USD especially when you consider open source alternatives like VSCode. Having said that, in some way this price is justified with the overall experience and the potential of Nova improving rapidly. Their robust API gives me a lot of hope that the language support and features will only get better.

Conclusion

I went in thinking I'll get a cool new IDE but I believe Nova can be best termed as a code editor on steriods that looks pretty (and this is important cause you spend 8 - 12 hours on this screen).

Folks in web development should definitely try it and consider it for themselves. If you're working in Android, Flutter, etc then you may need a full fledged IDE where I can recommend Intellij Idea and VSCode.