DEV Community: Praneet Loke

Running Folding@Home...in the cloud

Praneet Loke — Tue, 19 Jan 2021 21:35:37 +0000

You've probably read about Folding@Home in 2020 and its role in the drug discovery for treating COVID-19.

To help tackle coronavirus, we want to understand how these viral proteins work and how we can design therapeutics to stop them.

https://foldingathome.org/diseases/infectious-diseases/covid-19/

All over the world people were running F@H to support their call to fight the virus with computing power. And the world responded to this call. There were many notable organizations (CERN, EVGA, AWS, NVIDIA and more!) running the Folding@Home client on their machines, thus donating spare cycles for a good cause. For this effort, the team behind F@H put out a special call. They were asking people with dedicated GPU hardware to run the client.

Unfortunately, for those of us who don't keep up with the world of gaming PCs anymore or simply do not have access to high-end GPUs, all we could do is run the F@H client on our regular machines. It was (and still is!) a good way to help them in their efforts.

For me, the solution was the public cloud providers. If I can't buy one, I'll rent one. In my case, I ran high-end instances on AWS and Azure for several months back in 2020. But there was no way I would be able to afford running those machines at their full price.

Enter Spot Instances. The big 3 cloud providers all allow users to run workloads on machines that have unused capacity. Sometimes at a whopping 80% discount on the hourly pricing! This means one can run high end machines at a fraction of the hourly price depending on availability in a certain data center (region). I have been able to amass quite a few points on F@H and climbed up the ranks quickly. From being nowhere on the leaderboards to somewhere in the top 30k (my rank has slipped quite a bit since I haven't run any work units in sometime) in a matter of few months. This is due to the way their system credits clients running high priority work units specifically designed to use GPUs.

After running this infrastructure for several months last year, I am proud to announce that I am open-sourcing the very application that will allow you to run Spot Instances on AWS or Azure as well. It comes complete with the Pulumi app, as well as GitHub Actions workflows to automate the whole thing. All you need to do is, to configure the cloud credentials on GitHub and set some configuration for Pulumi.

There is a lot to explain in the Pulumi app, but I hope to do another follow-up post on this topic. My hope is for you to get started on protein folding in the cloud in less than 10 minutes. I have tried to add as much details as I can in the README. But don't be shy if you have questions. Ask me! :)

praneetloke / FoldingInTheCloud

Run Folding@Home...in the cloud.

I am open-sourcing this because F@H does important research. They grew in popularity in 2020 due to COVID-19. But they do a lot of other research too! You should definitely read more about them.

A closer look at Stadia and gaming in the cloud

Praneet Loke — Wed, 06 Jan 2021 03:43:40 +0000

What a time we live in when one of the most anticipated games of the last decade plays better on Google Stadia than on dedicated gaming hardware in your living room! I am talking about consoles. Regardless of whether you are a gamer or not, it is likely that you've read about Cyberpunk 2077 and its performance on last gen consoles, which can still be found in hundreds of millions of households. Especially given that its been in development for quite literally a decade and comes from one of the most acclaimed developers CD Projekt Red (CDPR).

CDPR is also the maker of The Witcher series of games. You know, the hit Netflix show of the same name. Both inspired by the books.

Nevertheless, as a gamer and a cloud technology enthusiast I was excited to take a look and experience this all for myself.

A New Hope?

When Google announced Stadia about 2 years ago, the gaming community was rather quick to judge it; myself included. That's because that sort of tech has been pushed upon the gaming community in the past. OnLive was a service that existed sometime ago. It seems that Stadia is breathing new life into this concept. I'd say their timing couldn't be better. I can't think of a better story for them to tell than the time when their platform outshined consoles. Admittedly, this is because of the way the game performs on traditional consoles, but also because some of Stadia's promises held their ground.

OnLive vs. Stadia

Requirements	OnLive	Google Stadia
Custom controller	Yes	No	You don't need the Stadia controller if you only plan on playing on your computer. But if you want to have a decent gaming experience on your TV without using your laptop/desktop to cast it to your TV you will need their controller. That's fair. More on this later on.
Custom streaming device	Yes	Yes	Although, in Google's defense, the Chromecast doubles as a Miracast device so it is more than just for Stadia.
Subscription	Yes	No	Stadia's Pro subscription is completely optional. That is, if you want to play in 4k with 5.1 surround sound, you'd better pony up $10/month. But to take advantage of your Pro subscription, you'll need the Chromecast Ultra. Google's 4k streaming device.

All in all, you can see where I am going with this. The similarities are there in terms of how this is setup. However, the technology running in the data center is vastly different. For instance, Wikipedia says OnLive used custom video compression chips on their servers which they operated in their own data centers across the US.

Stadia Hardware

There is no need for speculation on what hardware Stadia might be using, since they have it right on their website.

At the time of this writing, it is:

For good measure, Google partnered with AMD to design the GPUs for these machines. The consoles from both Microsoft and Sony have had custom AMD-designed GPUs too. The next-gen consoles have all AMD chipset too.

Those specs are no joke either. They are the sort of specs Microsoft and Sony are touting as next-gen hardware. More or less. But the benefit of Stadia is that hardware can be upgraded seamlessly whereas consoles cannot be. Or so they promise. How often these "transparent" hardware upgrades come about will surely meet the reality of economics, I say. Again, it stands to be seen.

The Stream

Despite Stadia's perceived dubious launch, the technology is no piece of cake. It's pretty amazing cloud technology. I have been playing Cyberpunk 2077 on Stadia for about 2 weeks now and sometimes I forget that I am not playing the game "locally".

Let me explain. Much like its video streaming counterparts like Netflix and Hulu, cloud gaming also has a video stream. But unlike video streaming, Stadia cannot buffer 5 mins of a game's stream on your Chromecast or computer or wherever it is you are playing. It's like watching live TV, but without any artificial delays other than the network latency itself. For the network latency to not affect the game too much, the video stream must be compressed and also adjust according to dips in network quality. It is in Google's best interest to really concentrate their efforts in ensuring the best gaming experience. The first sign of a poor gaming experience and they are done. The gaming market as big as it is, is also pretty unforgiving. The fact that a AAA game publisher like CDPR rushed to release -- a game that's been in development for about a decade -- in time for the holidays after a series of delays angered "gamers" on social media (to the point where the devs were receiving death threats!), has got to say something about the kind of pressure this market puts on companies. I don't judge them and would like to give devs the benefit of the doubt. CDPR is certainly not the first (Ubisoft, EA are guilty of this too!) and won't be the last company to find themselves in trouble for releasing a buggy AAA game.

Stadia's dedicated cloud gaming hardware renders the game using the dedicated GPU. I suspect these custom GPUs also have a dedicated hardware-based encoder of some sort. It wouldn't be too far fetched to think that because Nvidia's latest generation of GPUs support hardware-based encoding and thus freeing up the CPU for other things. This ensures that you have a stutter free streaming. These streams also need to be compressed efficiently (probably using the most common video codec H.264 or who knows Google might have even built a custom video codec for speedy delivery!)

With a video streaming to your Chromecast, it is now time to discuss the controller. The controller is its own technical marvel.

The Controller

As per another article, Google has a patent for the controller. I haven't read through the patent myself and haven't confirmed the relevancy to the Google Stadia controller. I wonder how different this is from OnLive's controller, though, since its controller setup may have had to be similar to Stadia's.

The controller needs to have an ultra-low latency connection with Google's servers for there not to be any perceived latency. Remember, what you see on your TV (or on your Chrome browser) is merely a video stream of a game that is being played elsewhere. Hopefully, at a data center closest to you. Your controller input needs to travel up to the Google's servers, rendered and the stream comes down to your TV. That's a lot of traveling if you ask me.

Digital Foundry's Richard Leadbetter demonstrated measuring the latency between the controller's input and the game's stream updating according to the input.

Getting The Best Experience

While I have tried to play Cyberpunk 2077 on Stadia using Wifi, the experience is just not good enough. Even while connected to the 5Ghz band of your router, in real world scenarios, you are likely to encounter some amount of interference or crowding. There is simply no replacement for a hard-wired connection here. To eliminate any possibility of a bandwidth reduction, you must use a wired connection and also ensure that you have nothing else in your network that is internet-heavy (like watching YT, Netflix, Hulu etc.) It is also the reason why Stadia recommends playing using a hard-wired internet connection. Although I think they should be stronger with that recommendation, almost to the point where they require it, like OnLive did

Perhaps with more devices certifying for Wifi 6, it may not be much of a problem in the near future since the standard specifically addresses maintaining QoS for multiple devices simultaneously. But there is nothing like a good ol' cable capable of handling Gigabit speeds. A nice Cat 6 cable will be more than enough in this situation. Beyond that, you are talking data center cabling which is not required and quite honestly you will never realize its potential in home use.

Lastly, I recommend getting the Chromecast Ultra. Its power adapter comes with an ethernet port and offers up to 4k streaming for games played on Stadia.

Cost

The cost of all this of course cannot be simple. Let's try to calculate the cost.

Since Stadia uses custom hardware, we can only draw approximations from similar hardware from their cloud services offering.

Resource	Quantity	Montly Cost (USD)
Nvidia Tesla T4 GPU	1	160 (approx.)
A2 instance type (accelerator-optimized CPU) vCPU (including memory)	1	30 (approx.)

The above prices are monthly costs for a consumer using those resources on GCP. That's easily over the price of one next-gen console. I haven't even included the storage costs here and probably some other costs that I am not even aware of.

Those machines are pushing out a lot of data with the 4k 60fps promises Google is making for Pro subscribers. The subscription costs just $120/year. They have got to be losing money here in order to gain gamers' trust. I doubt the free Stadia experience comes cheap either. There is still cost involved in operating those machines. And you only pay for the games in the free experience whereas in the Pro tier you pay for the games, get monthly free games (of Indie quality) and get 4k (whenever possible). That's a great value for $9.99 if you ask me.

It is a herculean effort to have people ditch their consoles (or just "one highly anticipated game performing poorly on consoles" 😉) so I guess this is Google's way of getting more people to signup for the service to maybe try and offset the cost in the future. I can't imagine the Pro subscription staying at $9.99/month for too long into the future quite frankly. OnLive offered a plan for roughly $15/month. That was back in 2010. Everything's way more expensive now and Stadia is certainly not using 2010 hardware. These are high end specs.

Not Alone

Stadia is not alone in its foray into the $60bn USD gamingmarket. Amazon tried its hand and although, it met with failure, it appears it hasn't given up just yet. Amazon's approach was to start a gaming studio from the ground-up, though. And not to compete with the console market like Google is.

But I wouldn't be surprised if Amazon enters changes direction or perhaps starts something parallel to its game development efforts. Seeing what Google is up to and the sudden success of Stadia is hard to ignore for a company like Amazon given that its cloud computing arm, AWS, has been partnering with Nvidia to create custom GPUs for all sorts of things. In fact, cloud gaming is one of the use cases for the AWS-specific G4 instances that sport an Nvidia T4 GPU. They are fully capable of what Stadia is doing along with RTX. They even have gaming drivers that you can install on Linux, Windows Server 2019, and Ubuntu.

Then there's Nvidia's own GeForce Now service and the Nvidia Shield console and controller. GeForce Now is a cloud gaming service, but not in the same way that Stadia is. With GeForce Now you need to buy the game from a service like Valve's Steam and give GeForce Now access to it. Then you can use Nvidia's cloud prowess to play the game with the highest specs, though, there are some limitations. I thought I read somewhere that you can't play the game whenever you would like and that you'll have to enter a queue during busy times. The one big reason why this could be attractive to gamers will experience Nvidia's famed RTX. A feature that is typically attributed to high-end gaming GPUs and most recently the next-gen consoles. RTX is one of the biggest selling points for the next-gen hardware.

Speaking of Steam, Valve offers Steam Link for free. A service that allows you to play a game running on another computer. Sounds like something you have read before, yes? 🙂

Finally, let's not forget Microsoft and Sony. Microsoft is already offering Project xCloud in the form of its Xbox Game Pass subscription. And what about Sony? Well, that's a different story. We'll have to wait for Sony's move on this one.

Final Thoughts

Since the early 2010s, cloud hardware has come a long way. I haven't even talked about the memory and the SSD that Stadia offers in its hardware stack right from its first generation. That in itself is a generational leap for the traditional living room consoles. Seriously, the next-gen consoles Xbox Series X/S and Sony's PS5 just now have built-in SSD along with GDDR6 memory. Cloud gaming certainly has some strict network requirements and its success is rooted in consumers having the best internet connection. Not to mention the proximity of these cloud gaming capable machines, housed in data centers, to the consumers. I don't believe it will replace traditional gaming experiences, be it through consoles or gaming PCs. If there is one thing 2020 has taught the world, it's that anything can happen.

I am excited to see what this next decade brings for gamers. One thing's for certain, a new contender has entered the arena. Its name is Stadia.

I wrote a browser extension to protect your privacy

Praneet Loke — Fri, 02 Oct 2020 01:05:11 +0000

Ever wonder if you could simply mask your tab while you were sharing your screen during a meeting or because someone is by your desk? Well, maybe not the latter recently because you know...🤷‍♂️. The extension allows you to mask any tab or if you want to mask all open tabs in the current window, there is a shortcut key for that. Oh and you can customize the shortcut key bindings to your liking. You are most likely going to have to customize it since it seems that most shortcut combinations that I could think of were in conflict with something else 🤦‍♂️.

The browser extension is available for Firefox, and Edge (Chromium) right now, with Chrome becoming available pretty soon! Download the extension for Firefox here and Edge here.

If you have any feedback or issues to report, please consider submitting an issue using one of the templates here.

Here's a gif showing you how it works.

Roll your own Instagram gallery in 5 mins

Praneet Loke — Fri, 28 Aug 2020 16:00:26 +0000

A client had asked me about showing Instagram pictures from their own account on their website. A bit similar to Twitter's own timeline widget.

However, Instagram doesn't have a widget to display pictures in a feed. So I built one myself. This is a post about how you could do that for your website too. It took a bit of reverse-engineering but it all worked in the end and hopefully stays that way.🤞

Although this post talks about a single Instagram account, you could modify what I show here to pull images from other accounts too. I make no guarantees!

The Facts

Instagram has a developer API platform.
The method this post covers is only for publicly accessible accounts.
If you need to show pictures from a private account, you must use the official Instagram API platform to register an app, get an API key, get it reviewed etc. The same applies if you want to use their platform to even show public images in a feed, unlike Twitter's timeline widget for public profiles.

The Query Hash

Navigate to the Instagram account you want to show pictures from and open the Developer Tools and click on the Network tab. With the Network tab open, refresh the Instagram page you are viewing. Then click on the XHR filter. Here's what it looks like for me in the latest Firefox browser.

Network tab showing only XHRs

Instagram uses GraphQL to fetch the posts, where each post contains URLs to different sizes of the picture, stats (likes, comments etc.), author and much more. We need the query hash from the request so we can make the same API call ourselves.

You can find the query hash from the request made to the endpoint https://www.instagram.com/graphql/query/. Once again you can filter the requests using the textbox to find the specific request.

Instagram GraphQL API request

Here's the full URL from Developer Tools window:
https://www.instagram.com/graphql/query/?query_hash=d4d88dc1500312af6f937f7b804c68c3&variables={"user_id":"249074882","include_chaining":false,"include_reel":false,"include_suggested_users":false,"include_logged_out_extras":true,"include_highlight_reels":true,"include_live_status":true}

Save the value of the query_hash query-parameter for later. We'll plug that into our own request to the same API endpoint.

The Vue Component

While the following is a Vue component, you could very easily do the same with any other front-end framework.

Declare some constants for our API calls.

// These two URLs will be the same regardless of the account
// you want to show pictures from.
const IG_API_URL = "https://www.instagram.com/graphql/query/";
const IG_POST_URL = "https://www.instagram.com/p";

// The `query_hash` value from before.
const TIMELINE_QUERY_HASH = "d4d88dc1500312af6f937f7b804c68c3";
// The number of images to fetch per page.
const IMAGES_PER_PAGE = 5;

Make the API call

async fetchData(after) {
  const variables = {"id":"249074882","first":IMAGES_PER_PAGE};
  const api = axios.default.create();
  const resp = await api.get(IG_API_URL, {
    params: {
      "query_hash": TIMELINE_QUERY_HASH,
      "variables": JSON.stringify(variables)
    }
  });
  return resp.data;
}

Transform the data for display. Feel free to modify this according to what you want to show.

transformData(timeline) {
  if (!timeline.hasOwnProperty("edges")) {
    return;
  }
  const edges = timeline.edges;
  edges.forEach(edge => {
    if (!edge.hasOwnProperty("node") || edge.node["__typename"] !== "GraphImage") {
      return;
    }
    const node = edge.node;
    const thumbnails = node["thumbnail_resources"];
    const urls = thumbnails.filter(t => t["config_width"] === this.imgProps.w);
    let url;
    if (!urls || !urls.length) {
      url = thumbnails[0].src;
    } else {
      url = urls[0].src;
    }
    this.posts.push({
      url,
      href: `${IG_POST_URL}/${node["shortcode"]}`
    });
  });
 }
}

And that's it. With just those few lines, we have made an API call and retrieved a page of Instagram posts just like Instagram's own official site.

Putting it all together

<template>
  <section class="instagram mb-10">
    <div class="d-flex flex-row flex-md-row flex-column flex-sm-column justify-center align-center">
      <v-card
        v-for="(post, index) in posts"
        :key="index"
        tile
        class="pa-2 ma-2"
      >
        <a
          :href="post.href"
          class="post"
          rel="noopener"
          target="_blank"
        >
          <img
            :src="post.url"
            width="100%"
            height="auto"
          >
        </a>
      </v-card>
    </div>
  </section>
</template>
<script>
import * as axios from "axios";

const IG_API_URL = "https://www.instagram.com/graphql/query/";
const IG_POST_URL = "https://www.instagram.com/p";
const TIMELINE_QUERY_HASH = "9dcf6e1a98bc7f6e92953d5a61027b98";
const IMAGES_PER_PAGE = 5;

export default {
    name: "Instagram",
    data() {
      return {
        posts: [],
        imgProps: {
          w: 320,
          h: 320
        }
      };
    },
    async created() {
      const resp = await this.fetchData();
      if (!resp.data.hasOwnProperty("user") || !resp.data.user.hasOwnProperty("edge_owner_to_timeline_media")) {
        return;
      }

      this.transformData(resp.data.user.edge_owner_to_timeline_media);
    },
    methods: {
      async fetchData() {
        const variables = {"id":"20214540375","first":IMAGES_PER_PAGE};
        const api = axios.default.create();
        const resp = await api.get(IG_API_URL, {
          params: {
            "query_hash": TIMELINE_QUERY_HASH,
            "variables": JSON.stringify(variables)
          }
        });
        return resp.data;
      },

      transformData(timeline) {
        if (!timeline.hasOwnProperty("edges")) {
          return;
        }
        const edges = timeline.edges;
        edges.forEach(edge => {
          if (!edge.hasOwnProperty("node") || edge.node["__typename"] !== "GraphImage") {
            return;
          }
          const node = edge.node;
          const thumbnails = node["thumbnail_resources"];
          const urls = thumbnails.filter(t => t["config_width"] === this.imgProps.w);
          let url;
          if (!urls || !urls.length) {
            url = thumbnails[0].src;
          } else {
            url = urls[0].src;
          }
          this.posts.push({
            url,
            href: `${IG_POST_URL}/${node["shortcode"]}`
          });
        });
      }
    }
}
</script>
<style lang="scss" scoped>

.instagram {
  overflow: hidden;
}

@media all and (max-width: 640px) {
  .instagram {
    .ig-container {
      .columns {
        display: flex;
        justify-content: center;
      }
    }
  }
}
</style>

That's it for now. If you like this post and want me to write about something else, please let me know! Until next time, see ya! 👋

Increasing Confidence in Infrastructure Changes

Praneet Loke — Fri, 28 Aug 2020 05:33:10 +0000

Pulumi is an infrastructure as code platform that lets you write cloud infrastructure using popular general-purpose languages (Go, JS, TS, Python, and .NET).

Running Pulumi in GitLab CI/CD is very easy and if you have ever wondered "is there a tool that tells me how my changes will affect my infrastructure", then that's exactly what running pulumi preview will give you.

Outputs of all Pulumi CLI commands are viewable in the Pulumi Console too. You could also view any Pulumi command executions in realtime on the Console.

Summary output as viewed in the Pulumi Console

Use the toggler to view different types of logs

Diff output as viewed in the Pulumi Console

There is one problem, though. How do you, as the reviewer, ensure that you see this summary during your review? You could check the Pulumi summary in the build pipeline logs, but it can be a bit cumbersome to have to navigate to the pipeline logs if you have to do that repeatedly. It is also likely that reviewers miss looking at the pipeline builds and thereby potentially missing some crucial unintended impacts to your infrastructure.

Enter Pulumi integration for GitLab Merge Requests.

Infrastructure summary in the comment

So there you have it, folks! A summary of your infrastructure changes delivered right to your comments in the merge request, so you never miss a change again.

Haven't started with Pulumi yet? What are you waiting for? Click here to get started.

Unit Testing Angular Components That Use MatIconRegistry

Praneet Loke — Fri, 12 Jun 2020 18:48:08 +0000

If you use an Angular Material component (part of Angular Components) that relies on the MatIconRegistry, chances are you have wondered how you could test it with mock data. I recently ran into this, and figured I would turn it into a post as a way to document it for myself. Who knows -- maybe one of you out there might find this helpful as well!

MatIconRegistry is useful if you have a custom SVG icon set that you want to use within your app. This allows you to explicitly add the icons under a custom namespace using the addSvgIcon(iconName: string, url: string) method.

Let's say you use the mat-icon component in your component's template:

<mat-icon svgIcon="icon-name"></mat-icon>

...testing your component means you'll need to ensure that the the icon is served properly, at least with a fake icon.

To do that, you'll need to first add the icon names that your component expects to find in the registry. Ideally, you'd do this in the beforeEach setup function.

import { HttpClientTestingModule } from "@angular/common/http/testing";
import { inject, TestBed } from "@angular/core/testing";
import { MatIconRegistry } from "@angular/material/icon";
import { DomSanitizer } from "@angular/platform-browser";

describe("TestingComponent", () => {
    const fakeSvg = `<svg><path id="someId" name="someSvg"></path></svg>`;
    const fakeIconPath = "/fake/icon.svg";

  beforeEach(() => {
    TestBed.configureTestingModule({
      imports: [HttpClientTestingModule],
    }).compileComponents();
  });

  beforeEach(inject([MatIconRegistry, DomSanitizer], (mir: MatIconRegistry, sanitizer: DomSanitizer) => {
    // The `MatIconRegistry` will make GET requests to fetch any SVG icons that are in the registry. More on this below...
    const sanitizedUrl = sanitizer.bypassSecurityTrustResourceUrl(fakeIconPath);
    // Make sure that the icon name matches the icon name your component would be looking up.
    mir.addSvgIcon("icon-name", sanitizedUrl);
  }));

  it("test my component", () => {
     // Create a test fixture of your component's instance and test your component.
  });
}

That should do it, right? Nope. Your test would likely fail with a timeout error. But the reason may not be immediately clear. The answer lies within the functionality of the MatIconRegistry and what happens when you use the attribute with svgIcon with a <mat-icon> component.

Looking at the docs for <mat-icon>, note how it says this:

In order to guard against XSS vulnerabilities, any SVG URLs and HTML strings passed to the MatIconRegistry must be marked as trusted by using Angular's DomSanitizer service.

MatIconRegistry fetches all remote SVG icons via Angular's HttpClient service. If you haven't included HttpClientModule from the @angular/common/http package in your NgModule imports, you will get an error at runtime.

That's good. We did that in our beforeEach call above. Note that for unit testing we use HttpClientTestingModule and not HttpClientModule because, well, it's a unit testing compatible HTTP client. One that doesn't actually hit a real endpoint. This means we need to control the response we send back to callers making HTTP requests based on the URL they request. Sort of like setting up a method spy and returning a mock value when the method is called. Only here we would be matching against the URL being requested and return an appropriate response.

So let's update our snippet above:

import { HttpClientTestingModule, HttpTestingController } from "@angular/common/http/testing";
import { inject, TestBed } from "@angular/core/testing";
import { MatIconRegistry } from "@angular/material/icon";
import { DomSanitizer } from "@angular/platform-browser";

describe("TestingComponent", () => {
    const fakeSvg = `<svg><path id="someId" name="someSvg"></path></svg>`;
    const fakeIconPath = "/fake/icon.svg";

  beforeEach(() => {
    TestBed.configureTestingModule({
      imports: [HttpClientTestingModule],
    }).compileComponents();
    httpTestingController = TestBed.inject(HttpTestingController);
  });

  beforeEach(inject([MatIconRegistry, DomSanitizer], (mir: MatIconRegistry, sanitizer: DomSanitizer) => {
    // The `MatIconRegistry` will make GET requests to fetch any SVG icons that are in the registry. More on this below...
    const sanitizedUrl = sanitizer.bypassSecurityTrustResourceUrl(fakeIconPath);
    // Make sure that the icon name matches the icon name your component would be looking up.
    mir.addSvgIcon("icon-name", sanitizedUrl);
  }));

  it("test my component", () => {
     // Create a test fixture of your component's instance and test your component.
     ...
     ...
     // Use this to capture icon requests
     // and flush it manually so that the source observable will emit a value. Otherwise, async calls will timeout waiting for a response.
     httpTestingController.match(fakeIconPath)[0].flush(fakeSvg);
  });
}

That's it! Happy testing!🙂

Coding Home Automation - Part 2

Praneet Loke — Fri, 12 Jun 2020 03:55:53 +0000

In the previous post, I walked-through a home automation recipe where I showed you how easy it is to wire-up existing automation platform with your own custom solution that runs in the cloud. Let's take a look at how to deploy that on Azure using code.

Getting Ready

Install the Pulumi CLI from https://www.pulumi.com/docs/get-started/install/.
The next step is to create an account on Pulumi Console, and I'll explain why you should signup for an account later in this post.
Optionally, create an Azure Service Principal, so you don't have to re-authenticate when your credentials expire.

Project Creation

Create an empty directory somewhere on your local disk. You can init a git repo as well if you'd like, but not necessary for using Pulumi for this project. If you are looking to use Pulumi at work, you should definitely use an SCM of some kind.

Open your favorite terminal window depending on your OS, and cd to your newly-created directory, then run pulumi new (assuming pulumi is on your PATH).

This will show you a few templates, that you can use to create a project. You don't need to use any of them. In fact, you don't even have to run pulumi new. You can do this all manually if you prefer that, but it's certainly an easy way to get started.

Imports And Config

import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure";
import * as path from "path";

Let's take a look at the imports at the top of the index.ts file. All pulumi npm packages will be under @pulumi.

@pulumi/pulumi is the core Pulumi SDK providing things like the configuration class, the programming model constructs for things components and custom resources. It also contains the types for recognizing inputs and outputs, and some helpers to work with both of those.

@pulumi/azure is the Azure-specific SDK for creating Azure resources.

path is the standard NodeJS package.

import { buildFunctionsProject} from "./projectBuilder";

projectBuilder is a TypeScript file that exports just one function buildFunctionsProject.

const namePrefix = "grge-mon";
const config = new pulumi.Config();

The Config class is used to retrieve config key/values stored in the Pulumi.<stack_name>.yaml file. Learn more here.

const twilioAccountToken = config.requireSecret("twilioAccountToken");

This is a really cool part. You can add secrets to your config and retrieve them easily in your code. Pulumi will track the value of this variable as a secret. You add secrets to your stack config using pulumi config --secret set <key> <value>. Learn more here.

buildFunctionsProject(path.join("..", "GarageDoorMonitor"));

As mentioned before, the buildFunctionsProject is an exported function used from another file. This function builds our .NET Core Functions project.

Creating The Infrastructure

Alright. Let's create some Azure resources. 🏗️

const resourceGroup = new azure.core.ResourceGroup(`${namePrefix}-group`);

First, we will need a resource group to put our resources into. This is an Azure construct and not a Pulumi-specific thing. If you have worked with Azure, everything you know about it applies even when you use a programming language in Pulumi.

Azure KeyVault

const kv = new azure.keyvault.KeyVault(`${namePrefix}-vault`, {
    resourceGroupName: resourceGroup.name,
    skuName: "standard",
    tenantId: azure.config.tenantId!,
    accessPolicies: [{
        tenantId: azure.config.tenantId!,
        // The current principal has to be granted permissions to Key Vault so that it can actually add and then remove
        // secrets to/from the Key Vault. Otherwise, 'pulumi up' and 'pulumi destroy' operations will fail.
        //
        // NOTE: This object ID value is NOT what you see in the Azure AD's App Registration screen.
        // Run `az ad sp show` from the Azure CLI to list the correct Object ID to use here.
        objectId: "your-SP-object-ID",
        secretPermissions: ["delete", "get", "list", "set"],
    }],
});

We create the KeyVault resource using the new operator. This sort of looks like the properties in the ARM template for creating a KeyVault, except, here you get the advantage of strongly-typed arguments. This makes it really easy to specify values without second-guessing what you need to provide to a property.

Note: A KeyVault resource in Azure uses Access Policies to restrict who can administer it. This also means that if you use a service principal (or your own personal account) to run the Pulumi app, it will need access to the KeyVault to add/remove secrets/keys/certificates. This is why we specify the object id of the account in the access policies initially.

Adding a secret

const twilioSecret = new azure.keyvault.Secret(`${namePrefix}-twil`, {
    keyVaultId: kv.id,
    value: twilioAccountToken,
});

Let's add a secret to the KeyVault. Notice how we specified the KeyVault to which the secret should be added by simply referencing the variable kv from the previous step. That's how you would normally pass values to anything that depends on a value from another object, right? But why am I calling this out like it's a big deal? Remember that this is no ordinary app. We are dealing with infrastructure resources here. These are not just some variables with values stored in memory.

These variables represent actual resources on Azure. This also means that when the KeyVault is still being created, a secret cannot be added to it. So how does Pulumi know when to extract the id property from it? Well, the answer to that is resource ordering. Pulumi knows that the resource represented by kv needs to finish creating before a new Secret resource is added to it. This is similar to how you would specify dependsOn in an ARM template to tell ARM how to order your resources and to flow outputs of one resource's creation to another as an input. Instead, in Pulumi this happens automatically as you just go about writing regular TypeScript code.

These variables represent actual resources on Azure. This also means that when the KeyVault is still being created, a secret cannot be added to it. So how does Pulumi know when to extract the id property from it? Well, the answer to that is resource ordering. Pulumi knows that the resource represented by kv needs to finish creating before a new Secret resource is added to it. This is similar to how you would specify dependsOn in an ARM template to tell ARM how to order your resources and to flow outputs of one resource's creation to another as an input. Instead, in Pulumi this happens automatically as you just go about writing regular TypeScript code.

String interpolation with infrastructure resource outputs

String interpolation in modern JavaScript (and TypeScript) is achieved by enclosing a string using the backtick character and using ${} to insert a variable. But here we are dealing with special resources. The variables used in the string format may not have been created (yet). This is why the Pulumi SDK provides a special interpolation function. In the TypeScript SDK it's pulumi.interpolate. Use that with the standard JS interpolation syntax and Pulumi will recognize that the interpolation must be considered for resource availability before evaluation.

Creating an app insights dashboard is a piece of cake

const appInsights = new azure.appinsights.Insights(`${namePrefix}-ai`, {
    applicationType: "web",
    resourceGroupName: resourceGroup.name,
});

There is not much to say here (and that's a good thing here!) The code is pretty self-explanatory.

Function App

const durableFunctionApp = new azure.appservice.ArchiveFunctionApp(`${namePrefix}-funcs`, {
    resourceGroup,
    archive: new pulumi.asset.FileArchive("../GarageDoorMonitor/bin/Debug/netcoreapp3.1/publish"),
    appSettings: {
        "runtime": "dotnet",
        "TwilioAccountToken": pulumi.interpolate`@Microsoft.KeyVault(SecretUri=${twilioSecretUri})`,
        "APPINSIGHTS_INSTRUMENTATIONKEY": pulumi.interpolate`${appInsights.instrumentationKey}`,
        "TimerDelayMinutes": config.getNumber("timerDelayMinutes") || 2,
    },
    httpsOnly: true,
    identity: {
        type: "SystemAssigned"
    }
});

// Now that the app is created, update the access policies of the keyvault and
// grant the principalId of the function app access to the vault.
const principalId = durableFunctionApp.functionApp.identity.apply(id => id.principalId);

Pulumi provides some higher-level helpers for well-known/popular resources such as Azure Functions to ease with the package/deployment. Normally you will have to package your code as a zip and deploy it to the Function App. You can do this using a built-in task extension in Azure DevOps, or you can manually zip up your functions and deploy them directly on Azure. Using Pulumi to do this is very easy. And because Pulumi tracks every resource, it will only trigger an update to the code package if you truly changed your functions code. Otherwise, nothing is changed.

KeyVault access

// Grant App Service access to KV secrets
const appAccessPolicy = new azure.keyvault.AccessPolicy(`${namePrefix}-app-policy`, {
   keyVaultId: kv.id,
   tenantId: azure.config.tenantId!,
   objectId: principalId,
   secretPermissions: ["get"],
}, { dependsOn: durableFunctionApp });

Our Function App needs access to the KeyVault to access the secret. So let's create a new access policy and attach it to the KeyVault.

Outputs

export const webhookUrl = durableFunctionApp.endpoint;

At a basic level, think of your Pulumi app as a thing that is creating several things, but there is typically some output that is of interest to you for your application code. For example, the URL for the API service, IP address of a loadbalancer, the hostname of a managed Cosmos DB instance etc. To create an output you simply need to export it.

You can retrieve outputs from your stack later if you would like using the Pulumi CLI by running pulumi stack output <outputName> where <outputName> is the name of the variable that you exported.

Outputs play an important role in making your infrastructure modular. In this post we only used a single stack. In a more practical scenario, you are perhaps working with multiple teams and each of them may want to have their own stack. But you may have a dependency on the output of one of those stacks, you can use outputs from another stack in your own stack using a StackReference.

Pulumi Console And The Managed Backend

Just like the Azure Portal, Pulumi has a Console UI. The Console gives you a detailed view of all the resources in each stack, their outputs, the timeline of events, activities etc.

Pulumi tracks the state of your resources. It shows you diffs from the current state as you make changes to your infrastructure. At the beginning of this post I stated I would explain why you should sign-up for an account. To get the state tracking and diffs you don't need an account on Pulumi Console. However, if you want to keep that state safe, highly-available, and make sure concurrent updates are not performed on your infrastructure, which you will need when you are developing for production, you should use the Pulumi-managed backend, and let it take care of all of that for you. The alternative to this is, to manage all that on your own. Learn more about that here.

Closing Notes

It is important to note that what you saw above is not a Pulumi-flavor of TS. It is just TS. The same TS you would use to develop Angular apps or whatever else you use TS for these days. This also means everything that the language provides is available for you to use. There are no restrictions, other than what the Pulumi resource provider inflicts on your app.
Review the Pulumi Programming Model page for some advanced concepts. Particularly, the Components.
For JS and TS-based Pulumi apps, the runtime is NodeJS. So each TS file is transpiled to JS and executed inside the Node runtime just like any other Node app. This also means that you can use just about any Node-compatible npm package, including the built-in Node packages as well. In fact, I used the child_process package to execute the dot net publish command.
If you are planning to use Pulumi on Azure DevOps, then checkout Pulumi's free Task Extension for build and release definitions.

Opinion: The New Full-Stack Dev

Praneet Loke — Mon, 08 Jun 2020 01:51:37 +0000

There was a time when calling oneself a full-stack dev was met with confused faces. It was a fairly new idea. No one really understood what it really meant and if it was real. It was sort of like gluten and people claiming gluten allergy.

Years later, both couldn't be more real. In the early days of being a full-stack dev, it meant you could write services powering the UI (aka the backend), as well as some really cool jQuery selectors (like $("#someDiv .items:nth-child(2)") -- I'll let you figure out what that means.), some HTML and some CSS. If you knew jQuery you were the dev to go to at work.

The full-stack developer movement couldn't have started without javascript templating, backbone.js, and later SASS. Pretty soon after those were introduced a lot more developers claimed to be full-stack developers, which was bolstered by the creation of Twitter's Bootstrap. And suddenly, there was an explosion in the number of developers who could comfortably write both UI, and backend services. The gates to UI development were knocked-down pretty hard. There was probably a hot new frontend framework every week. From BackboneJS (the OG of bringing Model/View/Collection paradigm to frontend development, in my opinion) to today's React/Angular/Vue -- today's big 3.

A new framework was announced on JS Weekly!? SWIIITCH!!
-- Web Developer

From there it was a natural progression for developers to try and tackle writing mobile apps with PhoneGap, which was later renamed Apache Cordova. Arguably one of the most contested ways to write a mobile app. It was nonetheless a way for a web developer to also try their hand at mobile development too. To this day, there are numerous frameworks that promise the "write-once-run-anywhere" type of semantics. Xamarin, Flutter, React Native, NativeScript, Ionic Framework are all popular frameworks in that space. Regardless of what you and I believe the best way to build a mobile app to be, it was yet another skill a full-stack developer could add to their skillset.

Other Developments

Back when cloud services were first introduced, almost no one cared how you got to "the cloud", so as long as you were "on the cloud". Scripting languages became uber-popular in the ops area. Knowledge of bash, and PowerShell were (and probably still are) indispensable. Tools like Chef, Puppet invigorated the infrastructure space with their ideas. There are probably more tools that I have probably never even heard of.

All this while every major cloud provider was cooking up their own template-based toolset to deploy services to their respective clouds. Surely all of the great minds that thought of the amazing ways to build software for the cloud could have seen the problem a mile away, right? Nope.

AWS has CloudFormation templates, Azure has ARM templates, and Google has...uh, I don't even know and I don't want to. This is a clear and present problem.

This is a clear and present problem. It is what kept the infrastructure space away from the general reach of your average developer who just wants to deploy a simple service to the cloud.

Redefining Full-Stack Dev

With HashiCorp's Terraform, this changed. Infrastructure "gurus" started to see the need for a system that would let them deploy repeatable, and predictable infrastructure to the cloud. This works really well if this is what you want. But if you don't want to learn a new DSL?

If you noticed at the beginning of this post, every innovative piece of technology introduced in the developer ecosystem, involved taking something that was accessible to a niche of developers and turning it into a programmable thing. HTML became JS template, CSS was too brittle, so SASS was created which brought some general-purpose programming language paradigms to CSS (when was the last time you wrote CSS? I mean, raw CSS, not SCSS as you know it today.), mobile apps could be built frontend frameworks. At each stage, the adoption of the respective technology was through-the-roof.

We have been going about trying to solve the cloud infrastructure tooling the wrong way all this time.

Allowing developers to do what they do best -- write software, will certainly bring infrastructure to the masses. This is why tools like Pulumi will disrupt this space. The movement to bring infrastructure to the masses has begun. Pulumi brings the programmability, the "coding" aspect to infrastructure. For the first time, it really feels like I can truly program the cloud.

That is not to say the tools that have been created so far have no place in building infrastructure. Nope. Quite the opposite. I think there is a need for these tools. For instance, not everyone is using the cloud the way cloud providers want us to use them. Many of them are tied to legacy on-premise systems, leading to the so-called "Hybrid Environments". Whatever the setup, there is a place for each of those tools.

And for those of us building something new for the cloud, there's Pulumi. So let me define the new full-stack dev...

A developer who can work on all tiers of an application, including the infrastructure.

What do you think?

(Cover image: Photo by mongrel67 on Unsplash)

Coding Home Automation

Praneet Loke — Sun, 07 Jun 2020 18:28:04 +0000

Home automation is easier than ever with a plethora of IoT-connected devices enabling everyday appliances in your home to be internet-enabled. When you write a custom piece of integration for your IoT, often times deployment to the cloud becomes an afterthought, only to become a nightmare when you actually want to update it later, and you don't remember where/how you deployed it. With Pulumi, you don't have to worry about that anymore. You can develop your IoT integration app, as well as your program your infrastructure as an app.

The Garage Door Opener

Most people have an internet hub connected to their automatic garage door opener, which allows them to remotely monitor the garage door as well as open/close it using a mobile app. But what about when you forget to close it and it stays open? Neither the app nor the existing home automation recipes on the home automation website IFTTT have a way to remind you that you left it open. To solve this problem I tried not to build something of my own, but instead try to use Zapier, a task automation platform.

cloudy-sky-software / GarageDoorMonitor

Use IFTTT and Azure Durable Functions to monitor your garage door status.

The First Attempt

My first attempt involved using Zapier. It would have worked if there was a way to update a state while waiting for a timer to fire. I used IFTTT to connect the myQ service to fire a Webhook request each time the garage door opened or closed. The webhook receiver was a Zapier webhook "catch" action, which I then connected to a timer delay before sending me a text message via Twilio. It mostly worked, except, if I closed the garage door before the timer fires, there was no way for me to update the state and therefore, cancel sending the text message.

Here's the "zap" I ended up creating:

A Zap on Zapier using built-in actions.

Durable Functions on Azure Functions

Durable Functions is an extension to the already popular Azure Functions platform. This means, you can write functions with an external trigger (HTTP, Queue etc.) and have it trigger an orchestration. Each orchestration instance is automatically tracked by the platform. Checkout the API reference to see what you can control about an orchestration instance.

Durable Function Types

There are other durable function types, learn more about them here. The following are just the types used in this project.

Orchestration Functions

Each function has a trigger type that identifies how that function can be triggered. Orchestration functions are no different. Orchestration functions typically don't do any work other than, you guessed it, orchestrate other functions that do the work.

Activity Functions

Activity functions are responsible for most of the work in an orchestration. You can make HTTP calls, call other activity functions etc.

Entity Functions

Entity functions allow you to represent your orchestration instance with a state. It is up to you on whether each orchestration instance has its own entity or if your state is a singleton. This is controlled by the way that entities are persisted. Each entity is made up of two components:

An entity name: a name that identifies the type of the entity (for example, "Counter").
An entity key: a string that uniquely identifies the entity among all other entities of the same name (for example, a GUID).

IFTTT + Azure Durable Functions + Twilio + Pulumi

This is the high-level view of the solution I finally ended up with.

IFTTT receives signals from the garage door opener.
IFTTT then calls the function app.
The function app waits for couple of minutes and if the door still isn't closed by then, sends a text message using Twilio.

Function App

The only external trigger in the function app is the HTTP trigger used in this function. An orchestration instance is created only through the orchestration client, injected into the HTTP-triggered function as a param.

[FunctionName("Main")]
public static async Task RunOrchestrator(
    [OrchestrationTrigger] IDurableOrchestrationContext ctx,
    ILogger log)
{
    var delay = Environment.GetEnvironmentVariable("TimerDelayMinutes");
    var delayTimeSpan = TimeSpan.FromMinutes(Convert.ToInt32(delay));

    var maxRetries = 10;
    var count = 0;
    // The following loop will execute for as long as the garage door remains open
    // up to max number of retries.
    while(true)
    {
        DateTime timer = ctx.CurrentUtcDateTime.Add(delayTimeSpan);
        log.LogInformation($"Setting timer to expire at {timer.ToLocalTime().ToString()}");
        await ctx.CreateTimer(timer, CancellationToken.None);
        log.LogInformation("Timer fired!");

        try
        {
            using (await ctx.LockAsync(EntityId))
            {
                log.LogInformation("Entity lock acquired.");
                var currentState = await ctx.CallEntityAsync<string>(EntityId, "read", null);
                log.LogInformation($"Current state is {currentState}.");
                // If the door is closed already, then don't do anything.
                if (currentState.ToLowerInvariant() == "closed")
                {
                    log.LogInformation("Looks like the door was already closed. Will skip sending text message.");
                    break;
                }
                    await ctx.CallActivityAsync("SendTextMessage", null);
                }
            }
        catch (LockingRulesViolationException ex)
        {
            log.LogError(ex, "Failed to lock/call the entity.");
        }
        catch (Exception ex)
        {
            log.LogError(ex, "Unexpected exception occurred.");
        }

        if (count >= maxRetries)
        {
            log.LogInformation("Reached max retry count, but the garage door is still open. Will stop checking now.");
            break;
        }
        log.LogInformation("Door is still open. Will schedule another timer to check again.");
        count++;
    }
}

Deploying the Infrastructure using Pulumi

We will use Pulumi to deploy our function app. The Pulumi app creates the function app along with the Key Vault containing the Twilio account token necessary for the API call to send a text message. For more information, see the README file in the infrastructure folder of the source repo.

Once the Pulumi app is deployed, you can get the URL for your function app, in order to complete the IFTTT Applet creation in the next step.

IFTTT Applets

IFTTT allows you to create custom applets, which is basically creating your own recipe of "this" and "that". To create a new applet, click your avatar in the top-right corner on https://ifttt.com, and click Create.

Click on + This and choose the service called myQ. Most of the garage door openers here in the USA are made by the Chamberlain Group anyway and you are most likely using one of those. All of those openers work with the myQ internet gateway. Your alternative would be to buy a myQ Smart Hub.

Click + That and search for Webhook to select it. You will need the URL of the Function App that was deployed using Pulumi. You can get this URL by navigating to https://portal.azure.com, too. Since the infrastructure was deployed using Pulumi, we can easily fetch its output by running pulumi stack output webhookUrl in the infrastructure folder. We can now complete the Webhook action's configuration in IFTTT.

Note: Since the function app is exposed to the internet, we don't want just about anyone to be able to call it. Instead, we will use the built-in function app authorization keys to allow only IFTTT to invoke it. Any other caller without the function key will receive a 401 Unauthorized error due to this auth requirement.

Completing the IFTTT applet creation for webhook action.

Twilio

In order to send a text message, create an account on Twilio and purchase a Programmable SMS number. Your account SID and token can be found on the dashboard page of the Twilio Console or on the Settings page under the API Credentials section.

Final Notes

A few things important things to note:

The KeyVault in the infrastructure is not necessary for a project like this, but it is very easy to create one with Pulumi. And with Azure's new Managed Identity, it is even easier to configure application access to secrets.
To learn more about security best practices on Azure, read this excellent post by Mikhail Shilkov.

In the next post, we will take a closer look at the Pulumi app used to deploy the Azure Function App.