DEV Community: Danny Perez

Passwordless sign-in with Google One Tap for Web

Danny Perez — Mon, 26 Jul 2021 02:52:31 +0000

I sign in with my Google account everywhere I can to avoid having yet-another-password on another random website. I've been seeing an upgraded experience on some sites (maybe I'm just noticing now) like Trello/Medium where you can sign-in with Google with one click on the page without getting redirected. Turns out its called One Tap for Web and its Google's passwordless sign-in option and you can use it on your own website. I took it for a spin and set it up on a hello world React example and here's what I found, warts and all.

If you'd like to watch a video version of this tutorial, you can check it out on my YouTube channel here.

Create a new react app and add a sign-in state

Start with a bare react app... npx create-react-app one-tap-demo

For this example app, I'll be using the JS library to initialize the one-tap prompt. The reference guide shows you how to add it using mostly HTML, but if you're using a front-end framework, its easier to use just JS to configure it.

Add some state to keep track of when the user has signed in

function App() {
+   const [isSignedIn, setIsSignedIn] = useState(false)
    return (
        <div className="App">
            <header className="App-header">
                <img src={logo} className="App-logo" alt="logo" />
+                { isSignedIn ? <div>You are signed in</div> : <div>You are not signed in</div>}
            </header>
        </div>
    )
}

Add the GSI

Add the Google Sign-In library (also called GSI) dynamically when your application starts

function App() {
    const [isSignedIn, setIsSignedIn] = useState(false)

+     const initializeGsi = () => {
+        google.accounts.id.initialize({
+            client_id: 'insert-your-client-id-here',
+        });
+        google.accounts.id.prompt(notification => {
+            console.log(notification)
+        });
+ }
+
+    useEffect(() => {
+        const script = document.createElement('script')
+        script.src = 'https://accounts.google.com/gsi/client'
+        script.onload = initializeGSI()
+        script.async = true;
+        document.querySelector('body').appendChild(script)
+    }, [])

}

There's 2 API calls - one to configure the library, and one to show the user the prompt (after the library has been configured). To see all possible configuration options, check the reference here.

I added it as a useEffect hook with [] as an arg, so that it only runs once after the first render. ref

When you refresh the page, if you got it all right the first time, you'll see the prompt.

Get a Client ID from your Google Developer Console

Follow this guide for adding your Get your Google API client ID

When I refreshed the page after following the steps and adding my client id to the app, I got this error message:

[GSI] Origin is not an authorized javascript origin

For me, it meant that my Google OAuth Client ID is misconfigured. I missed the "Key Point" on the official walkthrough - which only applies if we're using localhost as our domain.

Confirm that your site URL (i.e. http://localhost:3000) is added as both an authorized Javascript origin and as a valid redirect URI, in the Google OAuth Client Console.
IMPORTANT You ALSO need to add http://localhost as an authorized Javascript origin. This only seems necessary in development when you might be using a different port in your URL (we are).

Using the sign-in button

Now when you refresh the page, it should be working, and you should see the prompt. If you don't see the prompt, check your developer console for errors, and if that doesn't help - see the section on Debugging below.

IF THIS IS YOUR FIRST TIME DOING THIS, DO NOT CLICK THE X BUTTON ON THE PROMPT BEFORE READING THIS WARNING!

WARNING 1: Clicking the X button on the one-tap prompt dismisses the prompt. If you refresh the page after this, you won't see the button come back. Why?

The One Tap library has some additional side effects around dismissing the prompt. If you've clicked the X button, a cookie has been added to your domain called g_state. Here's a screenshot of where you can find it - if you clear that cookie value, you'll see the prompt come back.

WARNING 2: Clicking the X button more than once will enter you into an Exponential Cool Down mode - see the reference here. What does that mean?

You cannot clear cookies or use an incognito window to get around it (at least I couldn't). It seems to be based on your browser and website (maybe IP?), its not clear. If you accidentally run into this, time to take a break. Or try a different browser/website URL.
After I dismissed it, I couldn't see it for 10-15 minutes, although the table on the developer guide suggests I wouldn't see it for 2 hours. In any case, its an annoying thing to have to run into during development.

Debugging issues with One Tap sign-in

The developer guide suggests this as example code for your prompt. But it flys over an important detail that the reason WHY your prompt did not display, or got skipped, or got dismissed is also in the notification object.

google.accounts.id.prompt(notification => {
      if (notification.isNotDisplayed() || notification.isSkippedMoment()) {
                  // continue with another identity provider.
      }
})

There are 3 types of notification moments - display, skipped, dismissed and each one has their own list of possible reasons, with its own API call to figure it out - see here for the full list. If you're having problems with the button and you don't know why, it might be helpful to use the snippet below to see what those reasons look like:

google.accounts.id.prompt(notification => {
+      if (notification.isNotDisplayed()) {
+          console.log(notification.getNotDisplayedReason())
+      } else if (notification.isSkippedMoment()) {
+          console.log(notification.getSkippedReason())
+      } else if(notification.isDismissedMoment()) {
+          console.log(notification.getDismissedReason())
+      }
        // continue with another identity provider.
    });

One of the reasons you might see is opt_out_or_no_session. This can mean that

A. your user has "opted out" of the prompt by dismissing it. You can try clearing the g_state cookie that might be on your domain, if you dismissed it by accident
B. your user does not have a current Google session in your current browser session.
- While this is a passwordless sign-on via Google, it does require you to have signed into Google (presumably with a password) at some earlier point.
- If you are using an Incognito window, make sure you sign in to Google within that window.

Now that your user has signed in

Once you have selected your account and signed in with no errors, its time to hook it into your React app. If you've used the Google Sign-In for Websites library before (see here for my guide on setting it up), there's an API to let you get at the user's info. But with the One Tap Sign-In for Web library, you only get the users id token (aka their JWT token).

That means you need to decode the ID token to get the users info. We can do that by adding the jwt-decode library with npm install --save jwt-decode

To do all this, add a callback to your initialize block:

+ import jwt_decode from 'jwt-decode'

function App() {
    const [isSignedIn, setIsSignedIn] = useState(false)
+   const [userInfo, setUserInfo] = useState(null)
+   const onOneTapSignedIn(response => {
+       setIsSignedIn(true)
+       const decodedToken = jwt_decode(response.credential)
+       setUserInfo({...decodedToken})
+   })

     const initializeGsi = () => {
        google.accounts.id.initialize({
            client_id: 'insert-your-client-id-here',
+            callback: onOneTapSignedIn
        });
        ...
 }
    ...
    return (
        <div className="App">
            <header className="App-header">
                <img src={logo} className="App-logo" alt="logo" />
+               { isSignedIn ?
+                   <div>Hello {userInfo.name} ({userInfo.email})</div> :
+                   <div>You are not signed in</div>
+               }
            </header>
        </div>
    )
}

To see all the user info that is available to you, see the dev guide here

Signing out

The docs suggest adding a <div id="g_id_signout"></div> to your page, but its not clear if the library is supposed to create a sign-out button for you. I think the answer is no, because I tried it, and nothing happens.

My current theory is that signing-out is meant to be left to your own application, and can be as easy as refreshing the page.

In this post, I'm only using the One-Tap button on the front-end since I have no login system myself. Whenever you refresh the page, you'll see the prompt even if you just finished signing in.
If I wanted to integrate this with an existing login system, "sign-out" would mean signing out of my own application (and not out of my google account)
This flow works out as long as you dont enable the auto sign-in option.

+ const signout = () => {
+     // refresh the page
+     window.location.reload();
+ }

return (
    <div className="App">
        <header className="App-header">
            <img src={logo} className="App-logo" alt="logo" />
            { isSignedIn ?
+               <div>
+                   Hello {userInfo.name} ({userInfo.email})
+                   <div className="g_id_signout"
+                       onClick={() => signout()}>
+                       Sign Out
+                    </div>
               </div> :
               <div>You are not signed in</div>
            }
        </header>
    </div>
)

Limitations of the One Tap Sign-In button

Also lost in the developer guide is this comment for the prompt snippet // continue with another identity provider. which brings us to the section on limitations of this sign-in prompt.

The One Tap Sign-In button only works on Chrome & Firefox across Android, iOS, macOS, Linux, Window 10. If you have users on Safari or Edge, they won't see the prompt. When I try it, I get a Not Displayed error with reason opt_out_or_no_session 🤷
If your users accidentally dismiss the prompt (see the warning above if you missed it the first time), they will also see opt_out_or_no_session as the Not Displayed reason, and they will be unable to sign-in.
The library (and the interface itself) is different from the Google Sign-In for Web library. The One Tap library uses google.accounts.id.initialize() to initialize the app, and the other one uses gapi.auth2.init() - That seems like a missed opportunity to put both login systems behind the same interface.
There's no sign out button - the snippet thats mentioned in the docs doesn't seem to do anything. My guess is that a sign-out button could mean refreshing the page which would cause the prompt to appear again, effectively signing you out.

It's highlighted on the main page of the dev docs, but you can't use this library alone. I did it here for the purposes of a hello world example, but its meant to be an upgrade to your login experience.

Try it out

I've pushed this sample code to github available on intricatecloud/google-one-tap-web-demo. Instructions to run the demo in the README.

It's worth taking a look at how some of these other sites have implemented the login workflow. Sign in to Google in your current browser session, and then visit medium.com to see the prompt in action.

Avoiding burnout as a new tech lead

Danny Perez — Tue, 28 Jan 2020 14:03:19 +0000

Does this sound familiar? The best dev on the team is promoted to being a tech lead, despite not having any management experience. That dev then all of a sudden has to deal with:

Angela, the senior dev who broke up with her partner and is emotionally devastated and needs to take a few days
Oscar, the new dev, saying he needs a few more days to refactor a logging module that needs a "cleaner" interface.
Kevin the PM who keeps making these damn status meetings
Michael, the sales guy who keeps promising features that don't exist yet

You spend all your time drowning in problems, trying to keep everyone happy. You found out that multiple teams were depending on your work only after it all fell behind. People start looking to you for answers, and you don't have any. You stopped caring. You're burnt out. I've been there myself. 👋 hi.

Almost as if the presence of programming skills also indicated that the person can manage a project with many other people. There are, at least, a good chunk of people (developers included) who honestly believe this. It sounds wrong, doesn't it?

How I started as a lead

First time I became a lead, I had no prep other than being a teammate and so my only goal for my first few months was to do everything the last person did - like I imagine the person before them did, and the person before them... I also started reading everything I could about being a tech lead - check out my list here

I also decided to keep taking on the same amount of work I usually did, in addition to my tech lead responsibilities. Sound familiar? :raises_hand: I

They were in a lot of meetings, so I felt I had to be in a lot of meetings. That blew apart my schedule for the week because I couldn't work on some relatively important things, and so schedules fell behind quickly after that. I would stay late at the office (seemed to be common among tech leads or maybe just consultants), and spend long evenings catching up on extra work after people started leaving for the day. In my bachelor days, this was fine - I had no other responsibilities and coding was fun - thats the excuse anyway.

All this meant that my days were filled with meetings and other "administrivia", my evening/nights were filled with coding, and I started burning myself out over time.

Anyway, long story short.

Here's a couple tips to help you avoid burnout as a new lead

1. Plan your time carefully

Block out time on your calendar to manage your week, set hard boundaries between when you start/end the day (especially if you work remotely) and stick to it.

You probably don't need to attend every meeting thats on your calendar. But if for some reason, you are required, depending on the meeting, I'll multi-task and be a "fly on the wall" and just listen, other times I'll have to be an active participant.

If you need to set aside a day to do some hands-on dev work, put it in the calendar and decline everything else. Unless you're working on life/death scenarios, it could probably wait until later. Some people have No Meeting Thursdays, Code Review Afternoons... Call it whatever you want.

When performance review season rolls around, its even more helpful to block out time to prepare everyone's review. I've tried to write all my reviews the night before in one week for my team, and it wound up taking me HOURS for each person as I tried to remember everything they did.

2. Stay out of the critical path

The critical path in a project is the set of tasks that NEED to be accomplished in order for something to work. For example - if you're building a feature that adds a button to your site, adding a linter or a CSS theme to your project is not required for that button to work. The button itself is obviously the important work.

As a tech lead, you gotta stop picking up any of that important work. If its a time sensitive thing, and you aren't able to get it done in time, it'll have some downstream consequences, and start blocking other projects from getting done. Thats what your team is for - they can go heads down on the work.

But what if some of your team doesn't know how to make changes to your CI/CD pipeline, or the front-end dev hasn't worked on backend SQL queries? Its on you as the lead to support the team to do the work - like that time we hired a junior dev who knew python and just a little javascript to do front-end work in an Angular app. He had never worked with a full-blown framework with all the bells and whistles before, but we had bugs to fix and no time to kill, and sure enough, with some training and mentoring, he got up to speed and started cranking through bugs.

Now, its important you stay technically involved at some level, but your focus will be on the big picture, like making sure that important features are getting shipped, and less on the small details, like why is [1,2,3].flatMap throwing an error.

Want to make your week easier? Try these to start.

start with blocking out a 4 hour chunk one day a week for heads-down coding time and decline everything. See how it goes and you can adjust depending on your workload.
Take a look at upcoming work that you would usually do like adding a new API endpoint to a backend, and offer to pair with another dev so that they can learn how to do that work.

If you're a new tech lead, definitely check out some of the 9 books that helped me navigate my first time being a tech lead.

I'd like to hear from you - what were some of the things that frustrated you when you first became a tech lead?

Setting up automatic failover for your static websites hosted on S3

Danny Perez — Wed, 16 Oct 2019 22:57:56 +0000

Up until late last year, you couldn't set up automatic failover in CloudFront to look at a different bucket. You could only have one Primary origin at a time, which meant that you needed additional scripts to monitor and trigger failover. (It was one of the gotcha's I had written about previously about using CloudFront)

Then AWS dropped the bombshell during re:invent 2018 that Cloudfront can now support it by using Origin Groups - Announcement

In this post, I'll show how you can configure this for an existing site via the console.

Adding an origin group to your Cloudfront distribution

I was working on this for one of my websites - gotothat.link - its not a static website, but its just a site that serves a bunch of redirects so all the same rules apply.

Step 1 - Configure replication on your S3 bucket

Go to the replication settings page for your S3 bucket.

If you haven't yet enabled Versioning, it will tell you that you're required to do so.

Enable replication for the entire bucket by clicking Add Rule. For our use case, we don't need to encrypt the replicated contents.

You can follow the prompt to create a new bucket in us-west-1 - N. California.

Once you've created the S3 bucket, make sure to update the bucket policy on your new replication bucket to allow the static website hosting to work:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::gotothat.link-west-1/*"
        }
    ]
}

For the IAM role, I'll let the console create one for me - this is whats generated automatically for you

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:Get*",
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::gotothat.link",
                "arn:aws:s3:::gotothat.link/*"
            ]
        },
        {
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags",
                "s3:GetObjectVersionTagging"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::gotothat.link-west-1/*"
        }
    ]
}

Once you click Save, you should see this panel appear now when you're on the Management page of your S3 bucket.

Step 2: Replicate data

Replication is now enabled. But we're not quite done.

When you enable replication, any objects created/modified from that point forward are then replicated to your target bucket. Any existing objects are NOT automatically transferred over - you have to do a one-time sync when you first set it up.

If all you have in your bucket is your website files and nothing else, then you can do an aws s3 sync s3://source.bucket s3://target.bucket to import all your existing objects. Depending on how much is in there, it can take awhile.

If you have some objects in your source S3 bucket that require you use the Website-Redirect-Location metadata (this is needed if you want to serve redirects out of your domain like this):

You cannot do a sync with the replicated bucket using something like aws s3 sync s3://my-bucket-name s3://my-bucket-name-us-west-1. While the sync may succeed, your redirects will not. This is annoying.
The S3 docs state that Website-Redirect-Location isn't copied over when using a sync command, although it doesn't specify why. See docs here
However, Objects that are automatically replicated will keep their Website-Redirect-Location metadata. But any files that are aws s3 sync'd between 2 buckets will not have that metadata.

Last step is to go to the replicated bucket, select Properties, and enable Versioning + Static Website Hosting for it. If you forget this, you won't receive the 301 redirect.

At this point, you should have a bucket in another region, with all the same contents as your primary bucket, with static website hosting enabled.

Configure Failover in CloudFront

Create a new origin with an origin id like gotothatlink-failover-prod and use the URL of the form gotothat.link-west-1.s3-website-us-west-1.amazonaws.com

Create a new origin group and add your two origins at the prompt. The primary origin will be our original S3 bucket, and the secondary origin will be the new failover origin we created.

For the failover criteria, we can check all the 5xx status codes since that would imply issues happening in S3. You can check 404s if you want to protect against someone accidentally deleting all your data, or 403 Forbidden errors to protect against accidental bucket policy changes.

Next, go to the Behaviors tab, and make sure to update the behavior to route all paths to your Origin Group (and not your Origin). Otherwise, even if it fails over, it will still send requests to your primary origin.

After you've set these changes, you'll have to wait for the CloudFront distribution to finish updating (could take up to a half hour).

Test it by "accidentally" trashing your primary bucket

Since I've enabled failover on 404 errors, I'll run a test where I delete everything in my bucket. I ran aws s3 rm --recursive s3://gotothat.link/.

And now to save you boatloads of time - here is every error I ran into when setting this up.

Troubleshooting:

You get a 404 - Cloudfront is still looking at your primary bucket. Turns out I hadn't updated the Behaviors for the CloudFront distribution.
You get a 403 - Your replicated bucket has incorrect permissions. See the section above labeled "Configure replication on your S3 bucket" to see the required bucket policy.
You now get a 200 - your object doesn't have the Website-Redirect-Location metadata associated with it
- Take a look at the replicated object by doing the following - note that it includes WebsiteRedirectLocation
- Check that you haven't done an aws s3 sync to import your redirects to your bucket.

☁  intricate-web  aws s3api get-object --bucket gotothat.link-west-1 --key newvideo  newvideo-1
{
    "AcceptRanges": "bytes",
    "LastModified": "Sun, 22 Sep 2019 05:21:08 GMT",
    "ContentLength": 0,
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
    "VersionId": "dynNjEGPeE3.z6kCCp.zgvVlbYkA.h3X",
    "ContentType": "binary/octet-stream",
    "WebsiteRedirectLocation": "https://youtube.com/video",
    "Metadata": {},
    "ReplicationStatus": "REPLICA"
}

Wrapping up

If you're doing it via the console, this doesn't take long to set up. Its a handful of steps that you can do in < 30 mins that will give you extra confidence that your site is backed up, and still functional even if S3 goes down (it usually has a high-profile outage at least once a year).

With this solution in place, you have an HTTPS website served from a global CDN with automatic failover in case of an S3 outage or data loss in your primary bucket so that you don't need to take any downtime. All this for just a few bucks a month. :mindblown:

Create your own short link redirects using CloudFront + S3

Danny Perez — Tue, 01 Oct 2019 19:14:17 +0000

Get branded short links under your own domain for dirt-cheap by self-hosting it on AWS using CloudFront + S3. It's a no-code server-less solution to get redirects on the cheap and in this guide, I'll detail how to set it up. If you're interested in some good reasons why - check out my first post on it here.

In this guide, I'll go over

what do our redirects/short links need to be able to do
how to set it up via the console if thats what you're comfortable with, or doing it via the AWS node SDK.
How to check that its working
A few things to check when it doesn't (it never really works the first time, doesn't it?)

What do we need out of our short links?

I need to be able to create custom links to redirect
I need to be able to change them later
I want to see traffic to the redirect, and how many times they've been clicked (this will be part of another post)

Technical Requirements:

Highly Available, Redundant (99.9% SLA)
Serve HTTPS 301 redirects
Serverless. Cloudfront + S3 means we don't have any servers to manage.

This is the architecture we're going to set up on AWS:

In short, we have...

a CloudFront CDN which gives us free HTTPS for our domain (courtesy of AWS Certificate Manager)
An S3 bucket with Cross-Replication enabled to another bucket in another region
Each redirect is stored as an object with the Website-Redirect-Location metadata pointing to our redirect

Setting it up via the console

Step 1: Set up Cloudfront + S3

I've set up this diagram for one of my domains - gotothat.link so I'll show you how I've set that up.

I own gotothat.link using Route53 as my registrar. You don't need to have your domain on AWS Route53 if you want to host your short links on AWS - if you have one already via something like GoDaddy or Namecheap, thats fine too, you can use that.

Cloudfront will be providing an HTTPS URL that we can use using S3 as an origin which will serve our redirects. Cloudfront will have its logs piped to another bucket, and we can get some metrics out of that.

An S3 bucket will be used to store the actual redirects.

For an in-depth guide on setting up those 2 services, you can check out my post which walks you through setting up via the console or if you want to set it up with terraform.

Once you have Cloudfront deployed and connected to your S3 bucket, we can add our redirects.

Step 2: Add a redirect

Create an empty file and upload it to the console, with Website-Redirect-Location metadata pointing to the URL you want to redirect to. In this case, I've uploaded an empty file named not-a-rick-roll and added the metadata to point to a particular YouTube video.

You can use the default permissions, default storage type, and no need for encryption.

In the screenshot, you'll see theres a warning under the Metadata header - You cannot modify object metadata after it is uploaded. I'm sure theres a reason this is included, but I've been able to just go into the object's metadata and change the values from the screen below.

By default, S3 will add a Content-Type metadata to your object when you upload it. So you can see in the screenshot that the Content-Type has been set to binary/octet-stream which implies this link will return some file. Its worth noting in this case, that by adding the Website-Redirect-Location metadata to your object, S3 will serve up a HTTP 301 Redirect to your target redirect - a 301 response only needs to include a Location header specifying the redirect. So it doesn't quite matter what the Content-Type is on your object. You can leave it alone, or delete it.

Thats it! Assuming you have many more of these, you can even automate the creation of your redirects.

Step 3: Automating it

If you use the CLI, you can script it using the AWS CLI - aws s3 cp not-a-rick-roll s3://gotothat.link/not-a-rick-roll --website-redirect "https://www.youtube.com/watch?v=dQw4w9WgXcQ"

If you have your redirects stored in a CSV, you can parse it and upload it using the node aws-sdk - check out this example:

# redirects.csv

notarickroll,https://youtube.com/video
no/really/not/a/rickroll/video, https://youtube.com/video

// redirects.js

const parse = require('csv-parse/lib/sync')
const fs = require('fs')
const AWS = require('aws-sdk')
const s3 = new AWS.S3();

const redirects = parse(fs.readFileSync('redirects.csv'))
redirects.forEach((redirect) => {
    console.log('Uploading ', redirect)
    s3.putObject({
        Body: '',
        Bucket: 'gotothat.link',
        Key: redirect[0],
        Metadata: {
            'Website-Redirect-Location': redirect[1]
        }
    }, (err, data) => {
        if (err) return console.log(err)
        console.log('done')
    })
})

Verify your redirects

You can check to see that its working by using good ol' curl to make a request to your site - gotothat.link, and verify that you get back a 301 redirect for that object. This is what the output of that command looks like:

☁  intricate-web  curl -i https://gotothat.link/not-a-rick-roll
HTTP/2 301
content-length: 0
location: https://www.youtube.com/watch?v=dQw4w9WgXcQ
date: Sun, 22 Sep 2019 01:52:57 GMT
server: AmazonS3
x-cache: Miss from cloudfront
via: 1.1 96b6c9282feceea8aa00c25902322bb6.cloudfront.net (CloudFront)
x-amz-cf-pop: EWR53-C1
x-amz-cf-id: rNwzh3rL-ErWpi0YuDrqQcCvvmbqa_ZWj7cPl71WrPPi1vlFbW0siA==

You now have URL redirects served out of your own domain so you can do things like gotothat.link/not-a-rick-roll (check it out - its totally not a rick roll 😎)

Troubleshooting

If you change a redirect and you don't see it taking effect, there might be one of two things happening:

If you're trying this in your browser, you might have a cached version of the redirect. Open an incognito window and try it there.
Cloudfront is still sending you back a cached version of the redirect. You'll have to use curl to inspect the request so you can see whats happening.

☁  intricate-web  curl -i https://gotothat.link/not-a-rick-roll
HTTP/2 301
content-length: 0
location: https://www.youtube.com/watch?v=dQw4w9WgXcQ
date: Sun, 22 Sep 2019 01:52:57 GMT
server: AmazonS3
x-cache: Hit from cloudfront
via: 1.1 c67ae9899d89f9402837da3a0ead9442.cloudfront.net (CloudFront)
x-amz-cf-pop: EWR53-C1
x-amz-cf-id: 7pDL0XgVjGmdMqcHf7tSsRILd0WfZpihpLF66SYVJLlzqfCJG2PMRg==
age: 4

In this example, we want to pay attention to the x-cache & age response header. In this case, it says Hit from cloudfront and the age == 4. The age is the time that object has been in the cache. If you've updated your redirect recently, and you're still seeing an old redirect in your response headers, then you'll need to invalidate your Cloudfront cache. You can do that from the Invalidations page

You'll know it updated when the age response header shows something recent.

Wrapping up

There you have it! By this point, you should have your own domain sending you back redirects that you can monitor yourself. Now, this isn't the whole story because theres still a feature or two that we need to set up. So two posts coming up next:

how to do this if you use an infrastructure-as-code tool like terraform
how to tell when specific links are getting clicked by monitoring your CloudFront logs

If you try out this setup, leave me a ❤️ or a comment to let me know how it went - feedback welcome! Stay tuned for the next post and follow me here on dev.to 😄

That time I needed to create 20k short links, and how to create them on AWS

Danny Perez — Mon, 09 Sep 2019 20:16:27 +0000

This problem landed on my lap awhile ago, and I've seen it come up time and time again so I wanted to share how we went about it. Our marketing team was submitting one of our products for a review for a client, it has a ton of content. There was a long list of deep-links in the product that we wanted the reviewers to see, but we needed to create short links so they could embed them on some doc.

There wound up being a total of 20k short links, which meant that:

I needed to be able to upload 20,000 in one go, it needed a scriptable API
whatever we used needed to be around for at least 6 months to be available during a certain review period
the links needed to work, and be available
we needed to be able to create the redirects from a Google Sheet
they wanted to know if/when links were clicked

We ultimately landed on a solution with just AWS that comes out dirt cheap - but there were a few things we had to consider first.

Why not just use something like bit.ly?

For this specific case - based on their pricing, we'd be in the enterprise range under the "Contact Us for a Quote" plan which puts us north of $400/year (and a "quick call" with an account manager) for something I only need once.

$400/year sounds steep for something that ...should.... be simple and cheap. 301 redirects have been around since 1999 (20 years ago!) - why isnt this a solved problem! (it kinda is, but it isnt)

Even outside of enterprisey considerations,

bitly comes with its own limits for the free tier, and even doing something like bulk redirects needs to be on their enterprise plan.
30/month for their basic plan is a bit steep for some 301s, and we can do it for cheaper. (you trade off some features though, primarily analytics - if you dont care about that, fantastic!)

Apart from that, its a good opportunity to learn the ins & outs of AWS, it doesnt involve a lot of services. If you're familiar with Cloudformation or Terraform, you can set it all up that way.

How do you build it?

We worked out a solution using Cloudfront + S3, where your Objects have a Website-Redirect-Location metadata attached (see docs here). We built it out in an afternoon, and had all 30k redirects generated and working, and it costs just a few bucks/month to run.

Let's say you own short.url and you want to be able to have short.url/2019barbecue redirect to a Facebook page.

At a high level, take a look at this architecture (images courtesy of cloudcraft.co):

You point your domain (which can be hosted anywhere, doesnt need to be route53) at a Cloudfront distribution that sits in front an S3 bucket served as a website. In that bucket are many objects, with the Website Redirect Location metadata set to redirect to our target url - short.url/2019barbecue redirects to example.com

Are your redirects business critical?

Maybe you're running a launch and you need people to click a link. You are now responsible for uptime and so AWS' uptime is your uptime.

In this case, your availability is the minimum of either Cloudfront or S3 (99.9%) - you do, however, have the benefit of a globally distributed CDN… and a highly durable and replicated object store… with failover to another region in case it craps out - not to mention its cheap-ish to implement.

Do you want analytics?

Bitly analytics are pretty neat - thats a tradeoff since its something you'll need to do yourself - but if you already have an analytics/monitoring platform, you can get your data through there. For things on my personal site, I don’t care for seeing analytics - i just want to share a customized short link.

Out of the box, you get some level of Cloudwatch monitoring around Cloudfront including error rates, data in/out, and total requests. (full list here)

If you want more info, then you need to dump Cloudfront access logs to another S3 bucket and use Athena (the docs for that here) for some quick analytics to see things like source IP, referer, requested path, response code

Any gotchas i should be aware of?

Be mindful of updating where a redirect points to. 301 redirects CAN be cached indefinitely by your browser since the spec indicates a 301 means Permanent Redirect.

If someone has already clicked your link before, it's possible that they may be taken to the old location for some period of time. In practice, I've seen a long tail of requests to the old value of the redirect kinda like this:

302 redirects are meant to be temporarily cached, so it has less issues. Some architectures limit you to 301 redirects, others give you the option to do a 302. In this case, Cloudfront + S3 only lets you do 301s and thats just a constraint we need to deal with unless we want to throw Lambda in the mix as well.

Wrapping up

If you find yourself needing to create some customized short links or redirects, consider using Cloudfront + S3 to serve them using the architecture above - you get all the benefits of an enterprise-grade solution at a fraction of the cost if you're already familiar with AWS.

I'm working on a series of posts to thoroughly explain how to set up short links on AWS. Next up, I'll be covering how to set up the short links in the console, and then how you could do it in terraform.

Give the article some ❤️ and sign up to my mailing list here so you don't miss the next post!

adding google sign-in to your webapp - a react example

Danny Perez — Wed, 07 Aug 2019 21:43:10 +0000

In this next part of the series, I'll be walking you through an implementation of google sign-in with a simple react app and a bonus react-router example.

Up until now, we've seen 2 different hello world examples of how to add google sign-in on the front-end - using plain HTML and vanilla JS. It's been all nice and dandy for a hello world, but one thing thats been missing while I was figuring out google sign-in is what a working implementation looks like - especially in React.

*There is a react-google-login component that configures all of google sign-in behind a <GoogleLogin> tag. It's quite useful and I've used it in a few instances - my one complaint is that you can't get at the return value of the gapi.auth2.init() method. This post will show whats going on under the covers if you prefer not to use a library.

creating a new react app with google sign-in

First - create the app create-react-app google-auth-demo. The files we'll mainly be working with are App.js and index.html.

Add the google sign-in script tag to your public/index.html

<head>
  ...
  <script src="https://apis.google.com/js/api.js" async defer></script>
  ...
</head>

add the login button

In App.js - add some state to keep track of when the user has signed in

contructor(props) {
    super(props)
    this.state = {
        isSignedIn: false,
    }
}

Add the button to the component

render() {
  return (
    <div className="App">
        <header className="App-header">
          <img src={logo} className="App-logo" alt="logo" />

          <p>You are not signed in. Click here to sign in.</p>
          <button id="loginButton">Login with Google</button>
        </header>
      </div>
  )
}

Wait, how do I avoid showing this if the user is signed in? We can use the state to conditionally show it.

getContent() {
  if (this.state.isSignedIn) {
    return <p>hello user, you're signed in </p>
  } else {
    return (
      <div>
        <p>You are not signed in. Click here to sign in.</p>
        <button id="loginButton">Login with Google</button>
      </div>
    )
  }

}

render() {
  return (      
    <div className="App">
      <header className="App-header">
        <img src={logo} className="App-logo" alt="logo" />
        <h2>Sample App.</h2>

        {this.getContent()}           
      </header>
    </div>
  );
}

Since conditionals are a little hard to write with inline JSX, I've pulled out the conditional block to another method to provide the component that we want.

At this point, you'll have a button that does nothing (the best type of button) and you'll see the "You are not signed in" message

add sign-in

To finish setting up google sign-in, you'll want to initialize the library using gapi.auth2.init(). A good place to do that is inside of componentDidMount() callback.

componentDidMount() {
  window.gapi.load('auth2', () => {
    this.auth2 = gapi.auth2.init({
      client_id: '260896681708-o8bddcaipuisksuvb5u805vokq0fg2hc.apps.googleusercontent.com',
    })
  })
}

To use the default styling, use the gapi.signin2.render method when initializing your component.

onSuccess() {
  this.setState({
    isSignedIn: true
  })
}

componentDidMount() {
  window.gapi.load('auth2', () => {
    this.auth2 = gapi.auth2.init({
      client_id: 'YOUR_CLIENT_ID.apps.googleusercontent.com',
    })

    window.gapi.load('signin2', function() {
      // render a sign in button
      // using this method will show Signed In if the user is already signed in
      var opts = {
        width: 200,
        height: 50,
        onSuccess: this.onSuccess.bind(this),
      }
      gapi.signin2.render('loginButton', opts)
    })
  })
}

When using this method, the button will automatically show whether you're signed in, but the onSuccess callback won't actually run unless the user clicks it when it says "Sign In". Otherwise, you are logged in automatically. One way to hook into the end of that auto sign in process is by adding a callback to the promise returned by gapi.auth2.init:

window.gapi.load('auth2', () => {
  this.auth2 = gapi.auth2.init({
    client_id: 'YOUR_CLIENT_ID.apps.googleusercontent.com',
  })

  this.auth2.then(() => {
    this.setState({
      isSignedIn: this.auth2.isSignedIn.get(),
    });
  });
})

making a "protected" route

If you're using react-router and you want to add a "protected" route to your React app, you can hijack the render prop of a <Route>. You can do something like this:

authCheck(props, Component) {
  return this.auth2.isSignedIn.get() ? <Component {...props} /> : <UnauthorizedPage/>

}

render() {
  ...
  <Route path="/home" render={this.authCheck.bind(this, HomePage)}/>
  ...
}

By hooking into the render property on <Route>, you can dynamically define what component will load when you try to access that Route.

This is the strategy employed by the react-private-route project library to make it a little bit easier to write, definitely worth checking out.

conclusion

If you're implementing google sign-in in a React app - check out my github repo intricatecloud/google-sign-in-demo to see all the code above in a working setup.

Throughout this 3-part series, we've covered going from a hello-world example of google sign-in, to using the javascript library to do some hacky things. Now, we've reviewed all the code you need to integrate with the Google Sign-In button.

Sometimes, tutorials like this can be hard to follow, and it just won't click unless you see it. I'm putting together this series as a live-coding walkthrough where you can see all the mistakes I make going along with the tutorial. Sign up to my mailing list here to get notified when it goes live.

adding google sign-in to your webapp - using the js library

Danny Perez — Tue, 23 Jul 2019 22:16:37 +0000

In the first part of the series, we decided to use the Google Sign-In for websites library to allow you to show some info about the user using javascript. We used the default Google Sign-In workflow with their default button.

In this part, we'll be going over how to use the gapi library to configure Sign-In and then actually sign-in the user, as well as a few snippets for handling some common user scenarios.

create your own sign in button

Load the api.js library instead of platform.js 🤷 (not sure why they're different)

Change https://apis.google.com/js/platform.js to https://apis.google.com/js/api.js?onload=onLibraryLoaded

Here, we configure our Sign-In client once the library has loaded using the ?onload=onLibraryLoaded callback that you can provide via the URL. api.js will add a global variable called gapi.

Add a button to your index.html with a button click handler

<button onclick="onSignInClicked()">Sign in with button onClick</button>

Add the following to your script tag in index.html to handle the button click

function onLibraryLoaded() {
    gapi.load('auth2', function() {
        gapi.auth2.init({
            client_id: 'YOUR_CLIENT_ID.apps.googleusercontent.com',
            scope: 'profile'
        })
    })
}

function onSignInClicked() {
    gapi.load('auth2', function() {
        gapi.auth2.signIn().then(function(googleUser) {
          console.log('user signed in')
        }, function(error) {
            console.log('user failed to sign in')
        })
    })
}

We can then access the gapi.auth2 library and initialize it using our client_id from the Developer Console.

how to handle some common user scenarios

listening for when the user has signs in
checking if the user is logged in
getting the users info
only allow users from a particular domain to log in
only allow certain users to log in
hide content until after a user logs in
sending your ID token to a backend (if you have one)

listening for when user signs in

In the above example, we can only run some code after we initialize and sign in the user. But what if you have different parts of the page in different files, each showing something different depending on the user? (this might be the case if you're using components)

class UserAvatarComponent extends React.Component {
    ...
    componentDidMount() {
        gapi.load('auth2', function() {
            gapi.auth2.isSignedIn.listen(function(isSignedIn) {
                console.log('user signed in ', isSignedIn)
                this.setState({status: isSignedIn})
            })
        })
    }    
}

checking if the user is signed in

function isUserSignedIn() {
  gapi.load('auth2', function() {
      var isSignedIn = auth2.isSignedIn.get();
      console.log('is signed in? ', isSigned In)
  })
}

A few things to note about using this function:

The Sign-In library will, by default, sign you in automatically if you've already signed in.
If you refresh the page, even after the user has signed in, then on first laod, you'll get auth2.isSignedIn.get() === false
After the user is signed in automatically (usually takes a sec), then auth2.isSignedIn.get() === true
Depending on how you handle your login UI, your user might see that they are not logged in for a hot second. It's helpful to use the isSignedIn.listen() callback if you want to know the precise moment this happens.

getting the users info

function showCurrentUserInfo() {
  gapi.load('auth2', function() {
      var googleUser = auth2.currentUser.get()
      console.log('users info ', googleUser)
  })
}

only allowing users from a particular domain to login

This is a little bit of a hack and is probably easy to circumvent, but you can use the getHostedDomain() method to get the G-Suite domain the user comes from. If the user does not have a G-Suite domain, then it'll be blank.

function onSignIn(googleUser) {
    if(googleUser.getHostedDomain() !== 'mysite.com') {
        // show a Not Authorized message
    } else {
        // show the users dashboard
    }
}

only allowing certain users to login

This is even more of a hack, but seems to be the only you can do it from within javascript. You really shouldn't. Don't know why I'm including it. The brute method.

function onSignIn(googleUser) {
    var profile = googleUser.getBasicProfile()
    if(profile.getEmail() === 'admin@example.com' ||
       profile.getEmail() === 'client@example.com') {
           // show the user dashboard
    } else {
        // show a Not Authorized message
    }
}

hiding content until after the user logs in

This is also a hack. This is easy to workaround if you fidget with the CSS in your browser, but can work if it fits your use case. The reason this is a bad idea is that in a static website, all of the information thats available in your HTML is available to the user. DO NOT USE THIS if you have actual sensitive information to hide. It is a good candidate for showing your favorite cat pictures.

<body>
...
  <div id="greeting">
    You are not authorized to view this content
  </div>
  <div id="dashboard">
    ...
  </div>
  <script>
    // hide initially
    $('#dashboard').hide()

    function onSignIn(googleUser) {
      setTimeout(function() {
        // show the content
        $('#greeting').hide()
        $('#dashboard').show()
      }, 1000);
    }
  </script>
</body>

send the token to identify the user (not their ID)

If you're making a backend request, you'll want to send the users ID token to your backend as an Authorization header. On your backend, you'd then be able to validate and decode the ID token (see here for examples).

The benefit of the approach is that this ID token (a type of JWT token) is signed by Google, so you know the information hasn't been tampered with.

$.ajax({
  url: 'myapi/example',
  headers: {'Authorization': googleUser.getAuthResponse().id_token)},
})

Conclusion

In this post, we've seen how you can configure the Google Sign-In library via javascript, and use it to do things like get the users info and check if they're signed in (theres some nuances to be aware of with the login flow).

Demo code is available on Github intricatecloud/google-sign-in-demo. Replace YOUR_CLIENT_ID with your client ID, and you'll be able to see the sign in buttons in action.

For the last part of this series, we'll look at some examples of how you might use google sign-in in a React and Angular application. Check it out here.

Originally published on www.intricatecloud.io

adding google sign-in to your web app - pt 1

Danny Perez — Wed, 17 Jul 2019 19:58:59 +0000

Check out my new video where I walk through the code below!

Gotten stuck in a rabbit hole figuring out how to add "Log In with Google" to your web app? In this series, I'll cover:

why you might want to use the Sign-In for Website JS library, and getting started with it
Adding the library via javascript
adding it to an existing project (angular & react)

Their docs give you a decent place to start if you know what you're looking for, but there's a few areas where their docs and other guides online can be confusing. There's also minimal guidance as to how to use it within existing projects.

Am I supposed to be following the Google Identity docs or do I need Google Sign-in for Web??
Wait, why are those 2 guides so different? They even load different libraries!
This should be simple! I'm just trying to load a users avatar!!

The answer to "How do I add Log in with Google?" boils down to 1 question you need to be able to answer: What exactly are you trying to do?

Do you just want to be able to see the users name and picture, maybe their email? Use the Sign-In for Websites library (full walkthrough below).
Do you have your own user database and want to offer Google login as an extra option? This is Federated Login and you'd want to use their OpenID Connect protocol. This is used by platforms like auth0, firebase, oneidentity.
Do you want to be able to interact with the users Google account and do things like see their calendar, or create a Google doc? You'll need to use their OAuth workflow.

For the purposes of this series, I'm going to explore when you should use the Sign-In for Websites library which uses the OpenID Connect protocol (also used by Federated Login systems) - this library can be easily misunderstood, but is still incredibly useful. The goal of this authentication library is to let you identify a user with their Google account. Thats it.

So when should I use it?

Good reasons to use it:

you have some content stored on a backend service that you deliver via an API only after the user has signed in
you only have to support Google/G-Suite users
you dont need to authorize users (e.g. allow some users to be admins)
your site is public
you have a single-page app

Use cases where it's worth considering federated login:

you have pre-existing user content stored on a backend service
you have an internal site that you want to restrict to users from a specific domain. (e.g. only users from @example.com should see it)
you want to prevent people from seeing your page unless they've logged in. The best you can do with this library is show/hide elements on the page if the user logged in, but thats enough if you only load data from an API after a user has logged in.

The library is designed to be used with HTML/JS and only interacts with your page via the "Sign in with Google" button. You can integrate this with other frameworks like Angular/React which I'll be covering in the next part of this series.

Adding Google Sign-in step-by-step

1. Hello World HTML

To start with, all you need is an index.html file.

<!DOCTYPE html>
<html>

<head>
  <title>Google Auth Demo</title>
</head>
<body>
  <h1>Welcome to the Demo</h1>
</body>
</html>

2. Add the Google Sign-In Button

Before adding in the button, you first need to create a client ID/secret which is a way of identifying that you, the developer, are allowing users to get the identity information from Google and delivered to your website.

Creating credentials for Sign-In

Go to the Google API Console - https://console.developers.google.com/apis/dashboard
Create a new project, or use an existing project if you already have one set up.
Then click on Credentials -> Create Credentials -> OAuth Client ID

Here's what I put in for this demo:

Name: google-auth-demo
Authorized Javascript Origins: http://localhost:8080
Authorized Redirect URIs: empty

*A note about the Javascript origins: if you have a plain HTML file that you load in your browser using a file path like /home/dperez/index.html, this won't work. You'll need to "serve" your website on your computer so that you have a URL, even if its just localhost. You can use python -m SimpleHTTPServer 8080 (which is commonly available) to serve up your current directory or you can use an npm package like http-server.

You'll then get a Client ID & Client Secret. These are 2 identifiers for your Oauth Client. At this point, you have authorized that Client ID to accept logins and return you the users information. Copy them down. If you accidentally click out of the screen, you can always copy them again later.

Add the library + credentials + button to your HTML

In your index.html page, add the following to your HTML page:

...
<head>
  ...
  <meta name="google-signin-client_id" content="your-client-id-goes-here">
  <script src="https://apis.google.com/js/platform.js" async defer></script>
</head>
<body>
  <h1>Welcome to the Demo</h1>
  <div class="g-signin2"></div>
</body>

The script tag grabs the library from Google that reads your <meta> tag to use as the Client ID, and then automatically re-styles the button with the CSS class .g-signin2.

At this point, if you refresh your page, you should see a pretty Sign-In button. If you click it, you'll go through your Google Login flow in a popup, and you'll finally land back on your site, and the button will say, "Signed In".

Great, we're most of the way there! But in its current form, this is still useless.

3. Identify the user

Update the g-signin2 div to include the data-onsuccess attribute:

<div class="g-signin2" data-onsuccess="onSignIn"></div>

That attribute contains the name of the function that will be called once a user has successfully logged in with Google.

Create a function called "onSignIn".

<body>
  ...
  <script>
    function onSignIn(googleUser) {
      // get user profile information
      console.log(googleUser.getBasicProfile())
    }
  </script>
</body>

Your onSignIn function will be called with an argument containing the information provided by Google. If you refresh the page, you'll notice that

You're automatically signed in
The button gets updated very shortly after refresh (1s delay)

If you open up your javascript console, you'll see the users information printed:

{
  Eea: "108716981921401766503"
  Paa: "https://lh3.googleusercontent.com/-y_ba58pC4us/AAAAAAAAAAI/AAAAAAAAAYE/wMGKOxlWR90/s96-c/photo.jpg"
  U3: "perez.dp@gmail.com"
  ig: "danny perez"
  ofa: "danny"
  wea: "perez"
}

This is the basic profile containing a series of values that identify me as a user. Under any normal circumstances, this object would have fields like name or email, but for some unknown reason (I wish I had the answer but they havent given an answer either), its gibberish - but at least its consistent and hasn't changed.

You can either get the data directly by doing googleUser.getBasicProfile()['U3'] or use the more human-readable approach by using the functions like googleUser.getBasicProfile().getName() or googleUser.getBasicProfile().getEmail(). (See here for the javascript client api reference docs)

4. Sign out

Add a sign out button after the user has signed in by adding a button in your index.html and the click handler in your javascript.

index.html

<body>
...
  <button onclick="signOut()">Sign out</button>
</body>

function signOut() {
  console.log('user signed out')
}

Great! At this point, we've added a Sign-In with Google button and we can identify the user by name/email. Also, thats it. Thats all this library can help you do.

...but what about basic things like saving users data to the backend? or showing an admin page?! Why isn't this in the docs?!

That'll be covered on the next post coming 7/22 - using the js library to add it to your site without HTML.

If you found the post useful, give it a ❤️ and follow me to get notified of the next post.

Using webdriverio with headless Chrome in Docker

Danny Perez — Tue, 07 May 2019 18:05:55 +0000

Using headless chrome for your UI tests works great out of the box on your laptop, but it won’t work out of the box when you're trying to run your tests in Docker. One recent work project was getting webdriverio tests successfully running in a Docker container as part of a Jenkins pipeline. Locally, our tests worked fine, but when we ran them in docker, they became super flaky with errors showing random Chrome crashes, or “Chrome is unreachable” errors. Hope is not lost though - There is in fact a proper incantation of options to pass to Chrome to get it running successfully (and without random crashes).

tl;dr Here's a snippet of wdio.conf.js showing our Chrome configuration. To see why some of these need to be included, read on.

// wdio.conf.js
...
browserName: 'chrome',
chromeOptions: {
  args: [
    '--disable-infobars',
    '--window-size=1280,800',
  ].concat((function() {
    return process.env.HEADLESS_CHROME === '1' ? [
      '--headless',
      '--no-sandbox',
      '--disable-gpu',
      '--disable-setuid-sandbox',
      '--disable-dev-shm-usage'] : [];
  })()),
...

--disable-dev-shm-usage

Add this flag if you're seeing the error (like this guy did) “Session deleted because of page crash from tab crash” in your logs when your tests fail (or a message along the lines of a page crash), or Chrome is unreachable.

Why do we need this?

There’s a tmp folder that Chrome uses related to its internal memory management that by default tries to use the /dev/shm directory. I don't quite understand the internals of Chrome, but here's some details about this from the puppeteer project. Anyway, Docker containers have a default size of 64MB for /dev/shm (cmd+f for --shm-size). Chrome quickly runs out of space in there and your page will then crash, taking your tests down with it. No bueno. There’s two solutions to this:

The easiest way to deal with this problem is to just not use it. There are some small cons, but a good rule of thumb is that unless you're trying to parallelize multiple concurrent Chrome sessions within one container, you're probably safe to disable it by adding --disable-dev-shm-usage to your Chrome options.

If you are though (like this engineer), then you'll need to increase the size of your /dev/shm. For Docker containers,

there’s a flag for increasing the shared memory size --shm-size=1G that you can pass to your docker run command.
you can also mount /dev/shm from your host machine to the docker container via -v /dev/shm:/dev/shm

--disable-setuid-sandbox --no-sandbox

If you see errors failing to connect to Chrome like this, you might need these flags. Chrome uses sandboxing in order to protect your machine from random content on the internet (see details here). If you’re running Chrome inside of a disposable container, then your container is already acting as a sandbox for Chrome, and so you don’t quite need chrome’s sandbox (your mileage may vary here depending on what you’re trying to do).

I typically see these two flags passed along together in various writeups online. This answer from the Chromium team suggests that --disable-setuid-sandbox was only used in older versions of chrome.

--disable-gpu

The official Puppeteer docs suggest that this flag is only needed on Windows. Various snippets online usually use --disable-gpu alongside --headless because an earlier version of headless chrome had bugs with it.

--disable-infobars

This flag can help a minor annoyance - if you take a screenshot of the browser in your tests, then you’ll see a bar at the top of your browser page saying that Chrome is being controlled by automated process. This also pushes elements farther down the page. Its usually fine, but if it gets in your way, then try using this option. This flag was at one point removed around Jan 2018, and then added back.

Wrapping up

Yes, Chrome can run headlessly and it can run inside of a container, and be stable (for the most part). You do however need the right concoction of parameters to get it working right, but hopefully this saves you a few hours of debugging random page crashes. Try out those Chrome options in your configuration (they will also work if you're using puppeteer or other selenium/chrome combo).

Running into different errors? See how to attach the VSCode debugger to step through your UI tests to help you narrow down the issue.

Debugging webdriverio tests with VSCode

Danny Perez — Tue, 02 Apr 2019 13:21:40 +0000

EDIT - If you're interested in seeing me walk through this post live - check out my latest Youtube video!

Debugging tests with webdriverio can get frustrating when you’re trying to figure out why your test is sometimes clicking the wrong elements or just plain not working. Am I using the correct selector? Which element is it actually clicking on? There’s 3 things that you can do to help you drill down:

adding many console.log statements to your test
- using a debugger to step through the test one line at a time
- using webdriverio’s browser.debug() to get an interactive js session with the browser
- while this seems like the obvious choice, using browser.debug has its own limitations that I describe here

I searched and had found this post of getting webdriverio tests running inside of vscode to help me step through a test file line by line. That post is using an older version of node, and newer versions of node (v8.11.x+) throw an error like below:

node --debug=127.0.0.1:5859

(node:29011) [DEP0062] DeprecationWarning: `node --debug` and `node --debug-brk` are invalid. Please use `node --inspect` or `node --inspect-brk` instead.

There's 2 parts to get this to work with newer versions of node: VSCode configuration and webdriverio configuration

VSCode configuration

See here for creating a VSCode Launch Configuration. Adding this file will allow you to run the VSCode debugger and step through your code line by line.

Heres a working .vscode/launch.json file that you can use and adapt to fit your needs

{
  "type": "node",
  "request": "launch",
  "name": "Run in debug",
  "port": 5859,
  "timeout": 60000,
  "runtimeExecutable": "${workspaceRoot}/node_modules/.bin/wdio",
  "cwd": "${workspaceRoot}",
  "console": "integratedTerminal",
  "args": [
      "--spec", "main.js"
  // "--spec main.js" will be passed to your executable as
  // "wdio '--spec main.js'" (which isn't what you want)
  ],
  "env": {
      "DEBUG": "1" 
      // use an environment variable to be able
      // to toggle debug mode on and off
  }

This configuration will run your runtimeExecutable and by defining the "port" variable, it will attempt a connection to port 5859. Once that connection is successfully made, you'll be able to set breakpoints and step through your test.

webdriverio configuration

On the webdriverio side, we need to tell it to listen for connections from a debugger on port 5859.

In your wdio.conf.js file:

exports.config = {
  debug: true,
  execArgv: ['--inspect-brk=127.0.0.1:5859],

  // other wdio configuration
  specs: ['some/specs/here'],
  suites: {
    ....
  }
  ....
}

This snippet will start webdriverio and start listening for connections from a debugger on 127.0.0.1:5859 (which you did in your VSCode configuration). The program will stop at this point to wait for a debugger to connect, and if nothing connects, the command will fail.

Once you run it successfully, you should see this type of output

/Users/dperez/workspace/wdio/node_modules/.bin/wdio "--spec" "main.js"

Debugger listening on ws://127.0.0.1:5859/9698ad4c-8d7d-447f-a259-1c566cd511d6
Debugger attached.

You'll see the program pause at this point until a connection is made. If you never see "Debugger attached", it means VSCode has not connected to your program, and so you can't set breakpoints and debug.

If you run this as part of your CI process (Gitlab/Jenkins), you can make debug mode configurable. You can use env vars in your .vscode/launch.json configuration.

...
"console": "integratedTerminal",
"env": {
  "DEBUG": 1
}

This will run your program with that env var set by using:

DEBUG=1 /Users/dperez/workspace/wdio/node_modules/.bin/wdio "--spec" "main.js"

Then in your wdio.conf.js file:

exports.config = {
  debug: process.env.DEBUG === '1',
  execArgv: process.env.DEBUG === '1' ? ['--inspect-brk=127.0.0.1:5859'] : []
  ...
  // remaining wdio options
  ...
}

This way, the program will only wait to attach to the debugger when you run it inside of VSCode (where the environment variable is set), and everywhere else it will run normally without it.

Wrapping up

Add these snippets to your configuration if you need to step through your program and watch how the program is executing. It sure beats writing tons of console.logs all over your code.

Interested in using webdriverio's browser.debug() mode? check out my post on using wdio's debug mode to play with the browser.
In case you’re here because you’re trying to solve the dreaded ‘element is not clickable at point’ error message, check out this post here

If you want to read more stuff like this, give me a ❤️ and follow me here on dev.to.

9 books that helped me navigate my first time being a tech lead

Danny Perez — Wed, 12 Dec 2018 18:08:11 +0000

The tech lead was moving to another team for a long-term assignment, and I took over as the engineering manager/team lead. From the outside, the tech lead's job seemed doable, but I quickly realized I was getting in over my head. Unfortunately, my team was responsible for a lot of centralized infrastructure as well as day-to-day technical operations. I had no tech lead training, and how could there be? I was certain that the tech lead role was so different across companies, how could there be guidelines to it? In my previous role as the senior engineer on my team, I felt capable of tackling larger projects, but I only ever had 1 project to tackle. Now, I needed to manage 3-5 projects for my small team of 5 engineers.

The best I could do was do as the last person did which only gets you far enough to keep your head above water. I realized that the only way for me to get past this would be to hit the books, and learn all the management that I never learned in college. I read a lot that year. More than the past 3 years combined. The most helpful books I read all boiled down to 3 areas of the job that I (like many others) struggled with: dealing with team & individual performance, delegating, and making my team a great team to work on.

*Disclaimer: I've tried to link to the authors website where I could. Otherwise, the links go to Amazon (not referral links) if you want to get the book. I have no association with any of the below authors, just a fan of their writing.

Dealing with performance

One thing I felt affected the team was an under-performing team member. Surely like many others, I hadn't ever seen an effective performance review myself, or seen how other leads dealt with performance on their team (Thats probably a good thing, but unhelpful for me). While I know at a high level what you're supposed to do, like talk about and deal with the problem, I struggled to actually do it - how could I, a new lead, give feedback to someone who was previously a peer on my team? It's definitely awkward the first few times! Fortunately, in this area, there are people much smarter than I who have shared their experiences in great deal to help you get past these types of problems.

Radical Candor: Be a Kick-Ass Boss Without Losing Your Humanity by Kim Scott
- If you want to focus on giving great feedback - this provides a pretty powerful mental model for giving and receiving feedback without feeling like an asshole. It's focused on being upfront with people, lest you be plotting against them or humiliating them.
Leading Teams: 10 challenges and solutions by Mandy Flint, Elisabet Vinberg Hearn and Debugging Teams: Better Productivity through Collaboration by Brian W. Fitzpatrick, Ben Collins-Sussman
- If you want to focus on getting yourself un-stuck from some problems on your team - these 2 books go in-depth on common issues that teams might have, and some step-by-step guidelines for how to deal with them.
- This was an area I didn't feel comfortable asking any of the other managers about. We had many small teams, and we were all very friendly so it was hard to find people you can talk to. The next best thing was seeing what other more successful leaders had done in these scenarios.
Manager Conversation with Low Performers at UMCB on Youtube
- It's very rare you'll ever get to shadow someone else's performance review - they're private and personal by nature (or you're HR). If you've ever wondered what a "good" conversation about poor performance could look like, this video helped me a TON! There's some other related videos there that will show you what NOT to do, but it is great to see how you can quickly diffuse an awkward situation.

Lesson #1 summary: Be incredibly explicit about your expectations with their job so that they can never say, "How was I supposed to know?"

Delegating

Another awkward part about my job was telling people what to do - our team had a mission that we were mostly aligned on, but we don't always get to work on cool stuff and the work still has to get done on time. I had seen other people do it well, I had seen others do it poorly - but I wouldn't have been able to explain to you why. I felt awkward the first few times saying, "hey roger, can you take a look at this issue?" only to have that developer come back with something that I wasn't expecting (see Lesson #1).

When I was a fellow engineer on the team, I felt capable of working on bigger projects and making sure that we shipped the right things, but now, I was also accountable for all the projects the team was working on, not just mine. I had about twice as many things to do now, and the typical-engineer-turned-manager in a bout of frustration might ask, "When am i supposed to code if I'm stuck in meetings and dealing with people all the time?" It was difficult to juggle all the projects that I was now responsible for, as well as do the work on critical projects, and plan cross-team initiatives, and insert 20 more things here.

To summarize these books: Be incredibly explicit about your expectations with projects/tasks so that they can never say, "How was I supposed to know?"

While these two books sound a little cheesy, they gave me a great framework and process for delegating. After reading them, I started blocking out time on my calendar for going through our projects and trying to match people's goals and motivations with the work we had to do. A bunch of us got AWS certificates, one engineer earned with a promotion, and an intern joined us full-time. And we built great stuff too.

Making a good team

One way to build a better team is to see more teams and how they operate, and use them as guiding examples for building your own teams. The catch is, barring you leaving your job and working elsewhere, you won't get to see that many teams and so you might not even know what your team would look like at its best! I absolutely loved reading these books because they provided case studies of real teams with real stories across some high-profile companies. Some people had really crappy times at their job, others didn't and they explain why in-depth.

These books are more focused on software engineering teams:

Talking with Tech Leads: From Novices to Practitioners by Patrick Kua
- Patrick Kua is a great speaker with some of his talks available on Youtube about technical leadership covering things like What I wish I knew as a first time Tech Lead and Geek's Guide to Leading Teams
Building Software Teams: Ten Best Practices for Effective Software Development by Joost Visser, Sylvan Rigal, Gijs Wijnholds, Zeeger Lubsen

These books cover teams in general:

To summarize these in one sentence: Be incredibly explicit about your expectations with the team's culture so that they can never say, "How as I supposed to know?"

Wrapping up

I had a great time leading my team for over a year. While at times, it was incredibly daunting to think how I would get through a particularly problematic week, my team would stay on track and over time, we were able to move to more proactive work. There's many different areas in management where you could spend days and day learning, but if you do 1 thing and nothing else...

Tell your team to be incredibly explicit about their expectations from you as their lead so that you can never say, "How was I supposed to know?"

If you found this helpful, you can ❤️ 🦄 it and follow me here on dev.to 😄

Lastly to fellow tech leads on dev.to - what was the hardest part of being a first-time tech lead?

3 things you might see in your logs once your site is public

Danny Perez — Mon, 03 Dec 2018 23:44:54 +0000

You've finished deploying your website to its new domain. You start to see your normal user traffic, but then you also notice funny patterns in your access logs. Here's a few examples of things you might see in your access logs once your site or API is public in production.

Automated vulnerability scanners from all over the world
Crawlers
SQL Injection attempts

Scanners

You know what regularly shows up? Scanners. Lots and lots of open source scanners. IPs originating from all over the world China, Ukraine, Russia. Randomly throughout the week, we'll see scanner traffic. What does scanner traffic look like?

There's a few signs you can use to tell:

You have slightly elevated error rates for a sustained period of ~30 minutes and suddenly dies off. A mix of 4xx's and 5xx's.
It's requesting paths that don't exist in your application.
Some scanners use a specific User-Agent header that identifies itself as a scanner

You might see requests like this...

x.x.x.x - - [07/Apr/2018:02:50:27 +0000] "GET /wp-json/oembed/1.0/embed?url=..%2F..%2F..%2F..%2F..%2F..%2F..%2FWindows%2FSystem32%2Fdrivers%2Fetc%2Fhosts%00&format=xml HTTP/1.0" 404 132 "http://www.zzzz.yyy/xxxx" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.6 Safari/534.57.2"

In there is an encoded URL: ../../../../../../../Windows/System32/drivers/etc/hosts&. This type of request looks mainly like a recon attempt to see if it can even get to files in that directory.

If you're running a wordpress site, always be prepared to receive a barrage of known exploits to your site. We're not even running a wordpress website and we get this type of thing in our access logs. Included there is a revslider_show_image vulnerability to steal wp-config file from 2014.

| count | uri                                                                           |
|-------|-------------------------------------------------------------------------------|
| 12    | /wp-admin/admin-ajax.php                                                      |
| 6     | /wp-admin/admin-ajax.php?action=revslider_show_image&img=../wp-config.php     |
| 6     | /wp-admin/tools.php?page=backup_manager&download_backup_file=../wp-config.php |
| 3     | /wp-admin/wp-login.php?action=register                                        |
| 2     | /help/wp/wp-admin/setup-config.php                                            |
| 1     | /help/new/wp-admin/setup-config.php                                           |
| 1     | /help/wp-admin/setup-config.php                                               |
| 1     | /wp-admin/js/password-strength-meter.min.js?ver=4.9.1                         |
| 1     | /wp-admin/js/password-strength-meter.min.js?ver=4.9.2                         |

Depending on how your application is hosted, you may even see increased latencies during a scan:

Crawlers

These are innocuous. Just Google and Bing indexing content. Typical internet being the internet. This particular crawler is nice enough to include a User Agent header that sends you to a site that says exactly what they do with all the data they collect. Check it out - http://www.exensa.com/crawl

x.x.x.x [30/Mar/2018:04:39:15 +0000] "GET /robots.txt HTTP/1.1" 404 136 "-" "Barkrowler/0.7 (+http://www.exensa.com/crawl)"

SQL injection attacks

You might see these recon attacks - you can take a known public URL and add a URL-encoded SELECT statement to see if anything funny comes out in the response. Like this:

x.x.x.x [11/Apr/2018:16:09:37 +0000] "GET /REDACTED&response_mode=%28SELECT%20%28CHR%28113%29%7C%7CCHR%28107%29%7C%7CCHR%28120%29%7C%7CCHR%28107%29%7C%7CCHR%28113%29%29%7C%7C%28SELECT%20%28CASE%20WHEN%20%285423%3D5423%29%20THEN%201%20ELSE%200%20END%29%29%3A%3Atext%7C%7C%28CHR%28113%29%7C%7CCHR%28122%29%7C%7CCHR%28112%29%7C%7CCHR%2898%29%7C%7CCHR%28113%29%29%29&response_type=code&scope=openid HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/4.0.202.0 Safari/532.0"

There's a SQL injection string in there that is URL encoded. If you decode it, you get this

(SELECT (CHR(113)||CHR(107)||CHR(120)||CHR(107)||CHR(113))||(SELECT (CASE WHEN (5423=5423) THEN 1 ELSE 0 END))::text||(CHR(113)||CHR(122)||CHR(112)||CHR(98)||CHR(113)))

These are tests for injection. If the response contains something out of the ordinary, they know they can do it. Not entirely sure whats going on here though.

Defend against it

Here's a few things that you can use to keep most of these people away.

Use security groups (in AWS) / iptables to limit inbound traffic only to the ports that you need. If only and 80/443 are open, you're primarily prone to web application attacks instead of SSH/FTP/DNS and the like. Here's an AWS Whitepaper on Security Best Practices
Periodically review your logs to see what's going on. If you see SQL injection attempts, see which calls have returned a 200/500 - this means something potentially useful has made it to attacker.
Outright reject anything thats not even remotely close to your applications URLs - For example, you'd want to block Windows paths when your application is hosted on Linux like paths containing System32, cmd.exe, C:\\ in your access logs.
Run a scan against your application locally and be one step ahead of hackers. Know what you're exposed to first. Check out these open source scanners
Many of these problems go away if you host your static website on AWS S3 - here's a guide I wrote if you're doing it for the first time
if you're looking for an alternative to Wordpress? See this guide for building a blog with gatsby.js

If you found this helpful, you can ❤️ it and follow me here on dev.to 😄