DEV Community: Dan Stanhope

Generate a PDF & upload to S3 using AWS Lambda & Puppeteer.

Dan Stanhope — Thu, 09 Jun 2022 18:09:13 +0000

I've put together a working project that'll take all of your dev.to posts, create a .pdf and upload it to S3. Ba-boom.

Head over to GitHub to grab the code.

First, a little bit about the project. It's run using sam cli and CloudFormation(make sure you've got your sam/aws cli installed and configured properly. Here's how to do this).

Modules and custom code will be compiled into a Lambda Layer. This is where all the requisite node_modules will live, as well as some custom code and our html templates. Cool thing about layers is once you've built it you can share it amongst your other functions. We're also going to be including another layer(chrome-aws-lambda), which will be used to run puppeteer inside the Lambda environment.

Here's what the .pdf looks like with my data(I know, not great. It's a .pdf and I'm no designer. Let's move on).

Show me the code!

I briefly mentioned layers before, it's a pretty cool feature of AWS Lambda. If you've got a bunch of functions running and you want to share node_modules or custom code(i.e. adapters, helper functions, templates etc.) you can create 1 or more layers and attach them to your functions.

When importing anything other than a node_module you need to reference a different path(/opt/). Here you can see the modules being included vs. the custom code.

The function is quite simple, we start off by ingesting our html template from our layer and initializing a few variables(*note: change the bucket name to something other than my name). Next, we pull down some dev.to posts and compile our template.

Then we initialize puppeteer and pass in our template string. We'll create a buffer and give this to our S3 upload method.

Guys. That's it. It's done. We made a .pdf.

How do I run this?

Once you've cloned the repo, head into /layers/shared/nodejs and run yarn. This will install all the packages we need. You can add this to a build step at some point, too. When creating a layer, it's important to note that you need to include the function's runtime as part of the folder structure in order for lambda to recognize it(in this case nodejs).

Jump back into the root of the project and run: ```sam

local invoke PuppeteerFunction --no-event


If you've configured everything properly, you should have a nice .pdf created & waiting for you in S3.

One thing to note, there is some strange bug with the aws chrome package and it's not working with Nodejs14. That's why all the runtimes are set to Nodejs12. If you can get it working with a more up-to-date runtime, please let me know and I'll update the repo.

Thanks so much for reading! Hope this helps someone.

What's Automatic Batching? React 18 feature explained

Dan Stanhope — Thu, 31 Mar 2022 14:57:49 +0000

If you've ever built a component in React, chances are you've used state variables. If you've ever build a kinda-complex component in React, chances are you've got multiple state variables in use.

So, what happens when we update those variables? The component re-renders, right? Make changes to a bunch of state variables and a bunch of re-rendering happens. All this rendering could have performance implications for your app.

Introducing automatic batching. Now, batching has been around for a bit in React. But, React only automatically batched state changes for you if they were called inside a hook or browser event. Like say, a click event:

Console Output:

This is automatic batching. React takes multiple state changes and groups them together so they don't happen independently -- fantastic stuff.

So where's the improvement?

There are other places you may want to change state in your component(promises, timeouts). Let's say we have a component that fetches some data after a button click. We have two state variables, an array of users and a page counter. We want to update these inside a promise once the data is returned. In React 17, this will cause the component to re-render twice.

Console Output React 17:

Console Output React 18:

This is great! You can make changes to a couple state variables and React will apply them all at the same time, automatically, for you. Awesome!

If you weren't aware of how batching worked in previous versions of React, hopefully you know the limitations now. And if you've got some components out there changing state variables inside promises, might be time to upgrade :)

Thanks!

Building a Compound Interest Calculator using Chart.js + Next.js

Dan Stanhope — Thu, 03 Mar 2022 19:25:21 +0000

Overview

So, just for fun I decided to create a compound interest calculator using Next.js, TypeScript and Chart.js.

I wanted to kick the tires on Vercel, since I hadn't deployed anything to their platform prior to this. It's so awesome! The build times are so fast and it's really easy to get up-and-running.

You can view the calculator here(excuse the domain name, it was the cheapest one I could find that still made some sense haha.). Also, I've got all the code in a public repo here, if you want to take a closer look.

In terms of the calculations, I worked off of the formulas found here. I did my best to verify my results against some other sites out there and, as far as I can tell, it's working well -- famous last words.

Let's talk code

This is the first react project I haven't used Redux with in some time. Instead, opting to go with useContext and useReducer. Once I got it running, I thought it was great!

I had a few components that needed access to the input field values in order to generate the results and plot the graph etc. useContext made sharing the state between all the components that needed it really straight forward.

It'll be pretty long-winded to go through the entire project and explain every aspect, so I figured it'd be best to showcase a couple pieces that I found interesting.

Setting up useContext

In order to set up Context and share it amongst your components there's only a few things you need to do.

First, you need to create your context. Make sure you create this as a module, as you'll need it again in your components.

Then you need to wrap the components that will be sharing the state in a provider component. It's useful to note that you can share multiple contexts by simply nesting the provider components. As you can probably guess from the screen-shot, the Form, Graph and Table components will have access to both Contexts.

At the component level, you'll just need to import your context definition we used in the first step and initialize it.

Once you have this running, you'll be able to access your state across components. Sa-weet!

Crunching numbers

Calculating compound interest is pretty straight-forward when your compound frequency and payment frequency are the same. Like if you were to make an annual contribution and your interest was also going to compound annually as well.

Most of the calculators out there allow for the user to mix things up a bit. I mean, for real-world savers the payment & compound frequency seldom match. What if I want to make a monthly contribution but my interest compounds annually? Or semi-annually? Well, you've got alter the formula a little. This was the part that took me a bit to figure out as most of the tutorials out there never went into this and the calculators I was checking my results against did -- they never matched and my brain hurt.

In order to allow for varying payment and compounding frequency, you'll need to calculate the rate and the total number of payment periods variables slightly differently.

Here's the entire, working formula used in the codebase.

F = P*(1+rate)^nper + A*( ((1+rate)^nper - 1)/rate )
rate = ((1+r/n)^(n/p))-1
nper = p*t

*Head here for a more detailed explanation.

Once I had that working, it was just a matter of iterating for each year and adding the results to an array.

Anyway, if you're interested in seeing how this works clone the repo and let me know if you have any questions.

Apologies that the write-up wasn't super-involved. You'll need to get into the code anyway to figure this all out.

Thanks for reading!

Twitter Authentication using Supabase + React + Redux + Typescript

Dan Stanhope — Thu, 30 Dec 2021 16:22:43 +0000

As the title of this post suggests, we're going to be building a small React + Redux app that'll allow your users to authenticate using their twitter credentials and access auth-only parts of the app.

The code for this tutorial can be found here. Go ahead and clone that and get it running using yarn update and yarn start

Unfortunately, there's a bit of boring setup & config stuff we'll have to walk through in order to get this working for you. So bear with me and we'll get this going, I promise(If I missed something, let me know and I'll help you work through it).

Step 1: Twitter + Supabase Setup

Head over to Supabase and grab your project's api url(settings ->api->config->url) and append /auth/v1/callback to it. This is going to form the callback url we provide to Twitter, it'll look something like this: https://.supabase.co/auth/v1/callback

Twitter

Make your way over to https://developer.twitter.com/ and create a new project. Once you've filled everything out, make sure to keep your API key & secret(used as client_id and client_key in Supabase).

Next, you'll need to create an app. When you've done that, you'll need to enable 3rd party authentication -- this is where we'll add our callback url.

Be sure to add the localhost url to the callback section. For any of the remaining, required url fields just put your app url in for now.

Supabase

Under settings on the authentication tab enter the site url as well as our localhost callback url. A picture is worth a thousand words...

Next, enable Twitter authentication and enter your Twitter creds.

Geez, hopefully that wasn't too painful and I haven't lost too many people. If we did this correctly, the boring config stuff is over. Woot woot!

Step 2: Code walkthrough

First thing is first, you'll need to make a couple changes to the env.development file. Most importantly, adding your Supabase anon key(settings ->api->project api keys
->anon public).

The project is a pretty standard React + Redux app(styled with Tailwind). The most important part to take note of is how we're protecting the pages of our app that require the user to be authenticated. Take a look at src->pages->PrivateRoute.tsx & src->App.tsx and you'll be able to see how the PrivateRoute component ensures that only authenticated users can access certain pages.

PrivateRoute.tsx

App.tsx

After running our yarn commands, you should have a page open that looks like this:

If everything is setup properly, after clicking the sign in button you'll be sent to Twitter to enter your credentials and sent back to our dashboard page.

Hope this helps someone get started with Supabase authentication. As an aside, you could easily port this project over to Firebase or Amplify authentication with few code changes. There would be a bunch more super-fun setup and config for you to do though :)

If you get stuck or have any feedback, let me know!

Thanks for reading.

Making sense of Next.js rendering

Dan Stanhope — Tue, 21 Dec 2021 19:49:21 +0000

Gone are the days when you'd fetch data from your database, pass it to your view and render it. Well, these days aren't gone. But we can definitely do better.

Next.js has three different rendering modes: Server Side Rendering(SSR), Static Site Generation(SSR) and the newly minted ISR(Incrementation Site Regeneration). Each method serves a purpose and comes with its own pros and cons list -- let's get into those!

To illustrate how the three modes function, I've included a small amount of code you could use in a Next.js project.

The component makes a request for a list of public repos and, to introduce some variability, randomly selects the sort direction before rendering.

If you'd like to follow along, create a Next.js project(yarn create next-app) and replace the code found in pages/index.js.

SSG(Static Site Generation)

In the early days of the web, this is kinda how websites worked. HTML files were just stored on disc on a webserver and with each request these pages were served up. No trips to the database or rendering to be done, it's all ready to go and sitting there. Changing the page content is a bit of pain, but because everything is built and you're reading from disc, it's blazing-fast.

With SSG your pages are generated during Next.js' build step, and once these pages are created, that's it. They don't change until you run the build step again. This step is probably done as part of a CI/CD deployment process, so updating the site means running through that again before heading to production.

To try this out, yarn build and start your project. Once your site is ready, hit refresh a few times and you'll notice the order of the list never changes. That's because only 1 request for data was made and that happened at build time.

export default function Repos({ repos }) {
  return (
    <>
      <h3>Repos</h3>
      <dl>
        {repos.map((repo, index) => (
          <div key={index}>
            <dt>{repo.name}</dt>
            <dd>{repo.description}</dd>
          </div>
        ))}
      </dl>
    </>
  );
};

export async function getStaticProps() {
  const direction = parseInt(Math.random() * 2) ? 'asc' : 'desc';
  const res = await fetch(`https://api.github.com/users/vercel/repos?sort=created&direction=${direction}`, {
    headers: {
      'Accept': 'application/vnd.github.v3+json'
    }
  });
  const repos = await res.json();

  return {
    props: {
      repos
    },
  };
}

SSR(Server Side Rendering)

Pages are generated and returned after each page view. Think about how a traditional WordPress site works(also think about how slow WordPress can be). The slowness(lag) is introduced by way of network requests for data(DB & API calls etc.). The benefit being that as the content creator edits the page data in the CMS, those changes are reflected right away to the user.

Definitely a convenient way to manage your website, but your pages are going to be kinda slow to load.

Try building & running your project once again and notice how when you refresh(do it a few times) the sort order flips from ascending to descending. This is because with each refresh, the component is making a request to the API for data.

export default function Repos({ repos }) {
  return (
    <>
      <h3>Repos</h3>
      <dl>
        {repos.map((repo, index) => (
          <div key={index}>
            <dt>{repo.name}</dt>
            <dd>{repo.description}</dd>
          </div>
        ))}
      </dl>
    </>
  );
};

export async function getServerSideProps() {
  const direction = parseInt(Math.random() * 2) ? 'asc' : 'desc';
  const res = await fetch(`https://api.github.com/users/vercel/repos?sort=created&direction=${direction}`, {
    headers: {
      'Accept': 'application/vnd.github.v3+json'
    }
  });
  const repos = await res.json();

  return {
    props: {
      repos
    },
  };
}

ISR(Incrementation Site Regeneration)

For most, this is kind of the best of both worlds. ISR will create each page using SSR initially and store the response the same way SSG does. Subsequent requests for the same page will now be read from disc and be, you know, super-fast.

Content creators can make changes to their pages using a CMS and after the expiration period has lapsed, view their changes. What's really cool here is that Next.js is basically creating a caching layer for you. When it's time to regenerate the page, it'll do it behind the scenes and, once complete, replace the SSG page for future reads. Amazing stuff, really.

Because this version of code is only going to request new data every 10 seconds(validate: 10), I added a time stamp to the component. So once you've built and run it, refresh a few times and you'll notice that the time stamp doesn't change after the initial render. Refresh 10 seconds or so later and you'll see the time stamp change(and hopefully the sort direction).

export default function Repos({ repos, time }) {
  return (
    <>
      <h3>Repos</h3>
      <em>{time}</em>
      <dl>
        {repos.map((repo, index) => (
          <div key={index}>
            <dt>{repo.name}</dt>
            <dd>{repo.description}</dd>
          </div>
        ))}
      </dl>
    </>
  );
};


export async function getStaticProps() {
  const direction = parseInt(Math.random() * 2) ? 'asc' : 'desc';
  const res = await fetch(`https://api.github.com/users/vercel/repos?sort=created&direction=${direction}`, {
    headers: {
      'Accept': 'application/vnd.github.v3+json'
    }
  });
  const repos = await res.json();
  const time = await new Date().toISOString();

  return {
    props: {
      repos,
      time
    },
    revalidate: 10
  };
}

It's pretty easy to see why Next.js is so popular!

And if you’re having a hard time selecting the best rendering method, go for ISR :)

Thanks for reading.

Update your mapping & reindex ElasticSearch easily...well, pretty easily

Dan Stanhope — Mon, 07 Jun 2021 19:05:26 +0000

If you've ever worked with ElasticSearch you've probably had to alter your original mapping and reindex your cluster's data. Unless you're super-awesome at predicting the search functionality you need from ElasticSearch and got your mapping perfect the first time. If this is you, no need to finish reading this post.

Tinkering with data in production isn't the most fun job in the world and I've never really found step-by-step instructions on how to properly/safely perform a reindex. So that's what this post is all about. Hopefully it'll help someone!

As always, test this out on your local, staging or any non-production ElasticSearch cluster before assuming it works. How many records you're reindexing will be one of the determining factors in how long it takes to process.

Create your new index + mapping

The first step is to create an entirely new index with your new mapping.

Let's say our ElasticSearch is running on http://127.0.0.1:9201, our old index is called testv1 and our new index testv2 .

For brevity, the mapping is empty.

curl -XPUT --header 'Content-Type: application/json' http://127.0.0.1:9201/testv2 -d '
{
  "settings": {},
  "mappings": {
    "_doc": {
      "properties": {}
    }
  }
}
'

Reindex your data

Here's where we actually begin running the data from your old index into your new one. One thing to note is the wait_for_completion flag. When this is set to false, a task will be create and allow to you issue subsequent requests to get progress updates. Without it, the command will just hang and you'll likely receive a curl timeout and have no way of knowing when it's done reindexing.

curl -XPOST --header 'Content-Type: application/json' http://127.0.0.1:9201/_reindex?wait_for_completion=false -d  '
{
  "source" : {
    "index" : "testv1"
  },
  "dest" : {
    "index" : "testv2",
    "version_type" : "external"
  }
}
'

This call will return a task id that can be used, to get status updates, like this:

curl -XGET --header 'Content-Type: application/json' http://127.0.0.1:9201/_tasks/<TASK_ID>?pretty=true

Once your reindexing task is complete, you can verify the document counts in your old index match the new one.

curl -XGET http://127.0.0.1:9201/testv1/_count
curl -XGET http://127.0.0.1:9201/testv2/_count

When you're satisfied that everything is looking good, move onto the last step...creating an alias and telling people how awesome you are at ElasticSearch.

Create an alias pointing to your new index and remove the old one

You likely don't want to have to alter all your queries to use the new index name. Fortunately, ElasticSearch allows you to create aliases. The idea being that we create an alias with the same name as the original index, but point it to the new one. This way there's no changes to be made to our queries.

curl -XPOST --header 'Content-Type: application/json' http://127.0.0.1:9201/_aliases -d  '
{
    "actions" : [
      {
        "add" : {
          "index" : "testv2",
          "alias" : "testv1"
        }
      },
    {
        "remove_index" : {
          "index": "testv1"
      }
    }
  ]
}
'

Voilà! You're done.

As an aside, there are things you can do to improve the reindexing performance. Like changing the refresh_interval and lessening the number of replicas. Whether or not this is something you need will largely depend on the size of cluster and underlying hardware.

Here's a sample of how to do this, should you need to. Make sure you run this prior to starting the reindex command.

curl -XPUT --header 'Content-Type: application/json' http://127.0.0.1:9201/testv2/_settings -d  '
{
  "index" : {
    "refresh_interval" : "-1",
    "number_of_replicas": "0"
  }
}
'

Make sure to set these values back to whatever you feel is best once the new index is fully populated.

Thanks!

Migrating DynamoDB data using Lamba + Streams

Dan Stanhope — Fri, 19 Mar 2021 19:44:11 +0000

The scenario

You've got an existing DynamoDB table and you'd like to migrate the data to another table. Or, you've got some data that pre-dates whenever you enabled streams and lined up that Lambda event listener. What's the move?

First, what are Streams?

When records are added or updated in your DynamoDB table change data is created and added to an event stream. This stream is super-easy to monitor & consume with a Lambda function. Basically, as records change data is added to a stream and you can capture that data with a Lambda function in near-realtime. Sweet.

One thing to note, event stream data is only stored for 24hrs, after which it's gone. Or is it? It is.

A common pattern, utilizing streams, is to write to a table, process the change data using Lambda and write to another location(i.e ElasticSearch, SQS). Maybe the data gets transformed a little bit along the way, too.

Let's say this is something you're doing -- you've got a nice pipeline running that sends data from dynamodb -> lambda -> elasticsearch but you've got some old data in the table that arrived before the stream was enabled. You can write a script that scans/queries the table and updates each entry with a flag(pre_existing_processed in our case, but change to whatever you like). By updating the existing record, it creates new change data and writes to the event stream. Pretty cool!

You could formulate a query that selects the records you'd like to get onto the event stream(date range, perhaps?) and update each record with a flag(something to indicate it's an old record).

The Code

I've created a small project that runs a paginated(DynamoDB will return up to 1MB worth of data per page) query and performs a bulk update(AWS allows a max. of 25 records per bulk update).

Clone the GitHub repo here.

Make sure you update ./aws_keys.json with AWS credentials that have access to DynamoDB before starting.

It's important to note that you'll likely need to increase your table's read/write capacity -- which comes at a cost.

Start by adding the requisite packages:

yarn

Run the script(you'll be prompted for your table name):

node migrate.js -t <YOUR_TABLE> -b <BATCH_SIZE>

There's also a batch limit parameter, in case you want to run a set number. Remember that depending on how much data you've got, it could take a long time to run. I recommend testing with a small batch size first to ensure everything is running how you expect it to.

This approach can be used to process millions of legacy/pre-existing records...but it'll take some time 😊

As always, be careful running this code and make sure you know the cost implications etc.

Hope this helps!

React file upload using S3 pre-signed urls

Dan Stanhope — Tue, 16 Feb 2021 22:05:46 +0000

Don't forget to like!

What are we building?

We're going to create a lambda function that generates a pre-signed url as well as a react front-end utilizing a really cool component library!

Traditionally, uploading files could be a bit of pain to implement & manage. Fortunately AWS allows you to upload objects directly to an S3 bucket using pre-signed urls. Pre-signed urls come with an expiration date, so you need to start your upload before then otherwise access gets blocked.

Walk through time.

The project is divided up into two sections, basically. The front-end and the back-end.

Head over to github to grab the code.

Back-end

We're going to be using CloudFormation and AWS SAM to create and deploy our Lambda function as well as create our S3 bucket. This function, when called, is going to generate our pre-signed url for us. You could just as easily host this code within your own API, too.

Firstly, make sure you've got aws-cli and aws-sam-cli installed and configured(setting up your keys & region etc.). Here's how to do this.

Once you're all set up and ready to go, all you need to do is run sam build followed by sam deploy --guided from inside the lambda function's root folder. SAM cli will guide you through the deployment and, once successful, you'll have a newly created S3 bucket and lambda function.

Make sure you copy your lambda function's api gateway url, as you'll need to make one small change in the Upload.js component.

Front-end

Update the Upload.js component with your API endpoint.

const SignedUploadDragAndDrop = () => {
  useRequestPreSend(async ({ items, options }) => {
    const files = items.length > 0 ? items[0] : {};

    let { file } = files;
    let { name, type } = file;
    let gateway = '<YOUR APIGATEWAY ENDPOINT URL>';

    const response = await axios(
      `${gateway}?` +
      new URLSearchParams({
        name,
        type,
      })
    );

   ....

After this, just run yarn and yarn start from inside the frontend folder and you should end up with a page that looks like the one in this post's hero image.

I've used a seriously awesome component library called React-uploady for this tutorial. Specifically, I've combined its upload button, drag-and-drop and progress components. But there are several others you can add on. Check it out!

When you select a file to upload a request is made to retrieve the pre-signed url and, once returned, the upload begins. Pretty sweet.

Hope this helps!