DEV Community: zahaar

Generate PDFs from HTML via Puppeteer on AWS Lambda + API Gateway

zahaar — Mon, 25 Apr 2022 01:45:00 +0000

“Evil cannot create anything new, they can only corrupt and ruin what good forces have invented or made.” - JRR Tolkien.

Preface

Would it be great to have the functionality that would enable you to generate PDF files using HTML && CSS capabilities without the need to rely on overly complex drivers that are dependent on a whole bunch of C libraries?

While also supporting all the latest features of HTML5 && CSS3?

Well, we have great news. There is a framework called Puppeteer that uses relatively new Chrome feature and makes it accessible though a NodeJS based API.

Essentially what Puppeteer does, is: Launches a Chromium browser instance in a headless mode ( not actually opening it ), and allows us to manipulate the browser via set of API command to parse website, retrieve images and generate PDF as if you were actually opening an HTML file in the latest browser version, etc..

While we can create a running Docker Puppeteer instance and deploy that on ECS or Heroku. The creation of stable && optimized image can be quite challenging...

Having a running instance in AWS Lambda IHMO in contrast would be much simpler in terms of development speed, debug and monitoring. Besides, serverless, is a nice concept for POC ( you pay for what you use )

Repo -> End Result

You can see the complete working example in this repo

zahaar / generate-pdf-lambda

Generate PDF document via Puppeteer running on AWS Lambda

This repo contains a serverless application that takes a HTML template and return a PDF in form of a binary

Diagram

Requirements

How to Run

Clone this repo git clone https://github.com/zahaar/generate-pdf-lambda
Import cUrl to Insomnia ( Postman is not recommended, as it can't visualize Pdf ).
Run make api-local to have local API GW running.
Send cUrl request via Insomnia.

You can also invoke Lambda bypassing API GW, by supplying an example event in file, and running make invokation-local. The response would be a base64 encoded PDF binary.

How to Deploy

A configured AWS CLI V2 is a must -> AWS Console Account && API Keys

make deploy
Fetch AWS SAM deploy output URL Value, and change the Url in Insomnia from localhost to that value execution result in…

View on GitHub

Requirements and Prerequisites

1. SetUp local AWS SAM Template with Chrome Lambda Layer

In this step the local SAM execution setUp will be complete. Once this is done, we will have a strong reference point.

The end version of this step can be fetched from 1_local-setup branch

We can create a basic SAM template by running sam init or reference a guide

but our end goal should be a sophisticated structure like this



├── Makefile
├── VERSION -- for VERSION tracking, helpful for CI
├── envs.json -- to sep envs for local execution ( if necessary )
├── events
│   └── api-gw-event.json -- an example API GW event for local execution
├── src
│   └── app.js -- main source code file
└── template.yaml -- AWS SAM configuration template

app.js contains simple code that will return the same event.body that it receives from example event.



...
...
var response = {
    statusCode: 200,

    body: event.body,
  }

  return response
...

while template.yml has a resource configuration for API GW Service



...
...
  ApiGatewayApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Staging
    BinaryMediaTypes:
      - application~1pdf  // Note the support for binary pdf media Type
...

and the Lambda. As per context of our goal, it's called PdfFunction
Take note of the Layer being used in this config. By setting chrome-aws-lambda, we have essentially ruled out the need to set package.json dependencies for puppeteer and chrome on Docker image thar AWS is using on EC2 for Lambdas, as this step can be quite challenging.



...
...
  PdfFunction:
    Type: AWS::Serverless::Function
    Description: Invoked by EventBridge scheduled rule
    Properties:
      CodeUri: src/
      Handler: app.handler
      Runtime: nodejs12.x
      Timeout: 15
      MemorySize: 3008

      Layers:
        - !Sub 'arn:aws:lambda:${AWS::Region}:764866452798:layer:chrome-aws-lambda:22'
      Environment:
        Variables:
          EXAMPLE_ENV: 'CHANGE_THIS'
      Events:
        ApiEvent:
          Type: Api
          Properties:
            Path: /pdf
            Method: post
            RestApiId:
              Ref: ApiGatewayApi
...

To test that all requirements are met, let's run a local event.



make invokation-local

The output should be essentially the same as the execution logs in CloudWatch



...
...
Mounting /Users/wparker/Dev/scheduled-website-screenshot-app/.aws-sam/build/PdfFunction as /var/task:ro,delegated inside runtime container
START RequestId: e4d7743d-5be2-4735-84c8-9d5160d9a750 Version: $LATEST
...

2. Configure Puppeteer in Lambda; Supply Template HTML

Next step is to program app.js to start puppeteer, consume HTML from an API GW event and return a base64 encoded response that would be decoded on Response by API GW.

The end version of this step can be fetched from 2_generate-pdf branch

We need to change the Lambda handler code to something like this. File ( File is too long to displayed here )

Key takeaways are:

Browser launch args parameters in this example are set specifically for AWS Lambda compatibility.



...
...
    browser = await chromium.puppeteer.launch({
      args: chromium.args,
      defaultViewport: chromium.defaultViewport,
      executablePath: await chromium.executablePath,
      headless: chromium.headless,
      ignoreHTTPSErrors: true,
    })

The return format goal was set to mimic A4 document.



...
...
    await page.setViewport({
      width: 1080,
      height: 1600,
      deviceScaleFactor: 1,
      isLandscape: true,
    })
    pdf = await page.pdf({
      format: 'a4',
      margin: {
        top: '0px',
        right: '0px',
        bottom: '0px',
        left: '0px',
      },
    })
...

The response headers are set for pdf file transfer. isBase64Encoded flag is set to true to inform API GW that it needs to decode the file.



...
...
  var response = {
    statusCode: 200,
    headers: {
      'Access-Control-Allow-Origin': '*',
      'Access-Control-Allow-Methods': 'GET, POST',
      'Content-type': 'application/pdf',
      'Content-Disposition': 'attachment; filename="foo.pdf"',
    },
    isBase64Encoded: true,
    body: pdf.toString('base64'),
  }
...

To test this code, an HTML template is needed. We will use this open-source one for demonstration.
The document is being sent as body with 'Content-Type: text/html'

Please note 'Accept: application/pdf', this is important.

The end result of cUrl request is in this file

To test our result let's start local SAM in local start-api mode. ( akin to a server, contrary to one time invokation)



make api-local

Import cUrl into Insomnia.

Result

3. Deploy Lambda + API GW via SAM

Refers to step in README.md on main branch

Tips

1. Insomnia vs Postman

Instead of playing tricks with Postman PDF Visualization. I highly recommend switching to Insomnia

Insomnia Visualized Response	Postman Visualization Response

2. `Error: Error building docker image: pull access denied for`

Jan 14 '21

Works fine here. You shouldn't need credentials for Public ECR (you can use auth for specific cases) but if you just want to consume it, remove the existing credentials

docker logout public.ecr.aws

and then try the build again.

That said, if you still want to make use of the…

Open Full Answer

3. Consider using latest AWS Lambda feature: Lambda URL

4. Be mindful of HTTP Headers, when dealing with API GW. As this can lead to a huge confusion.

SNS/S3/API AWS Lambda opinionated boilerplate cookicutter

zahaar — Fri, 31 Dec 2021 14:21:05 +0000

The goal of this article is to share and explain, a very basic and straightforward, AWS Lambda template for GoLang, based on cookiecutter templating capabilities.

Table of content:

What it is
How to use
What it is Not
Worth mentioning

What this template is

This is a so-called "tightly coupled" template, it was build with the purpose of mainly satisfying these goals:

Template that can be executed/tested locally, withoud deploying the Lambda to the Cloud, by running Docker image with AWS SAM framework
Abitlity to chose an example event, which can be: SNS, S3 or API Gateway
Template that generates all scaffolding files: LICENSE, AUTHORS, Makefile, SAM template, pre-commit, VERSION, gitlab CI example jobs, envs.json
An opionated template that ideally is meant to be testable locally in Docker encapsulated environment to provice the developer an ability to provide a test event ( ./events/event.json ), an used it to develop Lambda fast by executing it locally numerous times
Minimalistic code example (aka "Hello Works"), with backpack full of batteries included, and minimise the need for an additional configuration as much as possible

How to use

Here is the repo: https://github.com/zahaar/go-lambda-cookiecutter

README.md contains all core instructions and prerequisites needed to use this cookicutter

...
...
First, get Cookiecutter. Trust me, it's awesome::

    $ pip install "cookiecutter>=1.7.0"

Now run it against this repo::
...
...

What this template is Not

No deployment to AWS, leave it to Terraform, Claud Formation or other IaaC tool of your preference
Not a complex example. A complex example would complicate things, and negatively influence initial understanding of the structure. Therefore SNS, S3 and API example, contain a simple print-out as an output.
It's hard to simulate complex scenarios locally with AWS. P.e: the complexity of AWS SAM template will scale tremendously if we would include a complex chain of Lambdas that invoke one another. KISS.

Worth mentioning

Don't treat go serverless programm any different from other programs that you write > CC: Go Time Podcast, Serverless and Go
Take great care about business logic, and it's application. How it will shape Lambda code.
Serverless code should be small, minimalistic. Something in a sense of 50 lines microservice
Striving for Cloud Agnostic serverless code ( e.g something that will work on GPC, Azure and AWS simultaneously ) , Doesn't make sense. That why we aim only for AWS Lambda > CC: Go Time Podcast, Serverless and Go
Why use GoLang for AWS Lambda: link
Collection of cookiecutter from AWS SAM: link
Testing, unit testing, integration testing with AWS SAM is at this moment the best things that industry can offer. IMHO/
In case of failing to pull/build Docker image, refer to this thread

pdb | ipdb in docker-compose, enable interactive debugging

zahaar — Fri, 25 Jun 2021 08:29:44 +0000

pdb is a powerful debugging tool that is included as a Python standard library. It can seriously speed up, our debug process, while resolving certain bugs

for this

Problematic

Trying to initiate the pdb in your docker-compose container will result in the following error.

...
...
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/backend/app.py", line 25, in test_response
    return response
  File "/backend/app.py", line 25, in test_response
    return response
  File "/usr/local/lib/python3.9/bdb.py", line 88, in trace_dispatch
    return self.dispatch_line(frame)
  File "/usr/local/lib/python3.9/bdb.py", line 113, in dispatch_line
    if self.quitting: raise BdbQuit
bdb.BdbQuit

that happens only if you try to run pdb in docker-compose and not via docker run ..., due to up not being an interactive command by design.

more on this topic: https://github.com/docker/compose/issues/4677

you can try in out yourself:

step 1. clone the repo setup repo with the minimal Flask app in docker-compose.
https://github.com/zahaar/docker-pdb-interactive-debugging/
step 2. cd docker-pdb-interactive-debugging
step 3. checkout on docker-error branch
git checkout docker-error
step 4. run docker compose up, or docker-compose up if you are using old docker CLI version
step 5. in a separate shell run curl command.

curl -XGET 'http://localhost:5000/api/v1.0/the_answer'

you would see the expected pdb.DdbQuit exception error:

...
...
  File "/usr/local/lib/python3.9/bdb.py", line 113, in dispatch_line
    if self.quitting: raise BdbQuit
bdb.BdbQuit

Solution

pdb expects an open TTY, so let's add it to our compose file

...
...
    ports:
      - '5000:5000'
    stdin_open: true # für pdb
    tty: true # für pdb

Let's try recreating the steps:

Step 1. git clone https://github.com/zahaar/docker-pdb-interactive-debugging/, but on a main branch this time
Step 2. docker compose up
Step 3. In a separate shell window attach local input to a running container, by running.

docker attach docker-interractive-debugging_api_1 // container name

Note we have this capability only after opening an TTY for our container

Step 4. In a separate shell window run.

curl -XGET 'http://localhost:5000/api/v1.0/the_answer'

Step 5. now in the window where container output was attached you would see, pdb waiting for commands.

you can:

p true_response to print the value of the variable true_response
c to resume normal flow
to all other crazy things with pdb !!!

That's it. Easy.

Important Note: to stop attached window without stopping the main running container, you have to use,

escape sequence: CTRL + p -> CTRL + q

Tip:

checkout pdb commands cheatsheet

git workree, quick intro ( simplify your git workflow! )

zahaar — Sun, 30 May 2021 15:59:11 +0000

Here is a fast example of how to use git worktree, it's extremely helpful and at the same time extremely unpopular git feature.

Surprisingly it has been around for ages, but no one seems to know about, yet fall in love on first try!!!

I, personally discovered it from one of Guido's, tweets:

Guido van Rossum

@gvanrossum

Why did I not know before about "git worktree"? :-( git-scm.com/docs/git-workt…

20:27 PM - 07 Apr 2021

Problematic

to present a solution, we have to understand the problem it tries to solve, before jumping to a solution. Here is a simple scenario:

You are trying to work on your own feature for some time. Meanwhile, a coworker writes to you, with a request to review his work, that he has done on a separate branch of the same repo.

Most likely, your next steps would be:

1 - save your work in stash with command

git stash save <name>

2 - checkout to the branch that coworker wants you to review

git checkout feature-coworkers-branch

3 - there might be a need to install the dependencies, with

pip install -r req.txt
// or 
yarn

4 - after browsing though the branch, and finishing with the review, you would want to revert back to the branch that you have worker on

git checkout feature-your-old-branch

5 - HIDERANCE 1: you have to find && unstash your code. You can run, git stash pop/apply, in case the last stash that you have saved, is the one you need

you might need to run:

git stash list

git stash pop stash{0} 
// or 
git stash apply

please note that git stash apply won't delete the stashed item from the stash list

to look up for just the right stash, saving stash with helps a lot. But still sometimes, in a heat of the moment, while having multiple work processes saved in stash, we get lost, apply the the wrong stash and then another one...

6 - HINDERANCE 2: you might encounter a problem of dependency conflict. The dependencies that were installed in feature-coworkers-branch, might hinder your work, due to requiring different versions, demanding different configuration etc... The solution would be to delete/reinstall them, but that's once again an annoying process.

Note: A different workflow, might be cloning a repo in a different folder, doing the review there and discarding it afterwards. But what about the time it would take to install all the dependencies?

All in all, that's an uncomfortable situation to be in. We have encountered it frequently, it consumes our time and mental power.

Luckily, git have a better way

Solution

worktree is wonderful alternative, let's try to follow the same steps using this feature.

1 - While working on your feature, use worktree command to create a new folder/directory of the same repo, and specify which branch you want to start with

git worktree add ../repo_test feature-42-update-mat-view
# git worktree add <path> <optional:starting-branch-name>

and BOOM, now you are in a separate folder/directory, on a feature-42-update-mat-view branch.

2 - Install all the dependencies necessary to check the work

pip install -r req.txt
// or 
yarn

3 - Write to your colleague, remove rm -rdf repo_test / git worktree remove ../repo_test ( or keep it there ), and get back to folder containing your own work progress.

If a working tree is deleted without using git worktree remove, then its associated administrative files, which reside in the repository (see "DETAILS" below), will eventually be removed automatically ...

That's it!!!

No need to stash the code, revert installed dependencies and dig through stash list.

This way is not only faster, but also, much less error prone!
Having errorless git workflow is especially important to keep the concentration on your work.

Tips && References

1 - Official git documentation is very good

2 - If you create a worktree with intention to make some experimental changes, pass -d flag.

git worktree add -d ../repo_test

this quote from git worktree docs

For instance, git worktree add -d <path> creates a new working tree with a detached HEAD at the same commit as the current branch.

3 - Another good workflow would be is to start off by cloning a bare repo. Then use worktree add ... and have a list of your worktrees in a bare repo.

git clone --bare url...

git --bare docs

DEV Community: zahaar

Generate PDFs from HTML via Puppeteer on AWS Lambda + API Gateway

Preface

Repo -> End Result

zahaar / generate-pdf-lambda

Generate PDF document via Puppeteer running on AWS Lambda

Diagram

Requirements

How to Run

How to Deploy

Requirements and Prerequisites

1. SetUp local AWS SAM Template with Chrome Lambda Layer

2. Configure Puppeteer in Lambda; Supply Template HTML

Result

3. Deploy Lambda + API GW via SAM

Tips

1. Insomnia vs Postman

2. Error: Error building docker image: pull access denied for

3. Consider using latest AWS Lambda feature: Lambda URL

4. Be mindful of HTTP Headers, when dealing with API GW. As this can lead to a huge confusion.

SNS/S3/API AWS Lambda opinionated boilerplate cookicutter

Table of content:

What this template is

How to use

What this template is Not

Worth mentioning

pdb | ipdb in docker-compose, enable interactive debugging

Problematic

Solution

Tip:

git workree, quick intro ( simplify your git workflow! )

Problematic

Solution

Tips && References

2. `Error: Error building docker image: pull access denied for`