DEV Community: AWS Switzerland

Scalable Processing of Swiss PDF Documents using 2D Barcodes on AWS

Arlind Nocaj — Fri, 05 Nov 2021 10:15:53 +0000

Abstract

Processing of scanned documents, like tax and salary statements causes a lot of effort, when performed manually by humans. In this post we look in particular at swiss documents like, tax statements, salary statements, and invoices and how to automate their processing in a scalable way using open source and AWS technologies.

Our extraction approach utilizes 2D barcodes and can be easily extended to other documents or images which contain barcodes.

Introduction

In many industries, like Finance, Insurance, Accounting & Tax, documents are still a primary medium of record keeping, communicating and interacting with other parties. Processing of e.g. scanned documents, like tax and salary statements causes a lot of effort and is error-prone when performed manually by humans.

We show how to build a simple and scalable solution which can process swiss documents, like Zurich tax statements, salary statements, and QR invoices using 2D barcodes. In particular, we extend Document understanding solution (DUS) from AWS Labs to support the processing of these swiss document types by using 2D barcodes. Figure 1 shows two supported example documents, a swiss salary statement and a Zurich tax statement.

In the next section we show how to extend an existing sample solution for our specific case of processing documents with 2D barcodes. After reviewing the barcode extraction we show the user interface in the demo section. Then we discuss the scalability of the system and review the supported documents. Finally, we conclude in last section.

Fig. 1: Swiss salary and tax statement (Zurich). Large 2D barcodes encode the actual data from the document in a structured form.

Understanding Documents with Barcodes

DUS - Document understanding solution is an example application from AWS Labs which illustrates how to utilize multiple AI/Machine Learning capabilities to solve various document processing and understanding cases. It shows how to combine and leverage the power of various AWS technologies like Amazon Textract and Amazon Comprehend to extract information from structured and unstructured scanned pdf documents or images.

DUS works as follows:

A document (pdf or image) gets upload to Amazon S3, which triggers an event and the file gets added to a DynamoDB table for state management.
A change to DynamoDB table triggers an event, which gets processed by an AWS Lambda function and adds the file to the appropriate sync queue (image) and async queue (pdf).
The messages in each queue get processed by a lambda function which calls Amazon Textract and other services for information extraction and storage.

The architecture, see Figure 2, and its resources are being defined with AWS Cloud Development Kit (AWS CDK). AWS CDK is an open source software development framework which allows to define application resources using familiar programming languages.

To extend the solution with an additional pipeline for barcode processing we have to do the following

Add a lambda service for barcode processing which can read a pdf document from S3 and write the extracted barcode data to a specific S3 bucket.
Connect the barcode lambda service to the existing pipeline by extending the AWS CDK components in DUS.
Add additional tab to the frontend to support preview of barcode extraction results.

Fig. 2: Barcode processing pipeline (green box) added to existing Document Understanding Architecture (DUS).

The barcode processing extension from Figure 2 (Steps 1 to 4) can be mapped directly to human readable infrastructure as code using CDK:

1) Create Amazon Simple Queue Service for Barcode Processor

const syncBarcodeJobsQueue = new sqs.Queue(this,
    this.resourceName("SyncBarcodeJobs"), {
    visibilityTimeout: cdk.Duration.seconds(900),
    retentionPeriod: cdk.Duration.seconds(1209600),
    encryption: QueueEncryption.KMS_MANAGED
});

2) Add queue as event source to AWS Lambda function for Barcode Extraction

syncBarcodeProcessor.addEventSource(
    new SqsEventSource(syncBarcodeJobsQueue, {
        batchSize: 1
    })
);

3) Create AWS Lambda function for sync Barcode Processor by using a Dockerfile. That way we can run the image as well locally and debug the code if required using an IDE, like PyCharm.

// Configure path to Dockerfile
const dockerfile = path.join(__dirname, "../lambda/barcodeprocessor");

// Create AWS Lambda function and push image to ECR
const syncBarcodeProcessor = new lambda.DockerImageFunction(this,
    this.resourceName("SyncBarcodeProcessor"), {
    code: lambda.DockerImageCode.fromImageAsset(dockerfile),
    memorySize: 5024,
    timeout: cdk.Duration.minutes(2),
    tracing: lambda.Tracing.ACTIVE,
    environment: {
        OUTPUT_BUCKET: documentsS3Bucket.bucketName,
        OUTPUT_TABLE: outputTable.tableName,
        DOCUMENTS_TABLE: documentsTable.tableName
    }
});

4) Grant read and write permissions to the lambda function so that it can read the documents and store the results

// Permissions for barcode processor
documentsS3Bucket.grantReadWrite(syncBarcodeProcessor);
outputTable.grantReadWriteData(syncBarcodeProcessor);
documentsTable.grantReadWriteData(syncBarcodeProcessor);

Barcode Extraction

We use the docbarcodes package for the barcode extraction process. The extraction consists of multiple steps, since a document can have multiple barcodes distributed over various regions of the page.

The main steps of the barcode extraction are:

Detect candidate barcode regions: An image transformation heuristic based on morphological operations is being used to identify rectangular dark regions, which might contain a barcode.
Extract the raw barcode from candidate regions: The extraction of barcodes from the candidate regions is being performed with zxing, an open source library which supports many variations of 1D and 2D barcodes.
Combine multiple raw barcodes to decode the data: Depending on the document, the data might not fit into one single barcode. Therefore it typically is split across multiple barcodes. These chunks need to be combined in the right order and then a decompression, e.g. like zlib, needs to be applied to receive the original information. An example of a final decoded output, e.g. for the swiss salary statement, can be found below.

The following code shows the main part of the AWS Lambda Barcode Processing function in python. The full code will be available soon in the DUS dev branch. You can preview it at the forked branch DUS dev barcodes

// AWS Lambda function to extract barcodes from PDF documents or images
from docbarcodes.extract import process_document

def processPDF(documentId, features, bucketName, outputBucketName, 
                objectName, outputTableName, documentsTableName):

    with tempfile.TemporaryDirectory() as d:
        target = os.path.join(d, objectName)
        os.makedirs(Path(target).parent, exist_ok=True)

        s3.download_file(bucketName, objectName, target)

        # extract raw and decoded barcodes from document
        barcodes_raw, barcodes_combined = process_document(target, max_pages=None, use_jpype=True)

        # create proper json response
        raw_dict = [b._asdict() for b in barcodes_raw]
        combined_dict = [b._asdict() for b in barcodes_combined]
        response = {"BarcodesRaw": raw_dict, "BarcodesCombined": combined_dict}

        # store result 
        outputPath = '{}{}/{}{}barcodes.json'.format(PUBLIC_PATH_S3_PREFIX, documentId, 
                                                     SERVICE_OUTPUT_PATH_S3_PREFIX,
                                                     BARCODES_PATH_S3_PREFIX)
        print("Generating output for DocumentId: {} and storing in {}".format(documentId, outputPath))

        S3Helper.writeToS3(json.dumps(response, ensure_ascii=False), outputBucketName, outputPath)

        ds = datastore.DocumentStore(documentsTableName, outputTableName)
        ds.markDocumentComplete(documentId)

Barcode JSON Result

The generated json result is shown below. The raw field can contain binary data in ISO-8859-1 format.
The decoded content is shown in BarcodesCombined.

{
  "BarcodesRaw": [
    {
      "page": 0,
      "num_candidate": 2,
      "raw": "\u00b4c\u00fd\u00b8z\u0000\u0002V\u0001\u0003\u0000\u0001\u0000\u0001PK\u0003\u0004\u0014\u0000\b\b\b\u0000\u00e9v\u0003Q\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0004\u0000\u0000\u0000txabM\u0092\u00ddn\u00e20\u0010\u0085_\u00c5\u00f2}b;\u00a1!\u00ac\u001cW\u00e1\u00afEbY\u00d4\u00d0j\u00d5;\u0013\u00dc\u00105\u00b1+\u00db,\u00f0>\u00fb&\u00fbb;\t\u00d0\u00f6\"\u00ce\u00f1\u0099\u0019\u00cf\u00e7\u0091\u00f9\u00fd\u00a9m\u00d0\u001fe]mt\u0086YH1R\u00ba4\u00bbZW\u0019~\u00de\u00cc\u0083\u0014\u00df\u000b\u00beA\u0090\u00a6]\u0086\u00f7\u00de\u007f\u00fc \u00e4x<\u0086\u00eeX;\u00b7SeX\u00ee\u0089+\u00f7\u00aa\u0095\u00c4\u00edHD#J\u00a3\u0088\u0092B6\u00d2\u009e\u00a7\u00aa\u0084\u009f\u00f4p\u00fa\u00e6\u0094\u008f1*\u0016\u00d3\fS\nm\u008a\u00b3{\u00e9$\u00c3\u0082OL\u00fb!\u00f5\u0019=/\u00a6\u00c1x^dx\u00f28\u000bX\u0014\u0087\u00d7\u000f\u00a35 v\u0084kyh$Z)\u00e7\u001be1z|\n\u009e&\u00c1J\u00b6\nj~\u00fd\\/g\u00bf\u00d1\u00acQ\u00ef\u00de\u001a]\u00bf\u00a3\u00fc\u0001\u00a3\u00d7\u00c5:\u00c3q\u00d2\u00b5\u009c,3\u009co\u00bd\u00aa\u009b\u0083\u00aeP\u00e1\u00d5AYPJ\u0003\u008d\u00b7J\u00f9\f\u00cf\u00eaJ\u00d9\u00a3\u00aaP\u0002=\u008d\u00f3[s\u0082\u00b1D\u0003(\u00ae\u00fd9\u00c3\u009b\u00fd\u0001\u00b2\u00d7{\u00a3\u00a1#\u008dc\u0014\u00c5)\u001a\u008c\u00d0\u0090a\"\u00f8\u0085r1EK\u00e9\u00bc\u00ee\u00a9r\u00b5\u00ad1\u009a\u00d7\u00f6\u00d3\u00d0Z\u00de\u00a8\u00e8\u008d\u00ea\u000b`\u00f9\u00ef\u00af\u00ae*\u00e9\u009c\u00f3\u0016V\u0085\u00a2\u00ef$\u00c9\b\n\u0096\u00a6\u0094MOs\u00a3\u001a+\u00ab\u00d1\bv\u00e6\u00a0\u00bd\u00ed\u0002\u0082\u0017/A^\u0004++\u00ba\u0019\u000e\u00ee\u0092a\u0098\u008e(\u000b\u00a3\u0094\u0093\u00cf\u0010'7b\u00c1s\u00c1\u00a7\u00a6\u0004\u00c58\u00b9\u0088\u00ee:\u00b5\u00d9\t\u00fefM+\"\u00ca\u0092\u0080\u00d1\u0080B\u00bc78\u00f4\u00aa\u009b\u00ab\u00cf\u0082\u0098rrq\u00faS\u00fb\u00c2\u0005<\u00a5V\u0089A\u00caX\u001a\u000e!\u00e1j\u00f0\u0007k\u009c\u00bbn\u0092\u0014\u00c6\u0010R\u0088~w\u00f9J\u00f9\u00ab\u00bcK\u00e2A\u00dc'|y\u009c\u0000/\u00d9\u0088\u00ffPK\u0007\bK\u00a6nJ\u00ce\u0001\u0000\u0000\u00c1\u0002\u0000\u0000PK\u0001\u0002\u0014\u0000\u0014\u0000\b\b\b\u0000\u00e9v\u0003QK\u00a6nJ\u00ce\u0001\u0000\u0000\u00c1\u0002\u0000\u0000\u0004\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000txabPK\u0005\u0006\u0000\u0000\u0000\u0000\u0001\u0000\u0001\u00002\u0000\u0000\u0000\u0000\u0002\u0000\u0000\u0000\u0000",
      "format": "PDF_417",
      "points": [...],
      "resultMetadata": {
        "ERROR_CORRECTION_LEVEL": "2",
        "PDF417_EXTRA_METADATA": {...}
      }
    }
  ],
  "BarcodesCombined": [
    {
      "content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?><T xmlns=\"http://www.swissdec.ch/schema/sd/20200220/SalaryDeclarationTxAB\" SID=\"000\" SysV=\"001\"><Company UID-BFS=\"CHE-123.123.123\" Person=\"Paula Nestler\" HR-RC-Name=\"COMPLEX Elektronik AG\" ZIP=\"3600\" CL=\"Abteilung Steuerungen\" Street=\"Eigerweg 6\" Postbox=\"124\" City=\"Thun\" Phone=\"033 238 49 71\"/><PersonID Lastname=\"Aebi\" Firstname=\"Anna\" ZIP=\"3000\" CL=\"\" Street=\"L\u00c3\u00a4nggassstrasse 26\" Postbox=\"690\" Locality=\"\" City=\"Bern 9\" Country=\"\"><SV-AS-Nr>123.4567.8901.28</SV-AS-Nr></PersonID><A><DocID>1</DocID><Period><from>2016-10-01</from><until>2016-11-30</until></Period><Income>48118.70</Income><GrossIncome>68000.00</GrossIncome><NetIncome>56343.00</NetIncome></A></T>",
      "format": "PDF_417",
      "sources": [0]
    }
  ]
}

Decoded Barcode Content

The decoded content from the Salary statement contains structured xml with all the non empty fields from the statement.

<?xml version="1.0" encoding="UTF-8"?>
<T SID="000" SysV="001" xmlns="http://www.swissdec.ch/schema/sd/20200220/SalaryDeclarationTxAB">
    <Company CL="Abteilung Steuerungen" City="Thun" HR-RC-Name="COMPLEX Elektronik AG" Person="Paula Nestler" Phone="033 238 49 71" Postbox="124" Street="Eigerweg 6" UID-BFS="CHE-123.123.123" ZIP="3600"/>
    <PersonID CL="" City="Bern 9" Country="" Firstname="Anna" Lastname="Aebi" Locality="" Postbox="690" Street="Langgassstrasse 26" ZIP="3000">
        <SV-AS-Nr>123.4567.8901.28</SV-AS-Nr>
    </PersonID>
    <A>
        <DocID>1</DocID>
        <Period>
            <from>2016-10-01</from>
            <until>2016-11-30</until>
        </Period>
        <Income>48118.70</Income>
        <GrossIncome>68000.00</GrossIncome>
        <NetIncome>56343.00</NetIncome>
    </A>
</T>

Decoded barcode content for a swiss salary statement containing the fields and values in a structured form.

Demo Preview

Figure 3 shows a preview of the demo for the Zurich tax statement. On the right we can see multiple detected barcode chunks and the final decoded xml content.

Fig. 3: Demo UI - Preview of Zurich Tax statement (left) and the extracted raw barcode chunks (right). The final decoded data is the xml (upper right).

Scalability Discussion

The overall solution is event based and uses AWS Lambda, Amazon DynamoDB, and Amazon Textract, as shown in Figure 2. We revisit the scaling limitations and concurrency quotas of these services and how they can be overcome for particular use cases where higher throughput would be required. Please note that by default even if these quotas are reached, requests will be saved and processed after some time, when there is capacity.

Amazon DynamoDB streams is being used to trigger a Lambda function when a new file gets upload for processing. Lambda polls shards in your DynamoDB stream for records at a base rate of 4 times per second. It received batches of records and forwards them to the Lambda function. If the default settings are not enough, one can increase the throughput by increasing the batch size (up to 10K) and increase the ParallelizationFactor as described in the AWS Lambda with AmazonDynamoDB documentation

AWS Lambda is designed to provide scaling capabilities without any custom engineering in your code. As traffic increases, Lambda increases the number of concurrent executions of your functions. For an initial burst of traffic, the cumulative concurrency in a Region can reach between 500 and 3000 per minute, depending upon the Region. After this initial burst, functions can scale by an additional 500 instances per minute. There is also a default concurrent limit of 1000 per Region, for each AWS account, which can be increased by submitting a request in the AWS Support Center.

If there is more documents coming in at the same time than there is concurrency capacity, these documents will be lying on the queue (SQS) until they get processed.

There is also the possibility to use Provisioned Concurrency if a higher concurrency needs to be guaranteed for a particular use case.

Amazon Textract has service quotas, ranging from 2 to 10 requests per second, depending on the region. If these concurrency quotas are not enough for your particular use case, you can request an increase of these soft limits from the AWS console.

Supported Documents

While our focus was to show how to extract information from swiss tax and salary statements, the same approach can be utilized for other document types like:

Swiss Invoices (QR Code), introduced by Six-group
Swiss and European Covid Certificate
US drivers licenses with MRZ (machine readable PDF417 zone)

Ideally the public authorities related to these documents would publish the defined format for each document as part of an open government strategy, which would promote innovative business creation and innovation for these public services.

For Switzerland at least the following cantons utilize barcodes to encode information for tax declaration and processing: ZH-Zurich, VS-Valais, VD-Vaud, LU-Lucerne, AG-Aargau, TG-Thurgau. This solution can easily be extended to support all of these cantons.

Conclusion

We showed how to extract structured information from various types of swiss documents utilizing 2D barcodes and how to embed this barcode processing in a scalable event-based architecture utilizing AWS Lambda.

Our extension of DUS (Document Understanding Solution) for barcodes allows to deploy a full end-to-end solution using CDK and its CLI commands. The Demo allows to apply the barcode processing on all kind of documents and can be utilized as a starting point to increase automation in document processing workflows.

AWS Latency Optimization for Trading Applications

Scott Gerring — Tue, 24 Aug 2021 15:16:37 +0000

Solutions on AWS that need consistently low latency between components are an interesting class of problem. Cloud platforms virtualize compute and network resources to optimize for more common problems, simultaneously impacting latency. We can readily achieve single digit millisecond latency with solutions built on top of AWS, and for the majority of use cases, this is great, but for some domains, we may want to try and do better.

Despite the virtualization, there are plenty of tools we can reach for to drive our latency down - we just have to look a bit harder for them. A classic use case we often see in Switzerland is high-frequency crypto trading - a new and interesting industry that’s captured the interest of many startups.

Usually when people build solutions on AWS, or any other cloud for that matter, they focus on abstraction - load balancers instead of direct access to servers, layered networking, and so on. These are examples of best practices that we employ to build well-architected solutions - secure, low maintenance, and cost-efficient. In cases like crypto trading though, the latency impact of these architectural decisions can represent a poor trade-off, and other options must be considered.
In this article, we will look at the tools and levers that can be used within the AWS platform to drive down latency, both between instances and across the internet.

We will focus on:

Ensuring our instances have the best chance of low latency in general
Optimizing the latency between the instances we own
Optimizing the latency from our instances to the rest of the world

The options presented here generally represent a poor trade-off for conventional workloads, and a full understanding of their impact on the security, reliability and cost-effectiveness in particular on your workloads should be arrived at before using any of them. The AWS Well-Architected Framework and Tool are great resources to help you understand the impact of these and other architectural decisions on your workload.

1. Choosing an Instance

Instance Families, Generations, and Sizes

The biggest lever we have in terms of improving network latency is the instance family, generation, and size we choose. As we increase the bandwidth assigned to the interface, we may also see a decrease in latency. These changes have a cost impact, and before committing to using a very large top-tier network instance, we should measure it to see if any latency advantage it yields is worth the additional cost for our workload. At the time of writing the highest throughput available is 100 gigabit, which can be obtained within the following instance families:

c5n (e.g., c5n.18xlarge, c5n.metal) - compute optimised
g4dn - GPU instances, useful for machine learning
inf1 - AWS Inferentia, useful for machine learning inferences
m5n - balanced CPU and memory
r5n - memory-optimized

Generally you want the newest instance generation (indicated by the number), an ‘n’ indicating improved networking. The choice between the other details - e.g. memory optimized vs. CPU optimized vs inferentia should be chosen to align with your workload.

The trade-off here means we are inevitably choosing expensive instances to access the highest tier of network performance. Depending on the workload it is likely that these instances will end up underutilized. For typical workloads this would not align with the Well Architected Framework’s Cost Optimization pillar.

Tenancy & Metal Instances

Once we’ve chosen an appropriate instance family and size to match the workload, we move onto the choice of tenancy and metal instances:

	Description	Hypervisor?	Isolation?	Placement Groups?	Pros	Cons
Dedicated Host	Reserve a host and explicitly assign VMs to it	Yes	Explicitly assigned VMs	No	Explicitly group VMs onto host to maximize network performance, full control of all neighbors on host	Issues with host impact all instances on host simultaneously
Dedicated Instance	Ensure that VMs only placed on hosts running VMs for the same account	Yes	Shared with other instances within account	Yes	Can use placement groups to mix with regular instances	Because AWS picks the host for you, cannot guarantee same low latency between instances as dedicated hosts (instances may be spread between multiple hosts), Issues with host may impact all instances simultaneously
Metal	Deploy AMIs to bare metal, without a hypervisor	No	Single instance only	Yes	Direct access to the underlying hardware and and hardware feature sets (such as Intel® VT-x) and the ability to run your own hypervisor.
Shared Tenancy	Deploy instances to hosts potentially shared with other accounts	Yes		Yes	Use placement groups to best-effort place instances near each other, spreading instances across hosts lowers impact of any issues arising on underlying hardware.	Because AWS chooses hosts to place the instances on, there is a likely impact on latency between instances.

Choosing a dedicated host allows us to place our instances on the same hardware which is likely to result in the best latency we can expect between the instances. On the other hand, this means that a host failure will bring down all of our instances at once while EC2 moves the VMs onto a new dedicated host. Depending on your workload, this may be an unacceptable failure mode.

Metal instances remove the hypervisor from the equation completely, allowing us to provision standard AMIs to machines without the hypervisor. With EC2’s nitro hypervisor and hypervisor cards, the overhead added by the hypervisor is minimal, but nonetheless the impact of this may be a deciding factor for our workload. Metal instances tend to be used mainly where licensing constraints require it, rather than as a performance optimisation.

Ultimately the choice here comes down to the particular workload, how it responds to failures, and how far we want to sacrifice the Well Architected Framework’s Reliability pillar to drive our latency numbers down. Once again, the choices made here, focussed on placing the instances as closely together as possible within an AZ, represent a poor trade-off for traditional workloads where reliability is much more important than minimising latency at all costs.

2. Networking - Within the Region

Once we’ve picked our instances and their tenancy, we might need to hook them up to each other. For low-latency workloads, the less connectivity the better - ultimately avoiding hitting the network is always going to be the fastest option!
In some cases - when the workload can’t fit onto a single machine, for instance, or needs connectivity with other, discrete services within your AWS environment - this isn’t practical.

Availability Zones

Placing all of our instances into the same AZ will likely give us the best latency between them, but once again, at the expense of reliability. AZs are physically isolated from one another to improve the durability of the region they reside within, and this comes with a small latency cost. The best practice for traditional workloads is to take advantage of services such as EC2 Autoscaling’s inbuilt ability to spread instances across availability zones, so that in the unlikely event of an AZ outage, the workload continues to run. Keep in mind that AZs remain close enough for things like RDS’ synchronous database replication to work well; it is definitely worth measuring whether or not the improvement here is worth the trade-off!

We must carefully consider whether or not the latency improvements brought by targeting a single AZ only are worth the reliability impact for the workload.

Cluster Placement Groups

Within an AZ, we can go further still and use Cluster Placement Groups to hint to AWS that we want our instances packed closely together on the physical hardware, rather than spread apart - the default behaviour used to improve reliability. It is recommended that all instances for the placement group be launched in a single launch request to increase the likelihood they can be placed effectively.

Network Cards

We need to make sure we are using Enhanced Networking; on the newer instance generations we should have this turned on by default, but it is nonetheless worth checking.

Next, we can look at splitting traffic for different workloads apart onto different interfaces. For instance, for a trading workload, we may choose to separate the interface we use to receive market signals from the interface we use to place orders. This limits the ability of separate streams of traffic to impact each other, and allows us to tune the interfaces separately if required.

Network Cards - Elastic Fabric Adapter

Finally, for traffic within our cluster, we can look at OS bypass networking.

Elastic Fabric Adapter (EFA) is a network card that allows customers to run applications requiring high levels of inter-node communications leveraging operating system bypass. This allows tightly-coupled workloads to achieve improved latency, jitter, and throughput. Using EFAs with their OS bypass functionality imposes some constraints that mean they are not simply a “better ENA”:

Communications within the same subnet only
TCP/IP is not available; applications must leverage a library such as Open MPI for communications

EFAs also provide regular, non-kernel-bypass traffic, which can be used to reach the machine regularly, but as discussed earlier, we likely want to separate this class of traffic out onto separate network interfaces anyway.

Network Simplification

Anything in the path of traffic between our instances can affect latency, and we can therefore try to keep these paths as simple as practical while continuing to meet our other architectural goals.

VPC - use a single VPC where possible; VPC peering, VPN links, and traffic gateway all introduce latency
Subnets - passing traffic between subnets introduces latency. If we intend to use EFA and kernel-bypass, we will need to keep the clustered portion of our workload inside a single subnet anyway
Security Groups - attaching multiple SGs, or overly-complex SGs, may impact latency
Access Control Lists - attaching overly complex ACLs may impact latency

Here too we need to pay special attention to the ramifications of simplification. Simply removing all access control from our network is clearly a step in the wrong direction, particularly in terms of Well Architected’s Security pillar. The problems that we may cause by compromising on security can foreseeably be much more dire than those of our earlier reliability compromises.

Protocol Choice & Serialization

While REST + JSON + HTTP has become the defacto standard of public APIs exposed on the internet, it is much more optimized for common use cases like web applications than it is for low-latency. If we choose to use EFAs and kernel bypass as suggested earlier, we’ve already been pushed away from this for traffic within our cluster, but what if we can’t make this move, or if we have external interfaces?

If we must use HTTP, favour newer HTTP versions - e.g. HTTP3 / QUIC over HTTP2 over HTTP1
Favour gRPC over REST
Favour higher performance serialization options such as protocol buffers over JSON or XML
Favour websockets PUSH over polling - for instance, if we have incoming market signals

3. Networking - Beyond the Region

While it’s nice to have our instances talking to each other as quickly as possible, most real-world systems, especially trading systems, need to interact with the rest of the world. Fortunately there are some tools we can use here, too, to push our latency down.

PrivateLink & VPC Peering

AWS PrivateLink allows you to publish a service from one VPC into another, without having to directly peer the VPCs and without the traffic travelling over the public internet. In some cases, external API providers provide PrivateLink endpoints for their services in the regions and zones they operate in, which you can then use to communicate with them.
PrivateLink exposes a Network Load Balancer from one VPC into the other, which may have a latency impact. If you control both accounts you can also use VPC peering or AWS Transit Gateway, which allows traffic to pass between them without the intermediate NLB. This complicates the security situation and couples the two much more tightly together, but may be an appropriate trade-off in some situations.

Global Accelerator & Cloudfront

Global Accelerator allows you to expose a public endpoint for your AWS-hosted service using BGP Anycast such that traffic for the service enters the AWS network as close to the edge as possible. The traffic then travels through AWS’ internal network to your service, wherever that may be, rather than across the public internet. This can result in a marked improvement in stability and latency of the connection.

CloudFront is a content delivery network that provides HTTP[S] endpoints to your service distributed around Amazon’s global network. Traffic hits CloudFront at the edge, and then travels over AWS’ internal network to your service. This can be a simpler alternative to Global Accelerator where HTTP/HTTPS is all that is required.

In our example here - a trading system - it is likely not much use, as we would much rather place our workload as close to the exchange we are using as possible.

Direct Connect Partners

In case you are interested in arbitrage, you may want the lowest latency possible not just between a single region and an exchange, but to multiple exchanges spread out across the world, the idea being to take advantage of price differences quicker than anyone else can. This of course makes the problem much harder.

Some AWS Direct Connect partners such as Equinix operate their own global networks and may offer latency optimized offerings. If you are interested in exploring this possibility you can checkout our AWS’ Direct Connect partners.

4. Kernel & Process Optimisation

Once we’ve gone through all of the “big picture” pieces above, we can move into the instances themselves and start fiddling with local optimisations.

Core Pinning

Core pinning, sometimes also called CPU pinning or processor affinity, allows us to indicate to our OSes scheduler that a particular process should only run on a particular core. This can help avoid cache misses and improve process latency. This sort of optimization warrants benchmarking to ensure that it yields a positive performance impact. Most X86 AWS instance families maintain affinity between virtualized cores and physical hyperthreads, but this can be verified for the relevant family in the documentation. The T2 instance type and graviton instances like T4g are notable exceptions.

TCP & Network Card Tuning

TCP & network card tuning is a rabbit hole that we can also choose to dive into. Ultimately both have a number of levers and tweaks we can play with - window sizes, buffer sizes, and so on - that are generally tuned to provide reliability for normal use cases. Diving into this is beyond the scope of this document!

Conclusions

If you’ve gotten this far, congratulations! Despite running on virtualized infrastructure, there is still a lot we can do to drive down our latency.

Join the AWS Community Builders program!

Geoffray Schmitt — Tue, 20 Jul 2021 11:53:22 +0000

Are you building on AWS in Switzerland? Would you like to be recognized as an expert and level up your skills, through interaction with your peers? Would you be excited to receive benefits, like AWS free credits, direct interactions with AWS teams, unique training opportunities, cool SWAG and many more?

The AWS Community Builders program offers technical resources, mentorship, and networking opportunities to AWS enthusiasts and emerging thought leaders who are passionate about sharing knowledge and connecting with the technical community.

Check what current members of the community are publishing on dev.to: https://dev.to/aws-builders

Apply today to become an AWS Community Builders member and join an amazing Swiss and global community of builders:
https://aws.amazon.com/developer/community/community-builders/

Anonymize your data using Amazon S3 Object Lambda

Jérôme Van Der Linden — Fri, 09 Jul 2021 14:59:12 +0000

Introduction

Personal Data in Europe, Personal Identifiable Information (PII) in the US, Client Identifying Data (CID) here in Switzerland, … Whatever the name you give it, the definition is slightly the same: it defines a category of information about an individual that can be used to unambiguously distinguish or trace his/her identity.
For example, passport numbers, social security numbers, IBAN or biometric records, known as direct identifying data, clearly identify an individual. Full names, addresses, phone numbers, dates of birth or emails can also be used to identify someone. However, as they can be shared by several people, you need to combine them to explicitly identify an individual - we say they are indirect identifying data.
This data, when maintained by a company, especially highly regulated one (Financial Services, Healthcare, …) or governing agency must comply with security standard and compliance certifications (GDPR, HIPAA, FINMA circulars, …). These certifications require this kind of data to be highly protected, from public leakage of course, but also internally from your own employees.
In this article, I will mainly focus on this second point. Today more than ever, data is key is to take appropriate decisions, create new services or improve existing ones. And if you want to share data internally, in order to build clever solutions, leveraging analytics and machine learning for example, you need to keep control on that data and ensure it remains compliant with aforementioned certifications.

Anonymization / pseudonymization

Anonymization or pseudonymization are some of the technics commonly adopted to do protect some data. In both case, you want to remove the ability to identify someone and more important the link to his personal information (financial, health, preferences…), while keeping the data practically useful. Anonymization consists in removing any direct (and part of indirect) identifying data. Pseudonymization does not remove these information but modify them so that we cannot make a link with the original individual.

Multiple papers, algorithms (k-anonymity) and technics exist to perform anonymization and pseudonymization. AWS also provides 2 functions — available in the Serverless Application Repository (SAR) — that use Amazon Comprehend and its ability to detect PII:

On my side, as the input file is pretty straightforward, I don’t need Comprehend to detect sensible information.

Here is my (naive) approach:

Remove any (identifying) field that is not useful to the downstream process. In my example, the SSN (social security number) is clearly useless for a data analytics application or to perform machine learning. Same thing for the phone number, address and name.
Remove some precision, by extracting only the meaningful part. For example, we don’t need the exact date of birth, an age may be enough.
If for any reason, we need to keep some identifying fields, then we must pseudonymize them. For example, we can replace the name with another, randomly generated.

After this process, we should end up with the following information, clear from any identifying information (names have been replaced):

Now that we know what we want to do, let’s see it in the context of our workload.

Architecture

We have 3 main components in our workload:

A confidential application, that deal with these data, used by doctors and other medical staff. In that case, the data is not anonymized.
A storage area (Amazon S3), where the data is kept as CSV files for further analytics. Raw data (with identifying information) is kept and protected with appropriate policies.
Another application, used to perform some analytics on this data (without identifying information). Actually, there could be many more applications like this with each their specific requirements and compliance rules.

To provide anonymized data to these applications, we have several options:

Create and maintain as many copies as there are applications with different requirements so that each one has its own version of the data.
Build and manage a proxy layer with additional infrastructure, so that you can manage this anonymization process between S3 and the target application.

Both options add complexity and costs. So this is were I introduce S3 Object Lambda, a capability recently announced by AWS and that will actually act as this proxy. Except that you don’t have to manage any infrastructure, just your Lambda function(s).

Implementation

Let’s implement this solution. First thing to do is to create a Lambda function. To do so, use your preferred framework (SAM, Serverless, CDK, …). I use SAM and my function is in Python 3.8.

The function must have permission to WriteGetObjectResponse, in order to provide the response to downstream application(s). Note this is not in the s3 namespace but s3-object-lambda:

{
    "Action":  "s3-object-lambda:WriteGetObjectResponse",
    "Resource": "*",
    "Effect": "Allow",
    "Sid": "WriteS3GetObjectResponse"
}

And here is the code of my function (commented to understand the details):

My Lambda function is really simple and if you would like to get something more production-ready, I encourage you to have a look at the AWS samples, mentioned above.

Once the function is created and deployed, we need to create an Access Point. Amazon S3 Access Points simplify managing data access for applications using shared data sets on S3, exactly what we want to do here. Using the AWS CLI:

aws s3control create-access-point --account-id 012345678912 --name anonymized-access --bucket my-bucket-with-cid

Then we create the Object Lambda Access Point. It will make the Lambda function act as a proxy to your access point. To do so with the AWS CLI, we need a JSON file. Be sure to replace with your account id, region, access point name (previously created) and function ARN:

Finally, we create the Object Lambda Access Point using the following command:

aws s3control create-access-point-for-object-lambda --account-id 012345678912 --name anonymize-lambda-accesspoint --configuration file://anonymize-lambda-accesspoint.json

And that’s it! You can now test your access point and the anonymization process with a simple get. Note that you don’t perform a get directly on the S3 bucket, but on the access point previously created, using its ARN, just like that:

aws s3api get-object --bucket arn:aws:s3-object-lambda:eu-central-1:012345678912:accesspoint/anonymize-lambda-accesspoint --key patients.csv ./anonymized.csv

You can now provide this access point ARN to the analytics application so it can retrieve anonymized data and perform whatever it needs to.

Conclusion

In this article, I’ve shared how to leverage S3 Object Lambda in order to anonymize your data. In just a few commands and a bit of code, we can safely share data containing identifying information with other applications without duplicating it or building a complex infrastructure.

Note that you can use the same technology to enrich some data (retrieving information in a database), or modify it on the fly (eg. image resizing), or modifying the format (eg. xml to json, csv to parquet, …), and I guess you will find some usage too.

The code of this article is available in github, together with a full sam template to create everything (bucket, access points and Lambda function).

Photos by Jason Dent and Tarik Haiga on Unsplash

Monitoring and instrumenting a multi-stage SQS pipeline using Cloudwatch

Eric BIANCHI — Wed, 07 Jul 2021 10:29:42 +0000

Many of our AWS customers in Switzerland are doing some data ingestion right now, especially in the Financial Service Industry (FSI) where they are massively web scrapping all kind of data across different type of API and websites. We are talking hundreds of thousands of API calls per day. Building data ingestion pipelines at this scale requires setting up different layers and stages that need to be synchronized, monitored and governed.

There are different ways to build data ingestion pipelines in AWS. Reference architectures and solutions are available all over the place. In this article, we will discuss a data ingestion pipeline based on Amazon Simple Queue Service (SQS) and AWS Lambda functions.

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications while AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers. Running these 2 services together removes all the heavy lifting of setting up complex data ingestion and processing workloads.

Let’s take a really world use case of a multi-stage data ingestion pipelines as shown on the architecture diagram below:

The first queue (SQS 1) is filled-in very quickly with a list of tasks (it can be URLs with parameters, it can includes some specific instructions, some DSL, it can be anything really). Now the first lambdas are being triggered and start querying, processing, doing some data transformation and write their output in a new SQS queue (SQS 2). At this stage you can add other layered queues (SQS3, SQS4 … SQS-N). The number of queues really depends of your needs and use cases: for example you would like to enforce some separation of concerns or because the lambda function constrains the execution time.

Challenge

You can repeat this pattern over and over but at some point, you want to make sure that all the data pipeline processing is over so you can trigger the next action. You will face the following question: "How do I know that I'm done" or putting it differently:

"How do I know that all messages in the queues have been processed and that no lambda is still running”

They are different options available, which include using step functions, managing a static counter in a DynamoDB table (or in any other kind of storage), writing the final results directly on S3, etc. For the purpose of this article, we assume the end result is written in SQS and I will show a stateless way of triggering your services when your data is ready, relying only on CloudWatch.

Setting-up the stage

A producer (Lambda 1) is filling up a first SQS queue (SQS 1) with a list of instructions from different sources. This Lambda function is triggered manually or by a cron expression. The first queue is filled-in really quickly. For the purpose of the article, we will fill-in the SQS queue with 100 messages containing a random integer :

import boto3
from random import random

def handler_name(event, context): 
    sqs = boto3.client('sqs')
    queue = get_queue_by_name(QueueName='SQS1')

    for _ in range(100):
        response = sqs.send_message(QueueUrl=queue['QueueUrl'], MessageBody=str(random()))

    return ''

At second stage, Lambda functions (lambda2) are polling the first queue, doing some data processing and transformation and filling up a new queue (SQS2) with the results (we just copy the message as-is here):

import boto3

def handler_name(event, context): 
    sqs = boto3.client('sqs')
    queue_source = get_queue_by_name(QueueName='SQS1')
    queue_target = get_queue_by_name(QueueName='SQS2')

    while True:   
        response = sqs.receive_message(QueueUrl=queue_source['QueueUrl'])

        if 'Messages' not in response.keys():
            break

        message = response['Messages'][0]
        receipt_handle = message['ReceiptHandle']
        sqs.delete_message(QueueUrl=queue_source['QueueUrl'], ReceiptHandle=receipt_handle)
        sqs.send_message(QueueUrl=queue_target['QueueUrl'], MessageBody=message['Body'])

Solution Overview

First, let's reframe the problem as:

“In a given time window (the time it takes for the whole pipeline to execute), if I have sent the same number of messages to the first SQS queue and the last SQS queue, then it means that I have processed all my messages and that all lambdas are done running ”

So let’s go to CloudWatch > Metrics > SQS > Queue Metrics and pick the right metrics:

a first metric for my first queue which is NumberOfMessagesSent in firstQueue (id = m1)
a second metric for my second queue which is NumberOfMessagesSent in secondQueue (id = m2)

Then we create a math expression e1, which compares both numbers and return true if they are equals and greater than 0 on my time window: IF(m1 == m2 AND m1 > 0, 1, 0)

You can see on the screenshot the queues being filled with 100 messages and being emptied in a 5-minute-time-window.

On the following screenshot, you can see that both queues have received the same number of messages in the specified time interval and that e1, our math expression comparing the number of messages in both queue is True.

You can build more complex expressions taking into account your different stages and dead letter queues if needed, but the methodology remains the same.

Next we set an alarm on e1, which triggers a SNS notification if the condition criteria is met (on the given time window), then we can trigger our final workload based on the SNS Notification:

Now you can trigger a lambda function that will launch the last piece of your multi-stage pipeline.

Note that it works with many other services where data is flowing in and out of these services, such as Amazon Kinesis, AWS API Gateway, etc.

Do you want to learn more? Passing your AWS Certified Sysops Administrator - Associate Certification is a great way to learn how to leverage one of the most powerful AWS service out there: Amazon CloudWatch.

Have fun with CloudWatch!
-Eric

Grant limited IAM permissions with AWS CDK - but fast!

Ramon — Tue, 15 Jun 2021 15:46:20 +0000

I love writing CDK apps but there is something I really loathe: Writing IAM policies and assigning them to resources like Lambdas. It feels like a step back because I don't have the same type-safety and syntax highlighting that I'm used to from native CDK constructs.

When I start writing an app, I want to make fast progress, so I fall back to using AWS Managed IAM policies, which are usually too broad and open. Then I add TODO's and promise myself to take care of it later...

Like in this example where the Lambda should only be able to read from a single bucket, but is granted with full S3 permissions:

const bucket = new s3.Bucket(...)

const executionRole = new iam.Role(this, "execution-role", {
  assumedBy: new iam.ServicePrincipal("lambda.amazonaws.com"),
  managedPolicies: [
    iam.ManagedPolicy.fromAwsManagedPolicyName(
      "service-role/AWSLambdaBasicExecutionRole"
    ),
    iam.ManagedPolicy.fromAwsManagedPolicyName(
      "service-role/AWSLambdaVPCAccessExecutionRole"
    ),
    //TODO lock down permissions!
  iam.ManagedPolicy.fromAwsManagedPolicyName("AmazonS3FullAccess"),
  ],
})

const fn = new lambda.Function(this, "MyLambda", {
  runtime: lambda.Runtime.PYTHON_3_8,
  code: ...,
  handler: ...,
  timeout: ...,
  role: executionRole
})

Wouldn't it be nice if there was an idiomatic way of granting permissions in CDK? Turns out, there is!

Many CDK constructs representing resources that can be accessed (like S3, DynamoDB, SQS, SNS...) provide various .grant...() methods (see CDK Grants). With this method you don't have to memorize or look up the names of any of the AWS Managed IAM policies and it's incredibly succinct:

const bucket = new s3.Bucket(...)

const fn = new lambda.Function(this, "MyLambda", {
  runtime: lambda.Runtime.PYTHON_3_8,
  code: ...,
  handler: ...,
  timeout: ...
})

bucket.grantRead(fn)

Note how I was able to remove the executionRole completely because CDK will chose a sensible default Execution Role that will include permissions to invoke the Lambda.
In the last line I just use the bucket object and call grantRead to give the Lambda permissions to read from this bucket only.

Not only is this easier and faster to write, it also makes our infrastructure more secure! Besides, you can always fall back to explicitly using IAM Roles and Policies if you encounter CDK constructs where grant isn't available.

Learn more

Other CDK modules follow similar mechanisms, for example @aws-cdk/aws-ec2 when it comes to allowing connections. No more manual writing of Security Group rules!

CDK is evolving fast and getting better by the day. If you want to dive deeper, check out the CDK Workshop for some hands-on learning and the CDK Construct Catalog to find hundreds of reusable CDK constructs from AWS and other developers.

Have fun building!
-Ramon