DEV Community: Arlind Nocaj

Debugging Python Code in Amazon SageMaker Locally Using Visual Studio Code and PyCharm: A Step-by-Step Guide

Arlind Nocaj — Wed, 15 Nov 2023 12:03:54 +0000

Introduction to Amazon SageMaker and Local Debugging

Many data scientist work on custom models and scripts. Efficient debugging and browsing the code using IDE’s like Visual Studio Code (vscode) or Pycharm enables fast model development and iteration. On the other side data scientist need to be able to train and deploy increasingly larger machine learning models, for which high compute and storage resources are required.

Amazon SageMaker is a flexible machine learning platform that allows you to more effectively build, train, and deploy machine learning models in production. Amazon SageMaker Python SDK supports local mode, which allows you to create pipelines or estimators and deploy them to your local environment. This is a great way to test your deep learning scripts before running them in SageMaker’s managed training or hosting environments. Local Mode is supported for frameworks images (TensorFlow, MXNet, Chainer, PyTorch, and Scikit-Learn) and your custom container images, see this blog for an introduction to the topic.

In this blog we show how you can run and debug your python code and Sagemaker Pipelines locally using VS Code or PyCharm Professional.

Setting Up Your Development Environment

First lets make sure you have the prerequisites installed and ready

Install python, e.g. as described here or here
Install Docker Desktop (Windows, Mac, or Linux) and docker-compose.
Install your favorite IDE, e.g. VS code or PyCharm Professional

If you have Windows I would recommend to utilize the Linux subsystem (WSL).
Note that the approach described here should also work on Ubuntu or other Linux systems.

Lets check out an example Sagemaker pipeline which contains multiple steps and start jupyter lab.



git clone https://github.com/aws-samples/amazon-sagemaker-local-mode/
cd amazon-sagemaker-local-mode/general_pipeline_local_debug
python3 -m venv .venv
source .venv/bin/activate
pip install jupyter
jupyter lab

Open sagemaker-pipelines-local-mode-debug.ipynb, which defines an example pipeline with multiple steps
Make sure that you have your AWS credentials set up and define the right profile in the first cell of the notebook.
Define your Sagemaker execution role default_sagemaker_execution_role

Debugging SageMaker Python Scripts with VS Code

The main steps to debug with VSCode are:

Add code to start debug server
Run the Sagemaker pipeline so that the container with the debug code gets started
Connect VS Code to the running container and attach the debugger to the waiting debug server

Add code to start debug server

The first pipeline step is a preprocessing step for feature engineering. We want to debug this preprocessing.py while it is executed as part of Sagemaker pipeline. Lets add a hook to start the debug server when the code will run within a docker container. First we need to install the debugpy package. Then we can import and start the listener with debugpy.
Note that you could also add the debugpy package to your requirements.txt or any other approach that makes sure that the package is available for the python execution environment. For simplicity we install it directly from within the python code.



%%writefile code/preprocessing.py
...

# For vscode debugging
import sys
import subprocess
# install debugpy
subprocess.check_call([sys.executable, "-m", "pip", "install", "debugpy"])
# required only for this specific scikit learn container to avoid a package conflict
subprocess.check_call([sys.executable, "-m", "pip", "uninstall", "-y", "typing"])

# import and run the debug hook from debugpy 
import debugpy
debugpy.listen(("0.0.0.0",5678))
debugpy.wait_for_client()  # blocks execution until client is attached
breakpoint() # add breakpoint

Run the Sagemaker pipeline to start the container with the debug code

We can execute all cells of the notebook. This will create a sagemaker pipeline and run it in local mode using docker and docker-compose. The preprocessing job will wait for the debugger to connect.

Run → Run All Cells

The first time when you run the Sagemaker pipeline locally, it might take some time to download the required container images. After the first time the steps will execute much faster, since docker caches the images.

Connect VS Code to the running container and attach the debugger to the waiting debug server

First, lets start VS Code. Click the open remote window button (lower left). Then Attach to the running container.
VS Code will attach to the running container and install vscode server into it. Then it will open a new window which is attached to that container. You can now browse or explore the directories of the container if you want. The preprocessing step code is located in /opt/ml/processing/input/code/preprocessing.py.

To start debugging we have to create a debug configuration and hit the play button to run the debugger with remote attach. This will bring us to the breakpoint which we defined in preprocessing.py.

Great, you are now able to debug your custom code step by step. The gif below shows these steps visually.

Debugging SageMaker Python Scripts with PyCharm

The debugging in PyCharm Professional works slightly different compared to VS Code. Instead of connecting from the IDE towards the code with a listener, in PyCharm the code connects to the debug server that has to be started in the IDE.
In the following we want to debug our preprocessing.py.

The approach is the following:

Start PyCharm Debug Server
Add the debug code to your python file
Run Sagemaker pipeline which automatically connects to the debug server

Start PyCharm Debug Server

First we need to start PyCharm and its Debug Server for python.
We create a new python project in PyCharm and then a new debug configuration. We select a Python Debug Server.
See the screenshots below for a visual flow.

Add the debug code to your python file, e.g. preprocessing.py

Lets add the code to the python file that we want to debug.



%%writefile code/preprocessing.py
...
# For PyCharm debugging
import sys
import subprocess
subprocess.check_call([sys.executable, "-m", "pip", "install", "pydevd-pycharm~=232.10203.26"])
import pydevd_pycharm
pydevd_pycharm.settrace('host.docker.internal', port=8200, stdoutToServer=True, stderrToServer=True) # host.docker.internal to route from container to your host
breakpoint()

Make sure that you have defined the right version of pydevd-pycharm package (e.g. pydevd-pycharm~=232.10203.26. The version needs to match the version shown above in the debug configuration screen of PyCharm.

Note that you could also add the specific pydevd-pycharm~=232.10203.26 package to your requirements.txt or use any other approach that makes sure that the package is available for the python execution environment. For simplicity we install it directly from within the python code.

Run Sagemaker pipeline which automatically connects to the debug server

Now, we can create and run the pipeline by executing all the cells in the notebook. This will create a sagemaker pipeline and run it in local mode using docker and docker-compose. The preprocessing job will attach now to the debug server from PyCharm.

Run → Run All Cells

Ones the container with the preprocessing step is running, it will connect to the PyCharm Debug Server and we will be able to debug our code. We just have to tell PyCharm, whether it should auto-detect the location of our python source file (preprocessing.py) or whether it should download the file from the running container. The animation with screenshots below shows this process.

In the same way you can also debug all the other steps of the pipeline, e.g. training (abalone.py), evaluation.py or inference.py.

Conclusion

We showed how you can run a full sagemaker pipeline with preprocessing, training, evaluation, and inference step in local mode and how to debug it locally with VS Code or PyCharm Professional. This allows you to iterate fast on your code and gain speed.

The same pipeline can now be also executed in the cloud by a simple switch parameter (run_locally=False), which allows you to use additional cloud compute and storage resources on AWS.

Scalable Processing of Swiss PDF Documents using 2D Barcodes on AWS

Arlind Nocaj — Fri, 05 Nov 2021 10:15:53 +0000

Abstract

Processing of scanned documents, like tax and salary statements causes a lot of effort, when performed manually by humans. In this post we look in particular at swiss documents like, tax statements, salary statements, and invoices and how to automate their processing in a scalable way using open source and AWS technologies.

Our extraction approach utilizes 2D barcodes and can be easily extended to other documents or images which contain barcodes.

Introduction

In many industries, like Finance, Insurance, Accounting & Tax, documents are still a primary medium of record keeping, communicating and interacting with other parties. Processing of e.g. scanned documents, like tax and salary statements causes a lot of effort and is error-prone when performed manually by humans.

We show how to build a simple and scalable solution which can process swiss documents, like Zurich tax statements, salary statements, and QR invoices using 2D barcodes. In particular, we extend Document understanding solution (DUS) from AWS Labs to support the processing of these swiss document types by using 2D barcodes. Figure 1 shows two supported example documents, a swiss salary statement and a Zurich tax statement.

In the next section we show how to extend an existing sample solution for our specific case of processing documents with 2D barcodes. After reviewing the barcode extraction we show the user interface in the demo section. Then we discuss the scalability of the system and review the supported documents. Finally, we conclude in last section.

Fig. 1: Swiss salary and tax statement (Zurich). Large 2D barcodes encode the actual data from the document in a structured form.

Understanding Documents with Barcodes

DUS - Document understanding solution is an example application from AWS Labs which illustrates how to utilize multiple AI/Machine Learning capabilities to solve various document processing and understanding cases. It shows how to combine and leverage the power of various AWS technologies like Amazon Textract and Amazon Comprehend to extract information from structured and unstructured scanned pdf documents or images.

DUS works as follows:

A document (pdf or image) gets upload to Amazon S3, which triggers an event and the file gets added to a DynamoDB table for state management.
A change to DynamoDB table triggers an event, which gets processed by an AWS Lambda function and adds the file to the appropriate sync queue (image) and async queue (pdf).
The messages in each queue get processed by a lambda function which calls Amazon Textract and other services for information extraction and storage.

The architecture, see Figure 2, and its resources are being defined with AWS Cloud Development Kit (AWS CDK). AWS CDK is an open source software development framework which allows to define application resources using familiar programming languages.

To extend the solution with an additional pipeline for barcode processing we have to do the following

Add a lambda service for barcode processing which can read a pdf document from S3 and write the extracted barcode data to a specific S3 bucket.
Connect the barcode lambda service to the existing pipeline by extending the AWS CDK components in DUS.
Add additional tab to the frontend to support preview of barcode extraction results.

Fig. 2: Barcode processing pipeline (green box) added to existing Document Understanding Architecture (DUS).

The barcode processing extension from Figure 2 (Steps 1 to 4) can be mapped directly to human readable infrastructure as code using CDK:

1) Create Amazon Simple Queue Service for Barcode Processor

const syncBarcodeJobsQueue = new sqs.Queue(this,
    this.resourceName("SyncBarcodeJobs"), {
    visibilityTimeout: cdk.Duration.seconds(900),
    retentionPeriod: cdk.Duration.seconds(1209600),
    encryption: QueueEncryption.KMS_MANAGED
});

2) Add queue as event source to AWS Lambda function for Barcode Extraction

syncBarcodeProcessor.addEventSource(
    new SqsEventSource(syncBarcodeJobsQueue, {
        batchSize: 1
    })
);

3) Create AWS Lambda function for sync Barcode Processor by using a Dockerfile. That way we can run the image as well locally and debug the code if required using an IDE, like PyCharm.

// Configure path to Dockerfile
const dockerfile = path.join(__dirname, "../lambda/barcodeprocessor");

// Create AWS Lambda function and push image to ECR
const syncBarcodeProcessor = new lambda.DockerImageFunction(this,
    this.resourceName("SyncBarcodeProcessor"), {
    code: lambda.DockerImageCode.fromImageAsset(dockerfile),
    memorySize: 5024,
    timeout: cdk.Duration.minutes(2),
    tracing: lambda.Tracing.ACTIVE,
    environment: {
        OUTPUT_BUCKET: documentsS3Bucket.bucketName,
        OUTPUT_TABLE: outputTable.tableName,
        DOCUMENTS_TABLE: documentsTable.tableName
    }
});

4) Grant read and write permissions to the lambda function so that it can read the documents and store the results

// Permissions for barcode processor
documentsS3Bucket.grantReadWrite(syncBarcodeProcessor);
outputTable.grantReadWriteData(syncBarcodeProcessor);
documentsTable.grantReadWriteData(syncBarcodeProcessor);

Barcode Extraction

We use the docbarcodes package for the barcode extraction process. The extraction consists of multiple steps, since a document can have multiple barcodes distributed over various regions of the page.

The main steps of the barcode extraction are:

Detect candidate barcode regions: An image transformation heuristic based on morphological operations is being used to identify rectangular dark regions, which might contain a barcode.
Extract the raw barcode from candidate regions: The extraction of barcodes from the candidate regions is being performed with zxing, an open source library which supports many variations of 1D and 2D barcodes.
Combine multiple raw barcodes to decode the data: Depending on the document, the data might not fit into one single barcode. Therefore it typically is split across multiple barcodes. These chunks need to be combined in the right order and then a decompression, e.g. like zlib, needs to be applied to receive the original information. An example of a final decoded output, e.g. for the swiss salary statement, can be found below.

The following code shows the main part of the AWS Lambda Barcode Processing function in python. The full code will be available soon in the DUS dev branch. You can preview it at the forked branch DUS dev barcodes

// AWS Lambda function to extract barcodes from PDF documents or images
from docbarcodes.extract import process_document

def processPDF(documentId, features, bucketName, outputBucketName, 
                objectName, outputTableName, documentsTableName):

    with tempfile.TemporaryDirectory() as d:
        target = os.path.join(d, objectName)
        os.makedirs(Path(target).parent, exist_ok=True)

        s3.download_file(bucketName, objectName, target)

        # extract raw and decoded barcodes from document
        barcodes_raw, barcodes_combined = process_document(target, max_pages=None, use_jpype=True)

        # create proper json response
        raw_dict = [b._asdict() for b in barcodes_raw]
        combined_dict = [b._asdict() for b in barcodes_combined]
        response = {"BarcodesRaw": raw_dict, "BarcodesCombined": combined_dict}

        # store result 
        outputPath = '{}{}/{}{}barcodes.json'.format(PUBLIC_PATH_S3_PREFIX, documentId, 
                                                     SERVICE_OUTPUT_PATH_S3_PREFIX,
                                                     BARCODES_PATH_S3_PREFIX)
        print("Generating output for DocumentId: {} and storing in {}".format(documentId, outputPath))

        S3Helper.writeToS3(json.dumps(response, ensure_ascii=False), outputBucketName, outputPath)

        ds = datastore.DocumentStore(documentsTableName, outputTableName)
        ds.markDocumentComplete(documentId)

Barcode JSON Result

The generated json result is shown below. The raw field can contain binary data in ISO-8859-1 format.
The decoded content is shown in BarcodesCombined.

{
  "BarcodesRaw": [
    {
      "page": 0,
      "num_candidate": 2,
      "raw": "\u00b4c\u00fd\u00b8z\u0000\u0002V\u0001\u0003\u0000\u0001\u0000\u0001PK\u0003\u0004\u0014\u0000\b\b\b\u0000\u00e9v\u0003Q\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0004\u0000\u0000\u0000txabM\u0092\u00ddn\u00e20\u0010\u0085_\u00c5\u00f2}b;\u00a1!\u00ac\u001cW\u00e1\u00afEbY\u00d4\u00d0j\u00d5;\u0013\u00dc\u00105\u00b1+\u00db,\u00f0>\u00fb&\u00fbb;\t\u00d0\u00f6\"\u00ce\u00f1\u0099\u0019\u00cf\u00e7\u0091\u00f9\u00fd\u00a9m\u00d0\u001fe]mt\u0086YH1R\u00ba4\u00bbZW\u0019~\u00de\u00cc\u0083\u0014\u00df\u000b\u00beA\u0090\u00a6]\u0086\u00f7\u00de\u007f\u00fc \u00e4x<\u0086\u00eeX;\u00b7SeX\u00ee\u0089+\u00f7\u00aa\u0095\u00c4\u00edHD#J\u00a3\u0088\u0092B6\u00d2\u009e\u00a7\u00aa\u0084\u009f\u00f4p\u00fa\u00e6\u0094\u008f1*\u0016\u00d3\fS\nm\u008a\u00b3{\u00e9$\u00c3\u0082OL\u00fb!\u00f5\u0019=/\u00a6\u00c1x^dx\u00f28\u000bX\u0014\u0087\u00d7\u000f\u00a35 v\u0084kyh$Z)\u00e7\u001be1z|\n\u009e&\u00c1J\u00b6\nj~\u00fd\\/g\u00bf\u00d1\u00acQ\u00ef\u00de\u001a]\u00bf\u00a3\u00fc\u0001\u00a3\u00d7\u00c5:\u00c3q\u00d2\u00b5\u009c,3\u009co\u00bd\u00aa\u009b\u0083\u00aeP\u00e1\u00d5AYPJ\u0003\u008d\u00b7J\u00f9\f\u00cf\u00eaJ\u00d9\u00a3\u00aaP\u0002=\u008d\u00f3[s\u0082\u00b1D\u0003(\u00ae\u00fd9\u00c3\u009b\u00fd\u0001\u00b2\u00d7{\u00a3\u00a1#\u008dc\u0014\u00c5)\u001a\u008c\u00d0\u0090a\"\u00f8\u0085r1EK\u00e9\u00bc\u00ee\u00a9r\u00b5\u00ad1\u009a\u00d7\u00f6\u00d3\u00d0Z\u00de\u00a8\u00e8\u008d\u00ea\u000b`\u00f9\u00ef\u00af\u00ae*\u00e9\u009c\u00f3\u0016V\u0085\u00a2\u00ef$\u00c9\b\n\u0096\u00a6\u0094MOs\u00a3\u001a+\u00ab\u00d1\bv\u00e6\u00a0\u00bd\u00ed\u0002\u0082\u0017/A^\u0004++\u00ba\u0019\u000e\u00ee\u0092a\u0098\u008e(\u000b\u00a3\u0094\u0093\u00cf\u0010'7b\u00c1s\u00c1\u00a7\u00a6\u0004\u00c58\u00b9\u0088\u00ee:\u00b5\u00d9\t\u00fefM+\"\u00ca\u0092\u0080\u00d1\u0080B\u00bc78\u00f4\u00aa\u009b\u00ab\u00cf\u0082\u0098rrq\u00faS\u00fb\u00c2\u0005<\u00a5V\u0089A\u00caX\u001a\u000e!\u00e1j\u00f0\u0007k\u009c\u00bbn\u0092\u0014\u00c6\u0010R\u0088~w\u00f9J\u00f9\u00ab\u00bcK\u00e2A\u00dc'|y\u009c\u0000/\u00d9\u0088\u00ffPK\u0007\bK\u00a6nJ\u00ce\u0001\u0000\u0000\u00c1\u0002\u0000\u0000PK\u0001\u0002\u0014\u0000\u0014\u0000\b\b\b\u0000\u00e9v\u0003QK\u00a6nJ\u00ce\u0001\u0000\u0000\u00c1\u0002\u0000\u0000\u0004\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000txabPK\u0005\u0006\u0000\u0000\u0000\u0000\u0001\u0000\u0001\u00002\u0000\u0000\u0000\u0000\u0002\u0000\u0000\u0000\u0000",
      "format": "PDF_417",
      "points": [...],
      "resultMetadata": {
        "ERROR_CORRECTION_LEVEL": "2",
        "PDF417_EXTRA_METADATA": {...}
      }
    }
  ],
  "BarcodesCombined": [
    {
      "content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?><T xmlns=\"http://www.swissdec.ch/schema/sd/20200220/SalaryDeclarationTxAB\" SID=\"000\" SysV=\"001\"><Company UID-BFS=\"CHE-123.123.123\" Person=\"Paula Nestler\" HR-RC-Name=\"COMPLEX Elektronik AG\" ZIP=\"3600\" CL=\"Abteilung Steuerungen\" Street=\"Eigerweg 6\" Postbox=\"124\" City=\"Thun\" Phone=\"033 238 49 71\"/><PersonID Lastname=\"Aebi\" Firstname=\"Anna\" ZIP=\"3000\" CL=\"\" Street=\"L\u00c3\u00a4nggassstrasse 26\" Postbox=\"690\" Locality=\"\" City=\"Bern 9\" Country=\"\"><SV-AS-Nr>123.4567.8901.28</SV-AS-Nr></PersonID><A><DocID>1</DocID><Period><from>2016-10-01</from><until>2016-11-30</until></Period><Income>48118.70</Income><GrossIncome>68000.00</GrossIncome><NetIncome>56343.00</NetIncome></A></T>",
      "format": "PDF_417",
      "sources": [0]
    }
  ]
}

Decoded Barcode Content

The decoded content from the Salary statement contains structured xml with all the non empty fields from the statement.

<?xml version="1.0" encoding="UTF-8"?>
<T SID="000" SysV="001" xmlns="http://www.swissdec.ch/schema/sd/20200220/SalaryDeclarationTxAB">
    <Company CL="Abteilung Steuerungen" City="Thun" HR-RC-Name="COMPLEX Elektronik AG" Person="Paula Nestler" Phone="033 238 49 71" Postbox="124" Street="Eigerweg 6" UID-BFS="CHE-123.123.123" ZIP="3600"/>
    <PersonID CL="" City="Bern 9" Country="" Firstname="Anna" Lastname="Aebi" Locality="" Postbox="690" Street="Langgassstrasse 26" ZIP="3000">
        <SV-AS-Nr>123.4567.8901.28</SV-AS-Nr>
    </PersonID>
    <A>
        <DocID>1</DocID>
        <Period>
            <from>2016-10-01</from>
            <until>2016-11-30</until>
        </Period>
        <Income>48118.70</Income>
        <GrossIncome>68000.00</GrossIncome>
        <NetIncome>56343.00</NetIncome>
    </A>
</T>

Decoded barcode content for a swiss salary statement containing the fields and values in a structured form.

Demo Preview

Figure 3 shows a preview of the demo for the Zurich tax statement. On the right we can see multiple detected barcode chunks and the final decoded xml content.

Fig. 3: Demo UI - Preview of Zurich Tax statement (left) and the extracted raw barcode chunks (right). The final decoded data is the xml (upper right).

Scalability Discussion

The overall solution is event based and uses AWS Lambda, Amazon DynamoDB, and Amazon Textract, as shown in Figure 2. We revisit the scaling limitations and concurrency quotas of these services and how they can be overcome for particular use cases where higher throughput would be required. Please note that by default even if these quotas are reached, requests will be saved and processed after some time, when there is capacity.

Amazon DynamoDB streams is being used to trigger a Lambda function when a new file gets upload for processing. Lambda polls shards in your DynamoDB stream for records at a base rate of 4 times per second. It received batches of records and forwards them to the Lambda function. If the default settings are not enough, one can increase the throughput by increasing the batch size (up to 10K) and increase the ParallelizationFactor as described in the AWS Lambda with AmazonDynamoDB documentation

AWS Lambda is designed to provide scaling capabilities without any custom engineering in your code. As traffic increases, Lambda increases the number of concurrent executions of your functions. For an initial burst of traffic, the cumulative concurrency in a Region can reach between 500 and 3000 per minute, depending upon the Region. After this initial burst, functions can scale by an additional 500 instances per minute. There is also a default concurrent limit of 1000 per Region, for each AWS account, which can be increased by submitting a request in the AWS Support Center.

If there is more documents coming in at the same time than there is concurrency capacity, these documents will be lying on the queue (SQS) until they get processed.

There is also the possibility to use Provisioned Concurrency if a higher concurrency needs to be guaranteed for a particular use case.

Amazon Textract has service quotas, ranging from 2 to 10 requests per second, depending on the region. If these concurrency quotas are not enough for your particular use case, you can request an increase of these soft limits from the AWS console.

Supported Documents

While our focus was to show how to extract information from swiss tax and salary statements, the same approach can be utilized for other document types like:

Swiss Invoices (QR Code), introduced by Six-group
Swiss and European Covid Certificate
US drivers licenses with MRZ (machine readable PDF417 zone)

Ideally the public authorities related to these documents would publish the defined format for each document as part of an open government strategy, which would promote innovative business creation and innovation for these public services.

For Switzerland at least the following cantons utilize barcodes to encode information for tax declaration and processing: ZH-Zurich, VS-Valais, VD-Vaud, LU-Lucerne, AG-Aargau, TG-Thurgau. This solution can easily be extended to support all of these cantons.

Conclusion

We showed how to extract structured information from various types of swiss documents utilizing 2D barcodes and how to embed this barcode processing in a scalable event-based architecture utilizing AWS Lambda.

Our extension of DUS (Document Understanding Solution) for barcodes allows to deploy a full end-to-end solution using CDK and its CLI commands. The Demo allows to apply the barcode processing on all kind of documents and can be utilized as a starting point to increase automation in document processing workflows.