DEV Community: Janos Tolgyesi

Mind the layers: a three-layer model for document AI

Janos Tolgyesi — Wed, 30 Sep 2026 08:28:19 +0000

In case you missed the previous episode: I argued that there is no single universal schema for what a document means, because what matters about a document is relative to the question you are asking. Knowledge is structured by use, the categories we extract are tools rather than universal truths, and the design move that follows is to equip the inquiry rather than model the document. That was the stance. This post is the architecture.

This is the second of a three-part series. Part one argued the case; part three will get into implementation. Here, in the middle, I want to make it concrete with a model I have come to rely on: the knowledge you pull out of a document lives on three layers, and the layers differ in how much of the work you can reuse from one document or task to the next. Then I will argue the single rule that keeps such a system debuggable when it breaks in production, which it will: never skip a layer. The model is the tidy part; the rule is the part that earns its keep at three in the morning. Let's dive deeper.

The three layers

Here is the model in one sentence: the knowledge you extract from a document sits on three layers, and each layer has both a different reuse profile (how much of it carries over to the next document or task) and a different epistemic status (the kind of question it answers). I will take each layer in three beats: what it is, how reusable it is, and what kind of question it settles.

Layer 1 is the document's intrinsic structure: pages, blocks, tables, reading order, sections, signatures, the geometry of the thing on the page. This layer is fully reusable, because every document is, first and before anything else, a document: a contract, an invoice, a vaccination certificate, and a novel are all pages with blocks and a reading order. Its epistemic status is perception: it answers "what is physically on the page?" It is domain-free and, increasingly, engine-detectable by general-purpose document layout analysis. Build it once, reuse it everywhere.

Layer 2 is the domain's entities and relations: the parties, dates, amounts, issuing authorities, and cross-references that a domain keeps caring about across documents. This layer is partially reusable. A sparse upper ontology generalizes (most legal documents have parties and effective dates), and each domain extends it with its own. Its epistemic status is grounding: it answers "what is this document about, in domain terms?" It is the layer that turns a block of text at coordinates on page four into "the governing-law clause naming the State of New York."

Layer 3 is workflow-specific knowledge: "is this a duplicate payment?", "is this clause enforceable?", "summarize this filing for the board." This layer is not reusable, and it should not be. Its epistemic status is inference: it answers "what does this particular workflow need to conclude?" This is part one's "equip the inquiry" layer, wearing its architectural hat. Each workflow builds the slice of knowledge its question needs, and retires it when the question changes.

None of this is specific to legal or financial documents. Run the same model over a novel and the layers still fall out cleanly: chapters, paragraphs, and dialogue at Layer 1; characters, places, and events at Layer 2; a question like "how does John change across the book?" at Layer 3. I will come back to that.

The following diagram shows the model, and the shortcut it warns against.

Diagram by the author

Hold on to the three epistemic statuses, because they turn out to be the point: perception, grounding, and inference are three different kinds of question, and a system that keeps them apart is a system that can tell you which one it got wrong.

The shapes of Layer 2

Layers 1 and 3 are easy to picture. Layer 2 is the interesting one, because it looks quite different from one domain to the next, and it is where most of the design judgment goes. Three contrasting examples make the point.

The first is the humble invoice, the same example that closed part one. An invoice has a rich, stable Layer-2 vocabulary: issuer, recipient, line items, amounts, tax, dates, a reference number. This is the reusable core I promised to give a home to, and here it is: capture that vocabulary once and reuse it across every workflow that touches invoices. For the invoice, Layer 2 is thick, and it is mostly a matter of naming the entities that are reliably there.

The second is a legal contract, and it behaves differently. Its stable vocabulary is comparatively thin: parties, governing law, a few effective dates. The real Layer-2 work is not naming entities; it is two harder jobs. The first is reference resolution. A contract cites statutes, articles of law, other clauses, and external agreements, and resolving each citation to a stable, canonical identifier (the task known in NLP as entity linking) is what lets an agent reason about the reference rather than merely follow the link: is the cited provision apt, is it still in force, has it been abrogated? Resolving a reference to a canonical identity is the precondition for reasoning about it, because the identifier is a key into a knowledge base where facts about the reference live, not just a pointer to its text. The second is defined-term binding. Contracts define their own terms ("Interest Rate" means what section 1.1 says it means, not what a dictionary says), so binding a defined term to the contract's own definitions section, a within-document coreference between the term and the clause that defines it, is a first-class grounding task. Bind "Interest Rate" to its everyday sense and you have made a grounding error, quietly, and everything Layer 3 concludes from it will be wrong in a way that looks perfectly reasonable.

The third is not a legal or financial document at all, and that is the point of including it. As a thought experiment, take a novel. Its Layer 2 is a cast of characters, the places they move through, the events that occur, and the timeline that orders them. The characteristic Layer-2 work here is neither a fixed vocabulary nor citation resolution but entity tracking across a long narrative: knowing that the "he" on page three hundred is the boy introduced on page five (coreference again), and reconstructing a chronology from a story told out of order. Put a Layer-3 question to it, "how does John's character develop across the book?", and the discipline is unchanged: get the structure and the entities in place before you reason about the arc. The model does not seem to mind the domain. It seems to work.

The point is not that one document is harder than another. It is that Layer 2 is real everywhere, but its thickness varies by domain: for the invoice it is mostly vocabulary, for the contract it is mostly resolution and binding, for the novel it is tracking a cast of entities and a timeline across a long text. The machinery that does the contract job well (detecting references, resolving them, binding defined terms) is a field note of its own, which I will write separately.

Keep Layer 2 sparse, Layer 3 disposable

There is a standing temptation to let Layer 2 grow. A workflow needs some derived fact, computing it once in the shared layer feels efficient, and so a workflow convenience quietly migrates downward into Layer 2. Resist it. Every workflow-specific convenience you bake into Layer 2 erodes the one property that made it worth sharing: its reusability. The discipline is a two-sided rule: keep Layer 2 sparse and stable, and keep Layer 3 rich and disposable.

A concrete case. A due-diligence review and a litigation-risk review run over the same contracts, and they overlap: both care about which obligations survive termination. The instinct is to compute "surviving obligations" once, push it down into Layer 2, and let both share it. Resist it: "surviving obligations" is not a stable domain fact but a task-shaped judgment, and the two reviews define it differently. What belongs in Layer 2 is the raw material they agree on (the termination clause, the obligations it names); the interpretation stays in each review's Layer 3, even though both interpret the same clause. That overlap is not duplication to factor away; hoisting the judgment into Layer 2 would bake one review's assumptions into the shared layer and couple the two. The test: a fact that is task-independent and stable belongs in Layer 2; a judgment whose definition depends on the question stays in Layer 3.

Never skip a layer

Now the rule the whole post is named for. Never go straight from the raw document to the language model. The shortcut is seductive: drop the whole PDF, or a text dump of it, into the prompt, ask your Layer-3 question, and let a capable model sort out the rest. In a demo it works beautifully. In production it fails, and worse, it fails in a way you cannot debug.

The reason is that the three layers are three distinct places where a mistake can happen, and the shortcut collapses them into one. Consider the chain, kept deliberately generic: a table cell is misread (a perception error), so an amount is attached to the wrong party (a grounding error), so the workflow reaches a wrong contractual conclusion (an inference error). That errors cascade and compound down a chain of dependent stages is not a new observation: it is a long-studied failure mode of language-processing pipelines (Finkel, Manning, and Ng, 2006), whose own remedy was to reduce the propagation with joint inference. The layered response makes a different and more modest bet. When you build the layers explicitly, each of those is a separate, inspectable step. When you throw everything into one prompt, all three failure modes happen inside the same opaque call, and when the answer comes back wrong you have no way to ask the only question that matters: which layer failed? Was the number misread, or read correctly and misattributed, or read and attributed correctly but reasoned about wrongly? One prompt cannot tell you. Three layers can.

That is the payoff, and it is worth stating plainly, because it is the entire reason to pay the tax of building the layers at all: explicit layers make a failure localizable to one layer. A layered system fails in a place; a skipped-layer system just fails. And what fails in a place can be tested in a place: each layer takes its own golden dataset, which is far less noisy than one end-to-end metric whose definition of success is hard to pin down (a topic I will take up in its own post). Diagnosability is not a nice-to-have you add later; it is a property you either designed in or gave away on day one. If this sounds like a familiar engineering discipline, it is: separation of concerns applied to document knowledge, the same reason strict layering keeps systems like the OSI stack tractable, where each layer builds on the one below without reaching around it.

Going back to the source is not the same as skipping a layer. A Layer-3 task will often need a detail the generalist Layer 2 never captured, and the right move is to reach back and re-read the exact span the lower layers point to: the full text of clause 4.2, or the chapter where the two characters first meet. That is using the layers to navigate back to grounded evidence, not bypassing them. What the rule forbids is the ungrounded shortcut, asking the final question of the raw document with no layers built at all.

There is a precondition hiding in here, which I will only gesture at now. Layering pays off only if the lower layers hand the upper layers something stable to point at. If Layer 1's identifiers churn every time you re-extract the document (re-run the extraction pipeline after, say, an OCR or model upgrade), then Layer 2's groundings and Layer 3's conclusions dangle, and the whole structure collapses back into re-extract-everything. Stable identity across re-extraction is what makes layering hold together, and it is the subject of part three.

A familiar split

If some of this feels familiar, it may be because you have met it under other names. In a talk I gave on document AI at AWS, I split the pipeline into extraction, knowledge, and generation. Those are the same three layers wearing different labels: extraction is Layer 1, knowledge is Layer 2, generation is Layer 3. I find the layer framing more useful now precisely because it makes the never-skip rule obvious, and the rule is what tells you the split is worth keeping even in an era when a single large model could, in principle, do all three jobs in one call. Being able to skip the layers is exactly why you need a rule not to.

What to carry forward

In summary:

Three layers, three reuse profiles. Intrinsic structure is fully reusable, domain entities are partially reusable, and workflow knowledge is not reusable at all (and that last one is a feature, not a gap).
Each layer answers a different kind of question: perception, grounding, inference. Keeping them separate is what lets a system tell you which one it got wrong.
Never skip a layer. The raw-document-to-model shortcut compounds errors into one undiagnosable failure; explicit layers buy you the ability to say where it broke.

The next time you are tempted to pipe a document straight into a prompt and ask the hard question, add the two intermediate stops: extract the structure, ground the entities, and only then infer. It costs more to build and it pays for itself the first time something goes wrong and you can point at the layer that did it.

The rule leans on a precondition I have only hinted at: the lower layers have to give the upper layers something stable to annotate. In part three I will build that out: a document object model that survives re-extraction, so identity does not churn underneath your groundings and conclusions. Stay tuned.

Janos Tolgyesi is an engineer building document AI systems, currently in legal AI, and an AWS Community Builder since 2020. Field notes on knowledge management, agents, and context engineering in document AI: mrtj.pro.

Pragmatism for document AI

Janos Tolgyesi — Wed, 23 Sep 2026 15:04:44 +0000

When a new document type lands on your desk (a contract, an invoice, a vaccination certificate), the first engineering instinct is almost always the same: design the schema. One data model per document type, rich enough to capture everything important, built once and reused across every workflow that will ever touch that document. It feels like the responsible thing to do. It is also, I have come to think, one of the most expensive mistakes in document AI.

This is the first of a three-part series on how I have learned to think about knowledge in document AI systems. Before the architecture (part two) and the implementation details (part three), I want to start with the stance, because the stance is what makes the rest coherent. That stance is borrowed, almost wholesale, from a hundred-year-old American philosophical tradition called pragmatism. Bear with the philosophy: it pays for itself quickly.

Let me state the claim up front, and state it carefully. It is not that structure never generalizes; for some documents a surprising amount of it does. It is that no single schema captures everything that matters about a document, because what matters is relative to the question you are asking. Knowledge is structured by use, and the categories we extract are tools rather than universal truths. Let's dive deeper.

The universal-schema instinct

The instinct has a respectable pedigree. It goes back to Aristotle's Categories, which set out to enumerate the fundamental kinds of things once and for all: substance, quantity, quality, relation, and so on. Every entity falls under a category, categories have structure, and reasoning becomes tractable. Computer science inherited the ambition directly. When Tom Gruber gave the field its canonical definition in 1993, an ontology became "an explicit specification of a conceptualization": a formal vocabulary of classes, relations, and constraints that a machine can reason over. The word was not borrowed by accident: computer science took over philosophy's ambition of naming, once and for all, the kinds of things a domain contains.

For part of the problem, this works beautifully. A document, considered as an artifact, really does have stable structure: pages, blocks, tables, reading order, sections, signatures. That layer generalizes across contracts, certificates, and transcripts, because every one of them is, first, a document. Build it once, reuse it everywhere.

The instinct rarely stops at the artifact, though. Having modeled the document's structure once, it reaches inward for the same universality: one schema for what the document means, rich enough that every future workflow can read what it needs off a single representation. That is the move I want to question. Some of what is inside genuinely generalizes, but the slice of meaning any given task actually needs does not settle into one shared form: how you carve up a document's meaning depends on what you are trying to do with it.

You might reasonably push back here, and the strongest counterexample is the humble invoice. An invoice is so regular that a single generic schema (issuer and recipient, line items, amounts, tax, dates, a reference number) really does apply to almost all of them. That is a genuine, reusable schema, and I do not want to wave it away: capturing that shared core once, rather than reinventing it per project, is exactly the right move. We will give it a home in the next post, as the middle layer of a three-layer model.

But notice what that reusable core is and is not. It is a stable vocabulary of the entities an invoice tends to contain; it is not a schema that captures everything that matters, because what matters still depends on the question. Ask whether an invoice is a duplicate payment, whether its tax treatment is correct, or whether it is fraudulent, and the decomposition that actually helps diverges each time: which fields you trust, what counts as a match, what "correct" even means. That task-shaped knowledge is the layer the universal instinct cannot pre-model, and the one the next post argues you should build only when a task demands it. A contract makes the point more starkly still: its useful "joints" are completely different when you are checking enforceability, computing tax exposure, or screening for fraud. There is no view-from-nowhere decomposition that captures everything important, because importance is relative to a question. The pragmatist way to put it: decomposition is downstream of use.

Wittgenstein gave us the sharpest tool for seeing why. Concepts like "contract" are not natural kinds with a shared essence; they are family resemblance concepts (Philosophical Investigations, sections 66 to 67), whose members overlap and criss-cross without a single defining feature. Their boundaries shift with jurisdiction and purpose. Trying to write the one definition that captures all contracts is chasing an essence that is not there. Mistaking a meaning problem for a structure problem is the root error, and it is an easy error to make precisely because our tools are so good at the structure part.

Concepts are tools, not mirrors

A category is just a concept put to work: the slice of meaning we decide is worth extracting. So if knowledge is structured by use, what is such a category actually for? Here the pragmatists are blunt. For John Dewey, a concept is not a picture of reality; it is an instrument for resolving a problematic situation. Peirce got there first with his pragmatic maxim of 1878: the meaning of a concept is the sum of the practical consequences of holding it. A category earns its place if it makes some downstream workflow succeed, and the same domain can legitimately warrant different categories for different tasks. A field that helps no inquiry is decoration, however correct it looks on paper.

That is the whole argument, and it points to a single design move: stop trying to model the document, and start trying to equip the inquiry. For any document type there is no one correct ontology, only the ontology that makes a specific downstream question answerable. Expect to hold more than one at a time. Expect to retire them when the question changes. This sounds permissive, but it is the opposite. "Structured by use" is not "anything goes", it is "justified by use": every category has to earn its keep against a real workflow.

We already do this, on the evaluation side

Here is the part I find quietly reassuring. Machine learning already practices pragmatism; it just practices it downstream. We do not ask whether a model's outputs correspond to reality. We ask whether they hold up across distributions, support good decisions, and stay stable under perturbation. That is Peirce's maxim almost verbatim, meaning cashed out in practical consequences, operationalized as a held-out test set. William James would recognize it at once: truth as what proves itself in practice.

So the field is pragmatist about how it judges models, and rationalist about how it designs schemas, still reaching for the one universal conceptualization that will capture everything. The fix is not more philosophy. It is to bring the stance we already trust for evaluation upstream, to the point where we decide what to extract in the first place.

Where to go from here?

In summary:

Knowledge is structured by use. A document has no privileged decomposition; its joints depend on the question you are asking.
Concepts are tools. A category is good if it helps a downstream workflow, so equip the inquiry rather than model the document.

Concretely, the next time you reach for the one universal schema of a document type, stop and ask which task it serves: build the schema for that question, expect to hold more than one, and let each category earn its place against a real workflow. None of this argues against structure. The document-as-artifact layer is real, shared, and worth getting right once. The argument is about where universality stops and use begins, and about being honest that most of the interesting knowledge lives on the "use" side.

In the next part of this series I will make that concrete with a three-layer model that puts each kind of knowledge where it belongs, and a rule that keeps the whole thing diagnosable: never skip a layer. Stay tuned.

Remote View your Computer Vision Models Running on AWS Panorama

Janos Tolgyesi — Mon, 09 May 2022 15:41:22 +0000

Real-time smart video analytics application development and edge device deployment is a tricks task. In the last few years, the industry players built platforms to support this activity. Notable examples include NVIDIA Metropolis, Intel’s OpenVINO, and AWS Panorama. While these solutions make some aspects of the video-analytics application development more straightforward, there are still many issues to deal with before deploying a video analytics application in production. This post introduces Telescope, the first in a series of open-source tools to make developing AWS Panorama application simpler.

AWS Panorama is a machine learning appliance and software framework that allows you to deploy video analytics applications on edge. For a thorough introduction and a step-by-step tutorial on deploying a Panorama application, refer to Deploy an Object-Detector Model at the Edge on AWS Panorama.

AWS Panorama framework eases many aspects of video analytics application development, including camera connection management, video decoding, frame extraction, model optimization and loading, display output management, over-the-air deployment of your application, and other features. Nevertheless, some tasks are still challenging, including diagnostic tasks when a model does not work as expected.

Introducing Telescope

The only available way to get visual feedback on the correct functionality of a Panorama application is to physically connect a display to the HDMI port of the appliance. The display will show the output video stream of a single application deployed on the device. However, physically accessing the appliance is not always feasible. Telescope allows to re-stream the output video of any Panorama application to an external service, for example, to AWS Kinesis Video Streams. This feature can be very convenient for monitoring an application remotely.

How does Telescope work?

Telescope instantiates a GStreamer pipeline with an appsrc element at the head. An OpenCV VideoWriter is configured to write to the appsrc: instead of saving the consecutive frames to a video file, it streams to the output sink. When opening the VideoWriter instance, the user should specify the frame width and height and the frame rate of the output stream. You can manually set these parameters or let Telescope infer them from the input dimensions and the frequency you feed new frames to it. If using this auto-configuration feature, some frames (by default 100) will be discarded at the beginning of the streaming because Telescope will use them to calculate statistics of the frame rate and measure the frame dimensions. This is the “warmup” of Telescope. If you send frames with different sizes, Telescope will resize the input, but this comes with a performance penalty. You are also expected to send new frames to Telescope with the frequency specified in the frame-per-second parameter. If frames are sent at a different frequency, the video fragments get out of sync in Kinesis Video Streams, and you won’t be able to replay the video smoothly.

Photo by Josh Redd on Unsplash

Telescope is an abstract base class that handles the state machine of the GStreamer pipeline. The concrete implementation KVSTelescope sends frames to Amazon Kinesis Video Streams service. It is easy to extend Telescope to support other services, especially if there is already a GStreamer plugin for that.

Integrating Telescope into a Panorama application

Configuring the Docker container

Telescope depends on custom compiled external libraries. All these libraries must be compiled and configured correctly in your application’s docker container to make Telescope work. These libraries include:

GStreamer 1.0 installed with standard plugins pack, libav, tools, and development libraries;
OpenCV 4.2.0, compiled with GStreamer support and Python bindings;
numpy (it is typically installed by the base docker image of your Panorama application).

In the case you want to use KVSTelescope, the Telescope implementation that streams the video to Amazon Kinesis Video Streams, it will need also the following libraries:

Amazon Kinesis Video Streams (KVS) Producer SDK compiled with GStreamer plugin support;
Environment variable GST_PLUGIN_PATH configured to point to the directory where the compiled binaries of KVS Producer SDK GStreamer plugin is placed;
Environment variable LD_LIBRARY_PATH including the third-party open-source dependencies compiled by KVS Producer SDK;
boto3 (it is typically installed by the base docker image of your Panorama application).

In the source code repository of Telescope, a sample Dockerfile is provided in the examples folder that shows how to install these libraries and Telescope in any container correctly. In most cases, it is required just to copy the relevant sections from the sample to your application’s Dockerfile. Please note that the first time you compile the docker container, it might take up to one hour to correctly compile all libraries.

Setting up AWS IAM privileges for KVSTelescope

KVSTelescope uses the Amazon Kinesis Video Streams (KVS) Producer library, wrapped in a GStreamer sink element, to send the processed frames to the KVS service. KVS Producer needs AWS credentials. It can use the credentials associated with the Panorama Application Role, but you must explicitly configure it. KVSTelescope needs privileges to execute the following actions:

kinesisvideo:DescribeStream
kinesisvideo:GetStreamingEndpoint
kinesisvideo:PutMedia

If KVSTelescope needs to automatically create the Kinesis Video Stream the first time it is used, you should also include the kinesisvideo:CreateStream action. An example policy allowing KVSTelescope to write data to Kinesis Video Streams could look like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kinesisvideo:DescribeStream",
                "kinesisvideo:CreateStream",
                "kinesisvideo:GetDataEndpoint",
                "kinesisvideo:PutMedia"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Setting up credentials

There are two types of AWS security credentials: static and temporary. The former never expire and in case of a leak, you should invalidate them manually and reconfigure the applications. For this reason, their use is strongly discouraged in a production environment. Examples of static AWS credentials include the IAM user’s Access Key Id and Secret Access Key pair.

Photo by Angela Merenkova on Unsplash

Temporary credentials expire after a predefined period. In case of a leak, they can be used only within their expiration time, typically in the order of several hours. Temporary credentials can be renewed before they expire so their lifespan can be extended, or they can be exchanged for new ones with a later expiration time. This process needs additional coordination from the application using this type of credentials. Examples of temporary credentials include the AWS Access Key Id, the Secret Access Key, and the Session Token.

We offer different options to provide credentials with KVSCredentialsHandler subclasses provided in the kvs module. If you want to use static credentials for testing purposes, create an IAM user in your AWS account and attach a policy like the one above to the user. You should configure this user to have programmatic access to AWS resources and get the user’s AWS Access Key and Secret Key pair: create a KVSInlineCredentialsHandler or KVSEnvironmentCredentialsHandler instance in the application’s code to pass these credentials to KVS Producer Plugin directly in the GStreamer pipeline definition or as environment variables. However, as these credentials do not expire, using this configuration in a production environment is not recommended. Even in a development and testing environment, you should take the appropriate security measures to protect these credentials: never hardcode them in the source code. Instead, use AWS Secret Manager or a similar service to provide these parameters to your application.

KVSTelescope can also use the Panorama Application Role to pass the application's credentials to KVS Producer. These credentials are temporary, meaning that they expire within a couple of hours, and the user should renew them before expiration. The Producer library expects temporary credentials in a text file. KVSFileCredentialsHandler manages the renewal of the credentials and periodically updates the text file with the new credentials. If you want to use this method, attach a policy similar to the example above to your Application Role. Remember to always test your Panorama application — KVS integration if it still works after the KVSFileCredentialsHandler refreshed the credentials. Let your application run for several hours and periodically check if it continues to stream the video to KVS. You will also find diagnostic information in the CloudWatch logs of your application when the credentials were renewed.

Environment variables

KvsTelescope needs two environment variables to have GStreamer find the KVS Producer plugin. The name of these variables is GST_PLUGIN_PATH and LD_LIBRARY_PATH. They point to the folder of KVS Producer GStreamer plugin and its 3rd party dependencies. In the sample Dockerfile provided, the correct values of these variables are written to a small configuration file named /panorama/.env in the container. You should pass the path of this file to KvsTelescope or otherwise ensure that these environment variables contain the correct value.

Usage example

When initializing a KVSTelescope instance, you should pass the AWS region name where your stream is created, the name of the stream, and a credentials handler instance. If you want to configure the frame rate and the dimensions of the frames manually, you should also set them here. When both parameters are specified, Telescope skips the warmup period and sends the first frame directly to KVS. When you are ready to send frames, call the start_streaming method which opens the GStreamer pipeline. After this method is called, you are expected to send new frames to the stream calling the put method periodically, with the frequency specified in the frame rate or inferred by KvsTelescope. You can stop and restart streaming on the same KvsTelescope instance.

The following example uses the temporary credentials of the IAM Application Role assumed by your Panorama application:

import panoramasdk
from backpack.kvs import KVSTelescope, KVSFileCredentialsHandler

# You might want to read these values from 
# Panorama application parameters
stream_region = 'us-east-1'
stream_name = 'panorama-video'

# The example Dockerfile writes static 
# configuration variables to this file
# If you change the .env file path in the 
# Dockerfile, you should change it also here
DOTENV_PATH = '/panorama/.env'

class Application(panoramasdk.node):

    def __init__(self):
        super().__init__()
        # ...
        credentials_handler = KVSFileCredentialsHandler()
        self.telescope = KVSTelescope(
            stream_region=stream_region,
            stream_name=stream_name,
            credentials_handler=credentials_handler,
            dotenv_path=DOTENV_PATH
        )
        # This call opens the streaming pipeline:
        self.telescope.start_streaming()

    # called from video processing loop:
    def process_streams(self):
        streams = self.inputs.video_in.get()

        for idx, stream in enumerate(streams):

            # Process the stream, for example with:
            # self.process_media(stream)

            # TODO: eventually multiplex streams 
            # to a single frame
            if idx == 0:
                self.telescope.put(stream.image)

If everything works well, you can watch the restreamed video on the Kinesis Video Streams page of the AWS console. Certainly, you can modify the image before sending it to Telescope: draw annotations on it based on the inference result of the deep-learning model.

Annotations

Annotations and annotation drivers provide a unified way to draw annotations on different rendering backends. Currently, two annotation drivers are implemented:

PanoramaMediaAnnotationDriver allows you to draw on panoramasdk.media object, and
OpenCVImageAnnotationDriver allows you to draw on an OpenCV image (numpy array) object.

Photo by Lucas George Wendt on Unsplash

The library can draw two types of annotations: labels and boxes.

Using annotations

Depending on the available backends, you can create one or more annotation driver instances at the beginning of the video frame processing loop. During the process of a single frame, you are expected to collect all annotations to be drawn on the frame in a python collection (for example, in a list). When the processing is finished, you call the render method on any number of drivers, passing the same collection of annotations. All coordinates used in annotation are normalized to the [0; 1) range.

Example usage:

import panoramasdk
from backpack.annotation import (
    Point, LabelAnnotation, RectAnnotation, TimestampAnnotation,
    OpenCVImageAnnotationDriver, 
    PanoramaMediaAnnotationDriver
)

class Application(panoramasdk.node):

    def __init__(self):
        super().__init__()
        # self.telescope = ... 
        self.panorama_driver = PanoramaMediaAnnotationDriver()
        self.cv2_driver = OpenCVImageAnnotationDriver()

    # called from video processing loop:
    def process_streams(self):
        streams = self.inputs.video_in.get()
        for idx, stream in enumerate(streams):
            annotations = [
                TimestampAnnotation(),
                RectAnnotation(
                    point1=Point(0.1, 0.1), 
                    point2=Point(0.9, 0.9)
                ),
                LabelAnnotation(
                    point=Point(0.5, 0.5), 
                    text='Hello World!'
                )
            ]
            self.panorama_driver.render(annotations, stream)

            # TODO: eventually multiplex streams to a 
            # single frame
            if idx == 0:
                self.cv2_driver.render(
                    annotations, 
                    stream.image
                )
                # self.telescope.put(stream.image)

Postface

Even if Telescope can be a helpful tool, its usage might raise two concerns that you should consider carefully. We discourage using Telescope in a production environment: it is a development aid or and a debugging tool.

Photo by Greg Rosenke on Unsplash

The first concern is of technical nature. Currently, the application code in a Panorama app does not have direct access to the onboard GPU; thus, all video encoding done by Telescope run on the device’s CPU. This behavior could slow down the appliance: streaming a single output stream with Telescope could require anything between 10–30% of the CPU capacity of the device.

The second concern is related to data protection. The Panorama appliance is designed to protect the video streams being processed. It has two ethernet interfaces to separate the network of the video cameras (typically a closed-circuit local area network) from the device’s Internet access. Using Telescope, you relay the video stream from the protected, closed-circuit camera network to the public Internet. You should carefully examine the data protection requirements of your application and the camera network before using Telescope. Moreover, you are responsible for keeping private all AWS credentials used by Telescope, as required by AWS Shared Responsibility Model.

Where to go from here?

Telescope is part of Backpack, a broader set of tools that aims to help software development on AWS Panorama. Other components will be presented in a future post, so keep tuned and follow us to get updated.

Backpack is open source and available on GitHub. You can build and run the example docker container on an ARM64 based system. For example, suppose you’ve already set up the Panorama Test Utility on a t4g type EC2 instance. In this case, you can build and launch a shell in this container with the following commands on the EC2 instance:

$ git clone https://github.com/Neosperience/backpack.git
$ cd backpack/examples
$ docker build . -t my_backpack_example:1.0
$ docker run -it my_backpack_example:1.0 bash

The container can be built and executed also on a MacBook Pro with the new M1 ARM-based chip.

The backpack library is extensively documented with docstrings. You can read the detailed API docs online or using the python3 interpreter in the container shell:

root@my_backpack_example:/# python3
>>> import backpack.telescope, backpack.kvs, backpack.annotation
>>> help(backpack.telescope)
>>> help(backpack.kvs)
>>> help(backpack.annotation)

You can also try other Telescope implementations found in Backpack. For example, RTSPTelescope allows you to host an RTSP server right in the application container, and connect directly to your Panorama appliance with an RTSP client to monitor your application.

Let me know how you use Telescope, which new features you would like to see and stay tuned for future posts about the other components of Backpack.

About the author

Janos Tolgyesi is an AWS Community Builder working as Machine Learning Solution Architect at Neosperience. He has worked with ML technologies for five years and with AWS infrastructure for eight years. He loves building things, let it be a video analytics application on the edge or a user profiler based on clickstream events. You can find me here, on dev.to, on Twitter, Medium, and LinkedIn.

The open-source project of Backpack was supported by Neosperience.

I would like to express my special thanks to
Luca Bianchi for proofreading this article.