DEV Community: Tom Harvey

Integration tests with CDK and Python - a more real world app, less hello world

Tom Harvey — Mon, 26 Feb 2024 21:25:11 +0000

In the previous post I took the fastest path to building a CDK app with Python with a passing Integration Test. It showed how to create AWS services using CDK and then use integ-runner to automate the deployment of the services, use the services and assert that the result is as expected.

But, the CDK stack, the test runner, and the assertions were all written in one file.

In a traditional CDK we would have a more sensible application layout. We want to define our Constructs and Stacks in the main application, then import those into a test file. This allows us to have re-usable Constructs and Stack, which is a core benefit of CDK.

We want an application layout more like:

my-cdk-app/
├─ my_cdk_app/
│  ├─ my_stack.py
│  ├─ lambda/
│  │ ├─ index.py
├─ test/
│  ├─ integ_my_stack.py
├─ app.py
├─ cdk.json

Now we have a green, passing test, we will refactor the code into more of a real world CDK app than a "hello world" app. We will also cover some of the rough edges in using integ-runner - a developer preview tool, which very much feels developer preview.

You can see the full code for this post here

Test ... or tests

The cdk init command will bootstrap your CDK app, including a "unit test" inside the /tests directory.

But, out of the box, integ-runner expects your tests to live in the /test (singular) folder.

Let's first fix this small inconsistency, and (more importantly) introduce ourselves to how we can better configure our integration tests.

Create a new file called integ.config.json. This file contains the runtime options you want for your testing. Here's mine:

{
    "maxWorkers": 10,
    "parallelRegions": [
        "eu-west-1",
        "us-east-1",
    ],
    "language": "python",
    "inspect-failures": true,
    "update-on-failed": true,
    "directory": "tests"
}

You can see what each of these options does in the integ-runner docs. These are the command line options, made into JSON.

The important thing we will do here is change the value of the directory config to "tests" and then move all of the contents of the test directory in there. We will create a new folder in tests called integration to keep these tidy and separate from the unit tests.

mkdir tests/integration
mv test/* tests/integration/
rm -r test/

And, now, we can re-run integ-runner to check that we're still green.

While it took 3 or 4 minutes to run initially, when we re-run it here it'll pass in around 10 seconds.

We've moved the files around, but the deployed stack would have been the same as before, so it doesn't need to deploy the services in AWS.

You should see a successful test run output like:

Verifying integration test snapshots...

  UNCHANGED  integ_hello 13.031s

Snapshot Results: 

Tests:    1 passed, 1 total

Running integration tests for failed tests...

Running in parallel across regions: eu-west-1

Test Results: 

Tests:    0 passed, 0 total

Reusable constructs

Our test file also contains the Stack that we would want to use in our App.

We can't import the Stack from that integ_hello.py file into our main App. So, we can only deploy it through the test.

Not much use.

Let's create a new file in our main application directory touch integration_tests_with_cdk_and_python/hello_world_stack.py and add the stack definition to that:

# integration_tests_with_cdk_and_python/hello_world_stack.py

import os

from aws_cdk import (
    Stack,
    aws_lambda,
)
from constructs import Construct


class HelloWorldStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        self.function = aws_lambda.Function(
            self,
            "MyFunction",
            runtime=aws_lambda.Runtime.PYTHON_3_11,
            handler="index.handler",
            code=aws_lambda.Code.from_asset(f"{os.getcwd()}/lambda"),
        )

Note that I’m now defining the path to the lambda relative to the ‘cwd’ instead of relative to the test file; which we were doing previously.

The cwd is the Current Working Directory; which is where you run your scripts from, almost always the root of your app.

So, move the lambda code out of the test directory into the root of the app:

mv tests/integration/lambda/ .

Now, we should be able to import that into our integ_hello.py file:

# tests/integration/integ_hello.py

from integration_tests_with_cdk_and_python.hello_world_stack import HelloWorldStack

...

When we run integ-runner now, we get an error:

ModuleNotFoundError: No module named 'integration_tests_with_cdk_and_python'
  ERROR      integ_hello.py (undefined/eu-west-1) 5.054s
      Error during integration test: Error: Command exited with status 1

It can't find our Stack now that we've defined it in our main app.

I've raised this bug as an issue here, so if it's fixed there, then this section might not be a problem anymore.

When integ-runner runs, it can only see files in the same directory (or below) the directory where our test file lives. So, only files inside tests/integration/ for us. That's no use.

To make this easier to work around, I created a package called cdk-integ-runner-cwd-fix which we can use.

Add cdk-integ-runner-cwd-fix to your requirements-dex.txt file and re-run pip install -r requirements-dev.txt

So, now, our tests/integration/integ_hello.py file can be updated to:

Import the Stack from the main directory
Use cdk-integ-runner-cwd-fix to fix the bug so that import works
Remove the old Stack definition we had in the test file

# tests/integration/integ_hello.py

import aws_cdk as cdk
from aws_cdk import (
    integ_tests_alpha,
)
from cdk_integ_runner_cwd_fix import fix_cwd

# This needs to be before any import from your CDK app to work around a bug in the CDK
fix_cwd()

from integration_tests_with_cdk_and_python.hello_world_stack import HelloWorldStack

app = cdk.App()
stack = HelloWorldStack(app, "HelloTestStack")

integration_test = integ_tests_alpha.IntegTest(app, "Integ", test_cases=[stack])

function_invoke = integration_test.assertions.invoke_function(
    function_name=stack.function.function_name,
)

function_invoke.expect(
    integ_tests_alpha.ExpectedResult.object_like(
        {"StatusCode": 200, "Payload": '"Hello world"'}
    )
)

app.synth()

Now, you need to set CDK_INTEG_RUNNER_CWD to your project's root directory before running integ-runner:

CDK_INTEG_RUNNER_CWD=$(pwd) integ-runner

And, we should be back to a green test.

Conclusion

Now, we have a Stack defined in the main application, as we would usually write a CDK app. We can import that into a test file and run tests against it. We can import that into an app.py file and deploy it for real.

So, from here, you should be able to return to your usual CDK workflow, but use integration tests to check that your lambdas behave, and your S3 buckets can accept files, or IAM roles can perform certain tasks ... whatever AWS service you want to automate the deployment, execution and assertion of.

Integration Tests with CDK and Python - is my cloud native app doing what I want?

Tom Harvey — Tue, 20 Feb 2024 17:12:41 +0000

CDK is a great way to setup your infrastructure, but all too often you deploy your new services and then need to go "click", "click", "click". But, we're all developers who like automated tests, and hate click-ops.

Here, I will show you how to build an AWS Lambda service and use the command line to run an automated test which will:

Deploy an AWS service (in 3 regions concurrently!)
Invoke the lambda
Assert that the response matches what you expect
Destroy the service and clean up.

That all sounds like a slow process, but the testing tool (integ-runner) creates snapshots of your stacks and only tests those which have changed.

We show how to test a lambda deployment, but you can use this approach to:

Check that you can (or cannot - because of permissions) get and put objects to an S3 bucket
Call an API gateway or Load Balancer to test your web service running in Fargate
Put a message into an SQS queue and wait for a timeout to check some async process completed.

You can see the complete code for this on GitHub - https://github.com/tomharvey/integration-tests-with-CDK-and-python/tree/part-one

Test first

We will write the test first and take the quickest path to making that test pass.

If you're new to CDK and python, see my blog post here for getting a running development environment. If you follow that post, you'll be ready to start.

1. Setup a new cdk project

This is all python, so cdk init --language python will bootstrap a new CDK project for us.

And, you'll want to install integ-runner with npm install -g @aws-cdk/integ-runner.

Warning - this “integ-runner” tool is in developer preview mode! So it will change and there are rough edges. Some very rough ones. But, please use the comments if you're following along and something stops working.

Finally, add aws_cdk.integ_tests_alpha to the requirements-dev.txt file and install all the requirements:

pip install -r requirements.txt
pip install -r requirements-dev.txt

2. Create your test file

CDK has bootstrapped a lot of directories and files for us. Including a tests directory. How kind.

That seems like a nice place to put your integration tests. But integ-runner expects them to be in a test (singular) directory. How irritating (the rough edges get rougher, don't worry). And, while we can tell integ-runner to use any directory, let's go ahead and stick with the defaults.

So, make a test folder with mkdir test in the terminal, or however you like to create new tests in your project.

And create a file in there called integ_.hello.py to create a new test file for the "hello world" Lambda we will create.

Use the terminal touch test/integ_.hello.py or however you like to add files to your project.

It's important that this file:

Starts with integ_.
Ends with .py

So be careful how you name these files. But anything can go between the integ_. and the .py

3. Write a CDK app and stack

We're going to write this directly in the test file.

You might be used to writing the app in app.py and creating your stacks elsewhere. Maybe even building re-usable constructs, but we will leave that tidy approach to another day. This is about getting quickly to a passing test.

So, below is a single file with the:

App
Stack
Test deployment of your services to AWS
Instruction to invoke the lambda
Assertion of the response

We will go though this file line-by-line at the end of the article. But, right now, it's a race to get a passing test.


import aws_cdk as cdk
from aws_cdk import (
    Stack,
    aws_lambda,
    integ_tests_alpha,
)
from constructs import Construct


class HelloTestStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        self.function = aws_lambda.Function(
            self,
            "MyFunction",
            runtime=aws_lambda.Runtime.PYTHON_3_11,
            handler="index.handler",
            code=aws_lambda.Code.from_asset("./lambda"),
        )


app = cdk.App()
stack = HelloTestStack(app, "HelloTestStack")

integration_test = integ_tests_alpha.IntegTest(app, "Integ", test_cases=[stack])

function_invoke = integration_test.assertions.invoke_function(
    function_name=stack.function.function_name,
)

function_invoke.expect(
    integ_tests_alpha.ExpectedResult.object_like(
        {"StatusCode": 200, "Payload": '"Hello world"'}
    )
)

app.synth()

And you will also need your lambda function code in a folder.

You can create this folder and file in the terminal with:

mkdir test/lambda
touch test/lambda/index.py

Or however you like to add folders and files to your project. But, yes, we're going to create our lambda code package inside the test folder for now. There is a "rough edge" otherwise, and we're aiming for tests passing.

And in that file in test/lambda/index.py you will want:

# test/lambda/index.py
def handler(event, context):
    return "Hello world"

Which is a lambda function, which, when invoked will return "Hello world".

4. Run your test for the first time

Now, you can run integ-runner to run the test, and we will get our first failure:

Verifying integration test snapshots...

  NEW        integ_.hello 5.812s

This is an expected error.

Because it takes time to deploy services and run them, integ-runner will create "snapshots" of your stacks. It saves what your stack looks like, so when you next run the test it knows if anything has changed.

But the first time you run it - there is no snapshot.

5. Really run your test

You want to use an additional option to tell integ-runner to go ahead and test any changed, or new stacks:

integ-runner --update-on-failed

This will take a few minutes...

You'll see that "NEW" error that you saw above, but then:

Running integration tests for failed tests...
Running in parallel across regions: us-east-1, us-east-2, us-west-2

Which is cool - it's deploying your lambda to 3 regions and running the tests in all 3 regions at the same time.

You can change the regions that you want to use. But let’s leave that for another time.

You can also change the props passed into your stack. So you could write tests to deploy your lambda with python version 3.12 as well as 3.11 and test things before you deploy updated 3.12 to production. Or ARM alongside x86. But, not today. We're going for the quickest route to a passing test.

Once we have a green test we can refactor and do more complex things.

After about 3 or 4 min you should get a passing result! Looking like the below

Verifying integration test snapshots...

  NEW        integ_.hello 5.715s

Snapshot Results: 

Tests:    1 failed, 1 total
Failed: /workspaces/py-integ-runner/test/integ_.hello.py

Running integration tests for failed tests...

Running in parallel across regions: us-east-1, us-east-2, us-west-2
Running test /workspaces/python-aws-cdk-integration-test-playground/py-integ-runner/test/integ_.hello.py in us-east-1
  SUCCESS    integ_.hello-Integ/DefaultTest 189.601s
       AssertionResultsLambdaInvoke24cf31b9b6a07a940ece1b49bb7eb7b2 - success

Test Results: 

Tests:    1 passed, 1 total

Yay!

What is the test file doing?

First up is the Stack that you would create. Here, it just defines a single lambda, but in a real world example it could define a lot more.

Note how the Code.from_asset sets the path to the lambda code files (‘./lambda’) and how the handler specifies that it should execute the function called “handler” inside the file called “index.py”.

class HelloTestStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        self.function = aws_lambda.Function(
            self,
            "MyFunction",
            runtime=aws_lambda.Runtime.PYTHON_3_11,
            handler="index.handler",
            code=aws_lambda.Code.from_asset("./lambda"),
        )

Next, we instantiate the App and the Stack. This is usually in app.py on the root of your CDK project. But we need a separate App for this testing process.

app = cdk.App()
stack = HelloTestStack(app, "HelloTestStack")

Then, we define a Test, and one single test case (this is where you could otherwise pass multiple test cases, where your lambda would have different python versions, architectures, or entirely different domain code).

integration_test = integ_tests_alpha.IntegTest(app, "Integ", test_cases=[stack])

We tell the Test what actions we want to take on each test_case stack. Here, we invoke the lambda.

function_invoke = integration_test.assertions.invoke_function(
    function_name=stack.function.function_name,
)

And lastly (almost), we assert that the response is as expected.

function_invoke.expect(
    integ_tests_alpha.ExpectedResult.object_like(
        {"StatusCode": 200, "Payload": '"Hello world"'}
    )
)

Finally - for real - and this is often missing from the IntegTest documentation, you need to app.synth()

Don't forget that final line there. And this synth function call needs to come after all of your assertions.

Now what?

We now have a green and passing test. It's time to refactor.

But, there are some very rough edges to refactoring. See https://pypi.org/project/cdk-integ-runner-cwd-fix/ for some hints on how to get around those edges.

You also have a pattern for how to test and run deployed services end to end. So, it's time to apply that to your SQS/RDS/S3 or whatever other services.

Finally, you have an integration test, but that doesn't mean you don't still benefit from unit tests. A unit test to check that the handler returns "Hello world" - without deploying infrastructure - would run in milliseconds. So, this 4 minute integration test is only one piece of your development framework.

How will you fit it into your testing approach?

Cloud native apps with AWS CDK, Python and VSCode Dev Containers

Tom Harvey — Sat, 17 Feb 2024 15:02:23 +0000

When starting a new project I reach for Python and AWS.

To help me build cloud-native Python apps, I've been using AWS's CDK. This "Cloud Development Kit" allows me to create storage or compute using Python and have my infrastructure live alongside my domain logic.

And I write all of this in VSCode, using their "Developer Containers" features to keep things standardised and 'simple'.

When I use a Dev Container, I don't need to worry about having python installed on my laptop. I don't need to fuss around with virtualenvs. My laptop can be left with the base OS and no funny business with Brew or Pyenv or Conda or whatever else I need to experiment with.

I just need my laptop to have:

Docker Desktop (or an equivalent)
VSCode

And that's it.

I can now write apps in whatever Python version I want. And you'll see how easy it would be to extend to TypeScript, Go, Rust or whatever.

So, install Docker Desktop (or an equivalent) and VSCode and then these few simple steps will get you building a cloud native app in no time.

What are "Dev Containers"?

These are machines running inside of your "host" machine. The "host" machine is your laptop (or desktop).

Self contained

Dev containers are self-contained environments. They contain Python and all of the other dependancies that you need to get the work done.

One doesn't conflict with another. So, if one project needs Python 3.7 and another needs 3.12 - no big deal. For all that you're using your one laptop these Dev Containers might as well be separate machines.

Your dev container will exist on your laptop, and your desktop in an identical way. Or on your colleague's laptop in an identical way. Or on a big powerful AWS instance in an identical way.

So, team mates can have identical environments. And, as a solo-coder you can rough out the easy stuff on your laptop and then easily run that dev container on an AWS instance with 96 CPUs 768GB of RAM, 8 GPUS and a 100Gbps network connection (if you have the $18.30/hour to spend on such hardware).

The point is, your developer environment is the same regardless of the underlying hardware. It's repeatable and reliable. It doensdoesn't rely on you remembering to brew install foo and if there is a version conflict you need to perform some magical steps.

Finally, these dev containers can be scrapped easily when you're done with some experiment; without leaving a trail of old versions of Python on your 'host machine'.

The template project

I have a template project at https://github.com/tomharvey/python-aws-cdk-devcontainer

This is really just one file!

Inside a directory called .devcontainer i have a devcontainer.json file.

It very very heavily commented, but as an overview, this file tells VSCode that you want a certain flavour of developer environment.

Python 3.12 on Debian Bullseye - yes, even if you're on a mac or on windows.
With some additional tools installed:
1. AWS CLI so we can interact with AWS
2. AWS CDK so we can build infrastructure
3. AWS SSO Util so we can login to AWS using SSO (if we need)
It's setup so we share the AWS Config folder on our machine with this new developer environment. That way our AWS Auth can be shared between the host machine and the dev environment.

Usage

1. Get my template

Go to https://github.com/tomharvey/python-aws-cdk-devcontainer and select the "Use this template" option to "create a new repository".

2. Create a new Project from the tempalte.

You can now setup your new project as a new GitHub repository.

Give it a name. "My Awesome new App" or "Python CDK Playground"
Keep it private or make it public. Your decision.

3. Wait for it ....

4. Copy that Github URL

You now have a new repo in your own github account. Time to open that in VSCode. So, copy the repo URL using SSH or HTTPS - however you like to interact with GitHub.

5. Open VSCode

Now, go into VSCode and ensure you have installed the Dev Containers extension. See here to install that extension

6. Use Dev Containers Extension

And bring up the "Command palette" from the "View" menu, or by ⌘-shift-P (ctrl-shift-p on Windows) and within that "Command palette" type "dev con clone" so it offers you the option you want:

Dev Containers: Clone Repository in Named Container Volume...

Click on that.

7. Clone the github repo

Then paste in the GitHub repo URL you copied in step 3 and hit the enter button.

And then select the main branch (which is likely the only option available to you)

Finally enter a new "volume name" - this is what Docker will name the volume which stores all your code - use the same name as the git hub repo.

8. Use your new code environment

After a few minutes (depending on your network connection) you will have a fully running Python environment with all of the AWS and CDK dependancies installed and ready to go.

Setup the name of your project. In the template I've named it "My awesome developement environment", but you will want to make it your own.

Open a terminal from the "terminal" menu - or with the ctrl shift ~ keyboard shortcut. And run python --version to see that you do indeed have python running as expected at version 3.12

Conclusions

You can now run the cdk command to setup a new cdk app and follow along with the AWS CDK Workshop or use the CDK API reference to build a cloud native app.

Your developer environment is entirely separate from your "host machine". It's transportable, and its definition lives alongside your code. Any new developer can pick up your app and use VSCode to immediately get an exact copy of your developer environment.

And, you didn't need to think one time about installing Python, setting up a virtualenv or whether you should use brew install or apt install or whatever when things went wrong.

Finally - while this will get you up and running, the devcontainer.json file is heavily commented so it should be self-explanitory. Please read through it and make it your own. Fork it. Raise a Pull Request to add or change things. And let me know how you prefer your developer environment.

Top Resources To Get A Role In Climate Change

Tom Harvey — Mon, 05 Feb 2024 12:50:49 +0000

So, you want to make the world a better place.

Your generation's defining challenge is right in front of you, and it's hard to waste your time and skills on anything else.

In a market of layoffs and downturns, climate companies are thriving and need your valuable skills.

So, to help you find a job that pays you to truly save the world, here are the best places to find your next climate impact role.

If you've got a great one to add - please post to the comments!

Job boards, communities and fellowships?

Most of these resources will provide at least one of these services.

The job board is one you likely know - there will be a list of jobs and you can apply for them. Simple!

The communities around these job boards are invaluable too. You'll find very active Slack, Whatsapp or Discord communities where people will share:

online job fairs where you can meet the companies who are hiring
hackathons
local and online events
reading recommendations to keep you informed
more jobs, or short term gigs
zoom workshops
open source projects
experiences in getting a job in climate
and so much more...

Get involved so you're first to know of new jobs and you're informed when you get that first interview.

Most require that you fill out a short form to talk about why you're joining. Once you're in, you will find me online - so fell free to find a "Thomas Harvey" and say "hi" 👋

Fellowships

Some of these communities provide "fellowships" or "accelerators" where they offer courses (for a price) to help you move into a climate career.

I get no affiliate payments from these - I want to mention them so you can decide if/when to use them for yourself

Both Terra.do and ClimateBase provide a really high caliber program with established industry contacts and both currently have open applications for fellowships - until Feb 23 2024 - so if you're interested, don't delay.

Where do I click?

Here are my favorites - plus one that I've never used. Dive in, and good luck! We need you over here building important solutions!

Climatebase

https://climatebase.org
A busy job board where you can browse by focus and location.

If you've a specific interest within climate - agriculture, electricity, carbon capture - then climatebase's job board will help you filter by your interest and location.

It's also got a very active, and growing community - here's a direct link to getting signed up.

Terra.do

https://www.terra.do
A community focused on learning.

I've heard great things about their accelerator course from alumni - some of whom I've hired.

Their focus on providing you with knowledge will ensure that you're confident when you go into that job interview.

Work on climate

https://workonclimate.org
A thriving community full of events

WoCl (yip - you gotta know the acronym) have been a driving force in making climate job hunting easy. Their strength is really in the events and the slack community that they organise.

You'll find all of the resources from WoCl to be really welcoming, and the fact that they and Terra.do work so closely together is testament to the collaborative and welcoming values they hold. Who cares who makes the world a better place - so long as it gets better!

Voyagers

https://voyagers.io/climate/
Super focused on community and growing together

If you're interested in networking with people in climate then this is the place for you. It's open for people at all stages of career, so you really get exposure to a broad spectrum of engineers, designers, investors, managers and executive.

This isn't a message board you can lurk around on, they will push you to grow your network. Their activities are often outside in nature, so get out and breathe that fresh air you're fighting to save.

Women and climate

https://www.womenandclimate.co
I'm not part of this one because I don't meet the criteria. But, I hear great things about it.

If you're looking to get into climate, it's super welcoming. If you're an experienced speaker, they are very pro-active in making sure there is more representation on speaking panels.

Leafr

https://www.leafr.work
Freelancers and short term gig opportunities.

The other communities usually have a "gigs" channel announcing short term contract work. This one is focused specifically on short-term engagements.

If you're looking to dip your toes in before you commit, or you want to spend some of your free time on a climate impact role, then this would be a great place to get listed and look out for that role.

Now what?

Getting involved is the first step.

Now, really get involved, take part in that thriving community. Find the local chapters and meetups that you can join ... or start.

And, if you're interested in what kind of skills help you stand out in this job market we can talk more about that another time.

You'll find me on the slacks I mentioned above, or ask questions in the comments.

Doubly meaningful variable names

Tom Harvey — Mon, 07 Nov 2022 17:58:00 +0000

Header image by Lucian Petrean CC

Giving meaningful names to our functions and variables helps communicate the context around our code.

Inspired by a LinkedIn post from Lowell Bander I recognised another layer of meaning in my code; one that communicates the motivation for why I code.

Here’s a snippet I was working on:

I’ve already posted about why now is the best time to work in the energy industry and it’s nice to have frequent reminders that the work is important. We’re installing solar and avoiding carbon emissions - tonnes of it!

Just as we all prefer different naming conventions, we all find motivation from different places. I hope that you can look at your code today, put aside debates about Hungarian notation, camelCasing or PEP 8 and recognise something which gives you a sense of meaning and purpose.

Put a spark in your career

Tom Harvey — Tue, 20 Sep 2022 10:02:15 +0000

The best time to work in energy is right now. It's the original high-tech industry but has been slow-moving. This stagnation, alongside recent transformations, has created the challenges and opportunities that make it a rewarding, engaging, and vital space to work.

Climate change is our generation's biggest problem, but you know that already. Solving that problem was the hook that drew me in, and it remains a daily driver, keeping me engaged. I won't be joining the chorus of voices convincing you to start your journey on a mission-led career path; you're already convinced. Right now, a confluence of factors is creating massive opportunity, and that's why now is the best time to join us, building in energy.

Decarbonization requires that the industry decentralizes generation and embraces digitization. This industry currently relies on 'test match cricket' as a communication protocol; dragging it towards current technologies is today's most exciting engineering challenge.

The existing grid operates under a century-old assumption that millions of consumers can and will rely on a few large generators. Energy flows in one direction from one place. That assumption is already broken and grows more wrong by the day. As more small local generation comes online, we each become both a source and a consumer of energy. Incumbents see this as a problem, while we see that as democratisation; and as a challenge that brings opportunity.

Commercial and industrial energy purchasers already ask for more than the industry and their energy suppliers can provide. They want real-time, high-frequency data from a supplier who measures latency in days and frequencies in half-hours. The physical grid operates at 50Hz, and as a software engineer, you're no stranger to serving inside of 20 milliseconds.

These are seismic shifts for something that underpins everything in our society; these shifts are only now beginning to accelerate, and InRange is taking advantage.

Image by mckaysavage is licensed under CC BY 2.0

At InRange, we use AI to understand, forecast, and optimize generation and consumption within local energy markets, extracting more value from each electron. The more value we can generate, the quicker we grow and decarbonize.

We're using modern data pipelines to enable high-frequency measurement of renewable generation, hyper-local consumption, and carbon avoidance. Energy consumers need this level of data for real-time, 24/7 matching of their demand with zero-carbon energy.

We're productizing a buying experience that has traditionally been a slow enterprise-led process. Our product pairs the user with AI to deliver our business proposition quickly, efficiently, and with minimal input. Business buyers have tremendous purchasing power, so giving them access to great software allows them to buy more faster.

These technologies and approaches have never been applied to the electricity industry like this before. Our success will write the operating system for the emerging energy market - decentralized, digitized, and decarbonized.

Across the industry, there are exciting opportunities in product, data, and engineering. Beyond "just" a chance to work on an exciting and vital problem, at InRange, we're building a team that holds its values above all else. Here's what we're looking to hire. We'd love to see you join us.

Running remote GUI apps with the infinite power of the cloud.

Tom Harvey — Tue, 20 Sep 2022 09:42:48 +0000

Remote development environments allow us to shift our computing needs to the infinite processing and high bandwidth offered by the cloud. It creates a shared, collaborative space that's close to our production environment. The growing toolchain around them is an exciting part of a new developer experience. But I'd never used it with a GUI app until now.

The below takes about 10-15minutes, and you get a remote, powerful desktop environment ready to run whatever Linux GUI tool you need; scientific research, 3D modelling, or just renting a faster computer for a while.

QGIS is a tool to help load, analyse and visualise geospatial data. But they don't have a build for my M1-based Mac; plus, I want to load a lot of data into it.

I'd turn to VSCode remote development or a SageMaker Studio Notebook for command line or scripting work. For a GUI app, I need to add a desktop environment to Ubuntu and access that through VNC.

Start with an Ubuntu 22.04 EC2 instance.

I've used CDK to spin this up, but go with the console, the AWS CLI, or whatever is comfortable for you.

instance = ec2.Instance(self, "DevInstance",
    instance_type=ec2.InstanceType('t3.micro'),
    machine_image=ec2.MachineImage.generic_linux(
        {'eu-west-1': 'ami-0d75513e7706cf2d9'}
        # The above is Ubuntu22.04 in the Ireland region
        # If you use another region, you'll want to find
        # the AMI for your region
    ),
    block_devices=[
        ec2.BlockDevice(
        device_name="/dev/sda1",
        volume=ec2.BlockDeviceVolume.ebs(
        )
    ],
    vpc=ec2.Vpc.from_lookup(self, "VPC", is_default=True)
)
instance.role.add_managed_policy(
    policy=iam.ManagedPolicy.from_aws_managed_policy_name(
        "AmazonSSMManagedInstanceCore"
    )
)
CfnOutput(self, "InstancePublicIp",
    value=instance.instance_id,
)

Gain CLI access to the EC2 instance.

I'm using AWS's SSM Session Manager, so I don't need to worry about ssh keys or opening ports.

aws ssm start-session --target $AWS_INSTANCE_ID

Install a desktop environment and a VNC server.

I'm using Tiger VNC and securing it with a password. Do this as the ubuntu user

sudo su
su ubuntu

sudo apt update

sudo apt install -y \
    xfce4 xfce4-goodies \
    tigervnc-standalone-server tigervnc-common tigervnc-tools

You'll now need to do a "first run" of the vnc server to do some initial configuration - like setting a password:

vncserver

This will prompt you to create a password and optionally create a "view only" password so that user can only view. I didn't set a view only, you probably don't need that either.

We can now continue with the setup of the VNC server:

cat > /home/ubuntu/.vnc/xstartup<< EOF
#!/bin/sh
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
/usr/bin/startxfce4
[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
x-window-manager &
EOF

chmod u+x /home/ubuntu/.vnc/xstartup

echo ':1=ubuntu' | sudo tee -a /etc/tigervnc/vncserver.users

sudo systemctl start tigervncserver@:1.service
sudo systemctl enable tigervncserver@:1.service

Allow access to port 5901 on the server.

I've used SSM Session Manager to create a tunnel. This way, localhost:5901 tunnels across to remote:5901; saving me from opening that port in AWS and making everything more secure.

aws ssm start-session --target  $AWS_INSTANCE_ID \
    --document-name AWS-StartPortForwardingSession \
    --parameters \
    '{"portNumber":["5901"], "localPortNumber":["5901"]}'

Open a VNC client connection.

macOS comes with one! Or, download a VNC client for your machine

In the Mac's finder, use the "Go" menu, select "Connect to server" (or press command-K), and add the VNC connection path:

vnc://127.0.0.1:5901

Now, I have the desktop environment of that Ubuntu server in a window on my local Mac. I can install QGIS on that server or whatever GUI app I want. That remote machine has an ultra-fast network connection and only needs to stream the (low bandwidth) screen activity down my network connection. I can keep costs down by using a small AWS instance, or if I need to load more data, I can have up to 24TB of RAM and hundreds of CPUs! My lowly laptop can really be a wolf in sheep's clothing.