DEV Community: Richard Forshaw

Serverless Email Processing

Richard Forshaw — Sat, 29 Apr 2023 03:35:56 +0000

There is a lot of information in the emails that fill our inboxes, and it is such a well-known form of communication that it's surprising that it's usually overlooked when providing an interface to a system. In this article, I'll go through how to set up basic lambda processing of emails using SES, Cloudformation and Serverless.

SES setup

First we need to set up receiving emails, which is done with SES. In order to capture incoming emails you will need:

An email identity in SES
An email address and DNS rule
A rule for what to do with the email when you get it

In this section we will go though these.

Domains

First a word on domains. To receive email through a 'real' email address you will need a domain. There are two ways to purchase domains in relation to this article: either in AWS Route53 or externally. I have done both and so I will describe both. In some ways external domains are better and in some ways Route53 ones are better. After experiencing both I will probably usually go with external domains but in some cases I will pick Route53. As with many engineering choices, it depends on the needs of the project or product.

Common SES Setup

To receive and process emails with Amazon SES, you will need the following:

A verified email identity
A SES rule set
A SES rule for incoming email

The rule set and its associated rules can be built with cloudformation but the email identity cannot. In order to do this, go to the AWS SES dashboard and then to the 'Verified Identities' section. From there click on 'Create Identity', to see the following form.

AWS have updated their domain verification to only use DKIM, and if you have a domain with Route53 then you can check a box to automatically publish the records to your hostedzone, which is great. This means that you don't have to put anything in to Cloudformation, but it does mean that these changes are not version-controlled in any way, which may or may not suit you.

Setting up email with Route53

So you have a Route53 domain. This is great for Infrastructure-as-code as you can specify your configuration in a Cloudformation stack which means it is easy to version-control and re-deploy if necessary.

Configuring Route53 to handle receiving emails only requires a few fairly simple Cloudformation resources:

A Route53 Hosted Zone
An incoming email MX record

A Cloudformation example of this is below:

Resources:
  MyHostedZone:
    Type: AWS::Route53::HostedZone
    Properties:
      HostedZoneConfig:
        Comment: "Domain registered with AWS"
      Name: "mydomain.com"

  # Forward email to us-west for SES processing
  IncomingEmailRecord:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId:
        Ref: MyHostedZone
      Name: "somefunction.mydomain.com"
      Comment: Incoming email record for sub-domain
      Type: MX
      TTL: 900
      ResourceRecords:
        - "10 inbound-smtp.us-west-2.amazonaws.com"

It is at this point that you should be aware of the limitations on SES services. SES is only available in some regions, and that list of regions is even smaller for handling incoming email. As of writing this article, those regions are as follows and are described here:

US-East-1 (N. Virginia)
US-West-2 (Oregon)
EU-West-1 (Ireland)

In order to receive and process emails, it is best to set up your infrastructure here, however because buckets are global you can probably have lambdas and downstream storage hosted in another region if that is beneficial, but it will make deployment and management more complicated.

You will see that the RecordSet item refers to a subdomain. I prefer to do this so that various functions which share processing requirements are grouped under a subdomain. Root domains can be left for general things like sending actual service-related emails.

External DNS

If you have an external DNS, you will need to add the DKIM keys by hand. This is straightforward but you should follow any guidance provided by your DNS provider.

The MX record must also be entered by hand, and is typically a record with the following information:

Type: MX
Name: somefunction.mydomain.com
Priority: 10
Target: inbound-smtp.us-west-2.amazonaws.com

The fields will vary from service to service.

Once the DKIM and MX records are set up, if they are correct then you should receive an email through your AWS account to say that verification is successful. If not, check the SES dashboard periodically.

SES Rules

Now we can move on to the SES rules. This is required to inform AWS what to do with an email that it receives. Setting up a rule requires you to define a rule set and also define rules within that set.

A common example is to simply store the incoming email so that you can process it later. In CloudFormation this looks like this:

  EmailReceiptRuleSet:
    Type: AWS::SES::ReceiptRuleSet
    Properties:
      RuleSetName: 'my-email-rule-set'

  EmailRule:
    Type: AWS::SES::ReceiptRule
    Properties:
      RuleSetName: !Ref EmailReceiptRuleSet
      Rule:
        Name: 'my-incoming-email-storage-rule'
        Enabled: True
        Recipients:
          - somefunction.mydomain.com
        Actions:
          -
            S3Action:
              BucketName: !Ref EmailStorageBucket
              ObjectKeyPrefix: "IncomingEmail/"
    DependsOn: EmailBucketDataStoragePolicy

This should be straightforward to understand: The ReceiptRuleSet is simply a container, and the actual rule instructs AWS to store any email addressed to somefunction.mydomain.com to be stored in the referenced storage bucket under the prefix IncomingEmail/. This acts as a kind of folder in S3.

You will notice that there is a reference to an EmailBucketDataStoragePolicy. This is important as you need to allow SES to be able to write to your bucket. This will look something like this:

  EmailBucketDataStoragePolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref EmailStorageBucket
      PolicyDocument:
        Statement:
          -
            Sid: "AllowSESPutObject"
            Effect: "Allow"
            Principal:
              Service: "ses.amazonaws.com"
            Action:
              - "s3:PutObject"
            Resource: !Join
              - ''
              - - !GetAtt EmailStorageBucket.Arn
                - '/*'

This Bucket Policy allows SES to perform s3:PutObject on the referenced bucket. Without this, you will be faced with some hard-to-diagnose permission problems.

(I haven't included the resource for the actual S3 bucket here - that should be a straightforward exercise for the reader)

Gotcha: Enabling the RuleSet

One thing that is a bit hard to find in the documentation is the fact that you need to manually enable the ruleset. You can enable the rule in Cloudformation but the ruleset must be enabled through the console. To do this, go into the 'Email Receiving' menu item, select your ruleset and enable it.

Lambda Setup

This is all the infrastructure set up - we now have:

A domain which sends email to AWS servers
A bucket to store the emails (set up separately)
A SES rule which listens to incoming mail and writes it to the bucket

Now we need to process it.

Readers should note that you don't have to write to the S3 bucket first, you can configure the rule to invoke a lambda function straight away, but I find it is useful to store the email anyway. So now we will trigger a lambda from when the email is stored to S3.

Serverless

I like using the Serverless framework. I just think that it makes things nice and easy and minimises what you need to do to get a lambda up and running.

We do need to do something though, and the basics are this:

Set up the provider details
Make sure we have a deployment role
Give our lambda some permissions to access S3
Define the function and the trigger event

Item 1 can be done when you set up your Serverless project, and all the info you need is on the Serverless tutoral page.

Deployment Roles and Permissions

Assigning the role and permission is probably the trickiest thing. All the big cloud providers implement some kind of security model, which is a good thing. It's just up to us to use it!

If you run through the Serverless setup, it will allow you to deploy using your local access keys, but this is not typically recommended. I generally provide a role for Serverless to assume, which restricts what it can and can't do. Setting it up is a bit cumbersome but it means you have full control over the permissions.

At minimum, it needs to:

Be allowed to assume Cloudformation and Lambda roles
Be allowed to pass a role to another service
Be allowed to perform certain functions on your resources

A little explanation of the above points:

is in order for Serverless to actually perform the deployment and create your resources
is a special case, which will be explained soon
is so it can perform actions on your resources

Because Serverless prefixes your project's services with the service name, I typically write the IAM PolicyDocuments to be limited to only those resources, which looks like this:

  ServerlessDeploymentRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: My-Serverless-Deploy-Role
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - 'cloudformation.amazonaws.com'
                - 'lambda.amazonaws.com'
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyName: ServerlessDeployPolicies
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow   # Allow S3 actions to buckets with service prefix.
                Resource:
                  - 'arn:aws:s3:::service-prefix*'
                Action:
                  - 's3:*'
              - [More permissions...]

A Special Case

I usually make an architectural decision to separate the incoming email infrastructure with the business logic in the lambda code. You don't have to do this, but I do it for two reasons:

Serverless is not great at handling resources that are not tightly coupled to your lambdas. It does allow you to add custom resources at the end of your serverless.yml file, but this is typically just CloudFormation code anyway, and brings us to the next point...
The email pipeline should be a more permanent fixture, and shouldn't be tightly coupled to the serverless deployment in my opinion. If you want to tear down and re-deploy your lambdas, if your email pipeline is managed by the serverless.yml file then it will be torn down as well. I think it is better to de-couple them.

Because of this, serverless needs to be told that you have a pre-configured S3 bucket and it needs to run some custom lambdas to be able to access it. In order to do this, the following policy needs to be added to the ServerlessDeploymentRole:

  MyServerlessDeploymentRole:
    # Existing stuff...
      Policies:
        - PolicyName: CustomResourceLambdaRole
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow   # Allow all IAM actions to roles prefixed with service
                Resource:
                  - !Sub 'arn:aws:iam::${AWS::AccountId}:role/My-Serverless-Deploy-Role'
                Action:
                  - 'iam:PassRole'

In essence, the role needs to be allowed to pass itself to another service. This is required so it can install and run a custom lambda to be able to configure your existing S3 bucket. (Once again, if you are happy to let Serverless handle your S3 bucket as well then you won't have to do this.)

Finally, this role must be named in your new serverless.yml file:

provider:
  name: aws
  # Your provider configuration...

  # AWS IAM specifications here
  iam:
    deploymentRole: arn:aws:iam::${aws:accountId}:role/My-Serverless-Deploy-Role

Lambda Permissions

Next we have to grant permissions to the lambda that we are going to create. It will need to be able to read the email that has been stored in the S3 bucket, so we will need to access the ARN of the bucket you are using. In the provider/iam section of the serverless.yml, add the following:

provider:
  # AWS IAM specifications here
  iam:
    role:
      statements:
        # Allow access to S3 objects
        - Effect: "Allow"
          Action:
            - "s3:ListBucket"
          Resource:
            - # Put S3 ARN here
        - Effect: "Allow"
          Action:
            - "s3:GetObject"
          Resource:
            - Fn::Join:
              - ''
              - - # Put S3 ARN here
                - "/IncomingEmail/*"   # Matches the SES rule action

There are a few ways to reference the S3 ARN. I like to export it from the Cloudformation and then reference it in serverless.yml using ${cf:stack-name.output-name}, but you can do all kinds of funky stuff with Serverless variables to make sure you don't repeat yourself.

One important detail is to make sure that the permission you grant to read from matches the location that you configured to write your email to in the SES rule. Once again, there are probably some tricks you can do to make sure these are synchronised.

Make the function

Finally, we can make our function! The contents of the function is out of scope of this article, but I will say that it is worth reading the Python Email Parsing Library Documentation. It undoubtedly contains information you will need if you are writing in Python. (There are NPM email-parsing packages for Javascript, but I like the fact the Python one is built-in).

The final thing to do is configure the lambda in serverless.yml:

functions:
  email_processor:
    handler: handler.process
    events:
    - s3:
        bucket: # Put bucket name here
        event: s3:ObjectCreated:*
        rules:
          - prefix: IncomingEmail/
        existing: true   # We are using an existing bucket

There are three important things to note here:

You need to supply the bucket name to the s3 event, NOT the bucket ARN. This usually trips someone up.
Make sure the S3 object prefix matches the SES rule. (Of course you don't need to use a prefix, but I think it is more informative)
Because we are using an external S3 bucket, we must specify existing: true. (Reminder: if you have chosen not to manage your own bucket then you don't need this and serverless will create it for you, BUT tearing down the function will also delete the bucket.)

Wrap Up

It may not seem simple but there is not that much to do in order to capture emails in AWS, and there are many applications. Once capturing and processing is done, SES can be configured to send emails as well. There are probably many applications which can be bootstrapped into a POC using email instead of building and deploying a website, and some users will find interacting through emails much easier anyway.

So go forth and keep email alive!

ChatTDD: Using AI as your Programming Buddy

Richard Forshaw — Mon, 24 Apr 2023 02:54:46 +0000

Previously I wrote about transforming an idea into working code using ChatGPT as an assistant. I took an idea and put ChatGPT through a BDD process and an Example Mapping process.

I was very impressed with the results, but also a bit disappointed from its lack of detail and also lack of determinism. It was not enough to rely on, but I do think it made a useful tool to simply speed this process up.

What about the second step - transforming an example scenario into code?

Picking up from Requirements

In the last post, ChatGPT helped with creating BDD-type scenarios for a simple email parser. Now we will try to convert some of those scenarios into tests and code, using a typical TDD process. We will use one failure case and one success case. The cases are not exactly how ChatGPT gave them to us because as we learnt it was not very good at going down to the detail level. So in between that step and this step there needs to be some manual review and refinement of the cases.

[In this post we will look at the failure case. The full article has details of implementing a success case.]

The failure case:

Scenario: Email with no file attachments returns error
  Given an email with no image file attachments
  When the email is sent to 'The System'
  Then the system should return an error code with the message "One image file attachment must be present."

Setup

Configure ChatGPT

To turn ChatGPT into a developer, I gave it the following system command:

You are an expert python software developer. You know how to understand BDD scenarios of the form:

Scenario: scenario title
Given (an initial context)
When (an event occurs)
Then (a testable outcome is satisfied)

You know TDD and how to write tests first using pytest and pytest-bdd. When writing code, you favour functional programming and avoid side-effects.

You have been asked to assist another python software developer to develop tests and code for a small library to parse email content according to the given BDD specifications.

Test 1: Simple failure case

Writing the test case

I prompted ChatGPT with this prompt:

Given the following scenario:

<scenario from above>

Write a pytest-bdd test which satisfies this scenario.

and I received the following code:

import pytest
from email_parser import parse_email

@pytest.mark.parametrize("email_content, expected_output", [
    ("Hello World", "One image file attachment must be present."),
    ("", "One image file attachment must be present."),
    ("image.jpg", "One image file attachment must be present."),
])
def test_parse_email_no_file_attachments(email_content, expected_output):
    # Given an email with no image file attachments
    email = {
        "content": email_content,
        "attachments": []
    }

    # When the email is sent to 'The System'
    output = parse_email(email)

    # Then the system should return an error code with the message "One image file attachment must be present."
    assert output == (False, expected_output)

Not quite the start I was expecting. Some things to note:

It assumed a name for the module and for the function under test, which is fine. We will see if it remembers this
It assumed an input format to the function under test as a single json structure
It assumed an output from the function under test as a tuple.
It has not used the pytest-bdd library as we requested in the system message
The parameterization of the test is not really necessary. The test really only needs one example of having no attachments.

The second item is interesting, and I like it. The design that initially came into my head was that the function in question would parse the entire email message and handle the extraction of body text and attachments. When I initially saw the output from ChatGPT I was annoyed but then it made me think: this is actually a better solution. Perhaps it was even following my direction to program in a functional way?

I like writing things in a functional way. Typically it is good to write simple functions that do only one thing with no side-effects, which means they are easier to tests. In the long run I probably would have refactored the code to do this, but the ChatGPT output made me take this decision earlier, and I changed the design in my head to first parse the email into its sub-parts and then process the parts according to my requirements.

To fix the other issues, I instructed ChatGPT to refine the solution, by asking it to:

use the pytest-bdd library format, providing it with a scenario file name
return a json structure instead of a tuple

The updated test function was this:

import pytest
from pytest_bdd import given, when, then, scenario
from email_parser import parse_email

@scenario('../features/email-parsing.feature', 'Email with no file attachments returns error')
def test_parse_email_no_file_attachments():
    pass

@given("an email with no image file attachments")
def email_with_no_file_attachments():
    email = {
        "content": "Hello World",
        "attachments": []
    }
    return email

@when("the email is sent to 'The System'")
def send_email_to_system(email_with_no_file_attachments):
    return parse_email(**email_with_no_file_attachments)

@then("the system should return an error code with the message \"One image file attachment must be present.\"")
def check_error_message(send_email_to_system):
    expected_output = {
        "success": False,
        "error_message": "One image file attachment must be present."
    }
    assert send_email_to_system == expected_output

That looks like a pretty good bdd-style test to me.

Implementing Code

When I ran the code, I got the usual errors, as you would expect from real TDD:

No module
No function
No scenario file

I fixed these up and then asked ChatGPT to write the code.

Now write the code which satisfies the test function.

The response was this (line-splitting done by me):

def parse_email(content, attachments):
    # Check if there are any image file attachments
    has_image_attachment = False
    for attachment in attachments:
        if attachment.get("name", "").lower().endswith(".jpg") \
            or attachment.get("name", "").lower().endswith(".jpeg") \
            or attachment.get("name", "").lower().endswith(".png"):
            has_image_attachment = True
            break

    # If no image file attachments are present, return an error message
    if not has_image_attachment:
        return {
            "success": False,
            "error_message": "One image file attachment must be present."
        }

    # If one or more image file attachments are present, return a success message
    return {"success": True, "message": "Email parsed successfully."}

The interesting thing for this first implementation is that because we had been talking about image attachments, it gave me code which already handled different types of image extensions. This was good but it is against the TDD philosophy of writing just enough code to pass the test. I decided to leave it in however.

Personally I don't find this code readable, and for the future it is likely the code will accept some extensions and not others, so we can ask ChatGPT to refactor it to use the split command and the in operator.

Running the test

The moment of truth... does this work? I ran pytest and got the following error:

E       fixture 'email_with_no_file_attachments' not found

Hmmm... why? Well, it turns out that the test code generated by ChatGPT had used the pytest 'fixture' style to write the tests, but it had been implemented incorrectly. It's possible that this style is too new for ChatGPT or it just didn't know how to do it. But it is something I had to dig into and fix, which took about an hour.

After fixing that, hey presto, test passed! The question remaining is: considering the time it took to fix the error, would it have been faster for me to write this myself?

Summary

This was a valuable experiment with many new things learnt about using ChatGPT in this way.

The main benefit I found was that it prompted me to think more about design decisions - it did almost feel like I was 'pair programming'. However I was hoping that I would be able to continue thinking at a high level, but that didn't happen; because some of the output had mistakes or was poorly written (in my opinion), I had to jump between design-level and code-level thinking.

This raised an interesting question though - does this in fact fit into the TDD 'Red-Green-Refactor' process? Maybe it does and I have just been executing the 'refactor' stage a bit too early? I admit I sometimes neglected to wait to see if the code that was produced passed, thus following the 'write the minimum code to turn the test green' rule.

Other associated points are:

defining the expected inputs/outputs for unit tests is a bit laborious
Some responses introduced assumptions in the details (e.g. I asked it to use URL-encoded-strings and it used outdated formats which I had to fix)
The output is often verbose, leading to lengthy, duplicated or redundant code. This was especially true when parameterizing tests.

Some of these may be improved in GPT4, we will have to see.

What about writing the more complex success-case of the experiment? You can read that at the original post

ChatBDD: Using AI in a BDD Process

Richard Forshaw — Wed, 12 Apr 2023 01:33:47 +0000

In embarking on a new side-project, I was faced with writing code again. I love writing code, but it certainly takes mental effort. I have noticed that writing entire modules on my own requires me to think about multiple things and draw an many areas of experience, from test design and using the core language libraries all the way up to trying to keep the overall goal and high-level architecture in mind. And at the same time trying to keep things simple.

Usually in this situation I would simply start coding (tests first of course), but sometimes I would fail to stick to a clear coherent path, instead getting lost in a sea of ideas due to my excitement in how amazing the new system would be. So could I use ChatGPT to help me focus, while using a BDD or TDD process?

Let's find out.

The Method: From behaviour to code

I decided to take this approach:

Define the intended outcome of my module in order to write scenarios
Use each scenario to turn into tests
Use the tests to write the code

I believe there is a huge benefit of using ChatGPT to do this, especially point 3, because as humans sometimes we can't help but be creative, but TDD dictates that once you write the test you should write the code to make it pass and ONLY enough code to make it pass. Thus using ChatGPT makes it easier to stick within this pattern.

Define the outcomes

When I was consulting, there were two key outcomes that I looked for when preparing for a sprint: defining the general desired behaviour and providing specific examples. This typically meets a 'definition of done', in that the general behaviour of the system is described and there are some testable examples which allow the developers to ensure that the code that they have written meets some specific testable criteria.

I started off asking ChatGPT this:

You are an expert business analyst with additional expertise in Software User Stories and BDD. You help turn business requirements given as User Stories into multiple BDD Given-When-Then scenarios.

A User Story is of the format:
As a (user type)
I want to (perform an action on the system)
So that (I meet a desired outcome)

A Given-When-Then scenario is of the format:
Given (an initial context)
When (an event occurs)
Then (a testable outcome is satisfied)

---

I am writing a software module which accepts an email from a user and extracts the components so that they can be used to post to Instagram. The user story is:

As an Instagram user,
I want to send an email to 'The System' which contains an image and some text
So that the image and text are extracted in a way that it can be posted to Instagram

Write BDD scenarios which meet the user story.

To my amazement, ChatGPT output the following:

1. Scenario: Email with image and text is successfully processed
   Given the email contains an image and some text
   When the email is received by 'The System'
   Then the image and text are extracted from the email
   And the image and text are formatted for Instagram posting
   And the formatted image and text are saved to a file

2. Scenario: Email with no image is received
   Given the email contains no image
   When the email is received by 'The System'
   Then an error message is returned
   And the email is not processed further

3. Scenario: Email with no text is received
   Given the email contains no text
   When the email is received by 'The System'
   Then an error message is returned
   And the email is not processed further

4. Scenario: Email with an image that cannot be processed is received
   Given the email contains an image that cannot be processed
   When the email is received by 'The System'
   Then an error message is returned
   And the email is not processed further

5. Scenario: Email with text that exceeds the character limit is received
   Given the email contains text that exceeds the character limit
   When the email is received by 'The System'
   Then an error message is returned
   And the email is not processed further

6. Scenario: Email with an image that exceeds the size limit is received
   Given the email contains an image that exceeds the size limit
   When the email is received by 'The System'
   Then an error message is returned
   And the email is not processed further

7. Scenario: Email with unsupported image format is received
   Given the email contains an unsupported image format
   When the email is received by 'The System'
   Then an error message is returned
   And the email is not processed further

8. Scenario: Email is received from an unauthorized email address
   Given the email is received from an unauthorized email address
   When the email is received by 'The System'
   Then an error message is returned
   And the email is not processed further

9. Scenario: Generated Instagram post is successfully saved to file
   Given an image and text are formatted for Instagram posting
   When the formatted image and text are saved to a file
   Then the file is saved successfully

10. Scenario: Generated Instagram post fails to save to file
   Given an image and text are formatted for Instagram posting
   When the formatted image and text are saved to a file
   Then an error message is returned
   And the file is not saved successfully

What a start! I am usually able to come up with reasonable failure scenarios, and the first thing I noticed was there were scenarios here which I hadn't even considered yet, for example the character limits and attachment size limits.

The fact that this was generated in about a minute should not be overlooked. I estimate that this would take me about half and hour to think of an type up. So we are looking at an incredible speed-up.

Examining the output, there were a few things that for an MVP I did not consider important since I would be the only user (un-authorised email addresses and image format). There were also some things which were missing, probably due to some vagueness in my starting story. So I asked for some refinements:

Please make the following changes:

 1. The image is received as an attachment and only one attachment is allowed.
 2. The outcome is not a formatted image. The outcome is that the image is extracted and saved, and the text is extracted and saved in a suitable format for subsequent processing.
 3. The text in the email can specify multiple image captions and a carousel caption.
 4. The product is an MVP and so scenarios 6, 7 and 8 are not required

I will spare the duplication, so here are the updated scenarios that it created:

1. Scenario: Email with one image attachment and text is successfully processed
   Given the email contains one image attachment and text
   When the email is received by 'The System'
   Then the image and text are extracted from the email
   And the image and text are saved in a suitable format for subsequent processing

2. [unchanged]

3. Scenario: Email with more than one image attachment is received
   Given the email contains more than one image attachment
   When the email is received by 'The System'
   Then an error message is returned
   And the email is not processed further

4. Scenario: Email with text specifying multiple image captions and a carousel caption is received
   Given the email contains text specifying multiple image captions and a carousel caption
   When the email is received by 'The System'
   Then the image captions and carousel caption are extracted from the text
   And the captions are saved in a suitable format for subsequent processing

5. Scenario: Extracted image and text are successfully saved
   Given the image and text are extracted from the email
   When the image and text are saved
   Then the image and text are saved successfully

6. Scenario: Extracted image and text fail to save
   Given the image and text are extracted from the email
   When the image and text are saved
   Then an error message is returned
   And the image and text are not saved successfully

Hmm... ok, so it removed 6, 7 and 8 as requested, but it also removed scenario 3, 4 and 5 which was weird. This then drew my attention to the fact that scenarios 5 and 6 were not really saying much. I decided to keep scenario 6 because maybe there would be an internal fault which is a fair test case, but scenario 5 is basically the same as scenario 1.

I have found that directing existing output from ChatGPT can be tricky and frustrating, but I could still hand-pick the best of all the outputs. However these scenarios are still quite high-level - what about that missing detail? I decided to try another approach.

Refinement: Example Mapping

When I was introduced to Example Mapping I loved it. As a developer, if given a detailed user story or requirement, I always found myself thinking of examples to test and often found that I was unable to think of them or that they were incorrect because of a misunderstanding.

Lack of examples could be fixed mid-sprint by talking to a stakeholder, but incorrect scenarios would not usually be found until the end of the sprint. Example Mapping seeks to change this, and involves stakeholders, developers and QA discussing the scenarios to define examples of what they mean.

The scenarios from the first experiment may already look complete, but they use language such as 'an error message is returned' which is not specific or testable. In order to get more detail from ChatGPT, I decided to put it in an Example Mapping scenario instead. Since I could wear at least 2 of the caps from an Example Mapping meeting, I could direct ChatGPT to write some examples. I used the rules for Example Mapping together with the rules for BDD scenarios and pushed through the user story with some more request details.

You are a software product specialist with knowledge of BDD, User Stories and Example Mapping. You are participating in an Example Mapping session. This session follows these rules for a given user story:

Start by writing the story under discussion on a yellow card.

Next write each of the acceptance criteria, or rules that we already know, on a blue card.

For each rule, we may need one or more examples to illustrate it. We write those on a green card and place them under the relevant rule.

Examples are written in the BDD Given-When-Then scenario format as follows:
Given (an initial context)
When (an event occurs)
Then (a testable outcome is satisfied)

As we discuss these examples, there may be questions that nobody in the room can answer. Capture those on a red card and move on with the conversation.

---

Take the following user story:

As an Instagram user,
I want to send an email to 'The System' which contains an image and some structured text
So that the image and text are extracted in a way that it can be posted as an Instagram post or carousel

Take the following rules:

1. The email must contain 1 image file attachment
2. The email must have a subject "INSTAGRAM POST <date>"
3. The email must contain some body text
4. The body text must be structured to create either a single image post or a carousel post of 2 or 3 images
5. The structured body text must contain a general post caption and a caption for each desired image in the post
6. Error messages are returned as a HTTP 400 code with an associated message

Write the examples as per an Example Mapping session, and also write any unknown questions.

I won't bore you with the text output - instead lets put them onto cards as they would be in a real Example Mapping session. Unfortunately image generators are not good at producing text (yet), so this one is a manual effort - please excuse any poor formatting as some output was quite long.

At first glance, this looks pretty good, and once again it only took about a minute to produce it. However there are some issues, and in a real session I wouldn't let this out of the room. Let's look at the positives first.

ChatGPT correctly structured the cards (in the text output the examples were correctly aligned under the rules)
ChatGPT did make examples according to the rules, and distinguished between input scenarios
The additional questions blew me away - they are all valid: attachment naming, additional body text and receipt address

Now the negative:

The user story outcome has again been interpreted as posting to Instagram, when it is not
The failure scenarios are testable but the success scenarios are not really testable. I prefer examples describing outcomes which can be translated almost directly into a test.
The carousel scenario is missing a test for 3 captions
The carousel scenario is missing a failure case for more than 3 captions

Now, you can say that these are because it has not been given sufficient direction (again). In the first two points this is perhaps true, but in the final two I think there is enough information for a human to have understood (like the rule for images). But this is often what is discussed in Example Mapping meetings.

Since this is an experiment with ChatGPT, let's try again with more direction, and answer some of the questions. Here is the updated request and rules:

Take the following user story:

As an Instagram user,
I want to send an email to 'The System' which contains an image and some structured text
So that the image and text are extracted and stored so that it can be later used to generate an Instagram post.

Take the following rules:

1. The email is addressed to "me@postrobot.com"
2. The email must contain 1 image file attachment and can have any name
3. The email must have a subject "INSTAGRAM POST <date>"
4. The email must contain some body text
5. Body text containing "POST TEXT" and "CAPTION" labels create a single image post
6. Body text containing "POST TEXT" and "CAPTION-1"..."CAPTION-N" labels create a carousel post
7. A carousel post contains no more than 3 images
8. Text extracted from the email is stored as JSON, containing the date, post text, captions and image name
9. Error messages are returned as a HTTP 400 code with an associated message

Write the examples as per an Example Mapping session, and also write any unknown questions.

For brevity, I will spare the detailed output, but below are the key results from the response:

The output now only contained a positive example scenarios, whereas previously it also contained negative ones
Because there were no negative scenarios, the only output involving an error (responding to (9)) was "Then a HTTP 400 code with the associated message is returned", which was not helpful
Some of the questions it now raised had already been deduced in the previous attempt, e.g. now it raised the question: "What happens if there is no image attachment in the email?". I think a human would have deduced this from rules (2) and (9)

Conclusion

Partnering with ChatGPT to expand a user story into testable scenarios once again showed how a LLM such as this is very useful from a general stand-point but not from a detailed one. On the positive side, all the answers above were obtained in seconds, meaning I spent more time analysing the results and looking for errors than typing the prompts and waiting for output.

All in all I probably only spent a few minutes actually interacting with the system. It gave impressive results to allow me to build up a list of scenarios to consider, and also questions that I might have missed. So in a sense, it acted very well as a team-mate.

The problems exist when searching for detail. Perhaps I am picky, but some of the output was not useful to move forward with. Using ChatGPT for high-level behaviour was good, but any thought of straying down to any level of detail left me disappointed. If it is common to re-prompt with more detail, then I might as well just add the detail myself.

I also found a lack of consistency between prompts. It mostly performed well but it forgot previous outcomes enough to frustrate me. If I saw this behaviour in a software developer working on my team, it would annoy me.

Perhaps this is expected - after all LLMs do not 'think', they simply perform highly-advanced statistical pattern-matching. For many purposes this is sufficient and provides plenty of useful responses and food for thought. But at the moment it takes a human to go that step further. (Note that this was done on ChatGPT3.5. Perhaps GPT4 goes further).

Overall, ChatGPT gets top marks as a productivity assistant. I suspect that the swift generation of output, with a little extra consideration on the inputs, would save a lot of time in the real world, if not simply for automating the output of boilerplate text. So I can see a huge benefit.

This was originally posted on my blog. To see part 2, head on over!

3 Kent Beck Software Insights

Richard Forshaw — Wed, 08 Mar 2023 03:08:33 +0000

I recently read Kent Beck's book 'Implementation Patterns'. While most of it is focused on Java and OO-programming, there is still much that he talks about which is relevant to programing and software development as a whole.

Programming is hard, and developing large-scale software long-lived software is harder still as it involves disciplines and behaviours that aren't taught in most programming courses, or at least they weren't when I was at University. Perhaps now there is more importance given to them, but in case there isn't here are three things from Kent's book which I think every developer needs to consider.

1. Cost

When you step out of a programming course and into a software development team, you are entering a much different world than the one that you left. The one you left was concerned with how to tell a computer to solve a certain problem for you, whereas the new one adds time and cost constraints into the mix, together with the dreaded 'L'-word (Legacy), and nobody really prepares you for that.

Kent dedicates a whole chapter to software cost; a short chapter but a chapter non-the-less. As with any business department or function, cost will eventually become a driving force behind its operation. This is of course still true today but still not fully understood by many developers.

Costs come in many forms, and Kent cites Yourdon and Constantine, saying that the cost of maintaining a software system is usually greater than the cost of developing it (in this case the book defines maintenance as including bug-fixing and modification). This is most likely still true today even with modern engineering techniques.

As Kent says under his laws of programs: "There is no such thing as 'done'". These days we tend to be aware of this fact, but he breaks this maintenance cost down even further, saying that it it is made up of:

Cost of understanding
Cost of changing
Cost of testing
Cost of deploying

On the first of these, Kent says "Learning what the current code does is the expensive part". This is certainly true today, because software engineering is still knowledge-work, and knowledge-work requires a depth of understanding prior to execution. Sadly much of this understanding is still lost when transferring a system, module or even function between developers and my career has seen many and continued attempts to fix this.

It is worth understanding the impact of these costs but you should also recognise that the costs are just a symptom - an indicator (albeit a critical one) that allows us to see where there may be another problem lurking.

Two approaches to lower these costs that may be elicited from Kent's book are:

fix the cost of understanding by improving communication; and
consider that costs can be reduced if there were less things to change or overhead to change them

We will look at these both below.

2. Communication and Naming

Communication is one of the most underrated properties of software programs, but one of the most dualistic. I wager there is not a developer reading this who has been frustrated not being able to understand another developer's code as quickly as they wanted. But by inference, it means that the code they have written has caused the same frustration. As demonstration, I willingly put my hand up and admit that I have re-visited code I have written within 12 months and been initially confused as to what it is doing.

One of my favourite quotes from Donald Knuth is this:

"The best programs are written so that computing machines can perform them quickly and so that human beings can understand them clearly."

And personally, I would not lose too much sleep if the "computing machines" do not perform them quickly, since not all programs need to run fast but they all need to be read by other programmers.

The biggest influence we have over a computer program, aside from how it is designed, is the words that we choose when writing it. Kent and I share the values that communication, and by extension, naming variables, functions and classes are among the most important considerations and deserve thought.

"Finding just the right name is one of the most satisfying moments in programming"

It is good that your algorithm keeps thousands of messages running through your system, but if the next developer to maintain your code doesn't realise its function easily then it is likely to spell trouble.

Kent's advice is:

"Convey purpose, type and lifetime of the variable to readers"

I admit that even I learnt something when reading this as I would rarely consider considering the lifetime in this equation, but it is important and often indicates the relevance and importance of a variable within the code.

A worthy example of appropriate naming is in Kent's description of an "Explaining Message". This is defined as a message (i.e. a method on an object) which distinguishes between its intention and its implementation. Specifically, Kent recalls when he saw a method defined like this:

highlight(Rectangle area) {
    reverse(area);
}

"Why is this useful?" he asked himself... why not just expose reverse() on the object. But this displays exactly what is meant between intention and implementation. The intent (i.e. you could say the API action that is useful to the user) is to highlight. Internally however, it is implemented as reversing the area of the given shape. Subtle but very useful.

3. Change and Trade-offs

The other way of looking at reducing the cost of maintenance is to reduce the likelihood of change that will be required. If you incur a cost per change then you can either reduce the cost (as above) or reduce the number of changes. This however is a slippery slope.

In saying this, we are advocating for introducing flexibility into the system, but we must recognise there are two types of flexibility:

one driven by the actual requirements
one driven by second-guessing future requirements

Kent points out that flexibility comes with a cost, and usually introduces complexity which may be unnecessary:

"Programs should be flexible but only in ways they change. If [something] never changes, all that complexity is cost without benefit."

This shares much with another of Donald Knuth's famous quotes:

"Premature optimization is the root of all evil."

Kent uses a feature of OO to demonstrate this in the section about behavioural messages. One beautiful thing about OO programming is polymorphism, a feature allowing conditional decisions to be made at run-time in an elegant way instead of what he calls 'explicit conditionals' (i.e the control-flow statements built-in to the language). Explicit conditionals are easy to read but provide little flexibility when it comes to extending behaviour. Polymorphism provides a method to control the execution flow in a more flexible way which does not require existing code to be changed (basically the Open-Closed Principle).

Kent however recognises that while this makes code more flexible, it requires more skill to read, understand and write, and introduces problems in communicating to other programmers. Thus it is a trade-off in the change domain.

The association of flexibility with complexity is one of programming's many great trade-offs, and because of this flexibility should be introduced judiciously. One approach to this is by doing agile development (and doing it correctly). Because agile (e.g. Scrum) involves releasing working software often, you must make the flexibility vs simplicity decision early and within your delivery cadence constraints. I suspect that this will typically come down on the 'simplicity' side, and thus validate Kent's statement that:

"The time to introduce flexibility is when it is definitely needed"

Instead of over-complicating with too much flexibility, Kent advocates for choosing simplicity and effective testing. I am a big proponent of TDD and automated testing and there have been many occasions where an effective suite of tests have given me confidence in being able to change code implementation safely and quickly. But only when it is needed.

Turbocharge Your AWS CLI Skills with JSON Parsing

Richard Forshaw — Thu, 09 Feb 2023 01:42:41 +0000

I am a self-professed want-to-be keyboard-wizard, and I was brought up on amazing text-manipulation tools like grep, sed and awk. Text-based data manipulation can still be very fast and powerful and might save you dozens of lines of code. This post shows how a little knowledge of JSON-parsing tools can go a long way.

Tools

The tools used in this article are as follows:

AWS CLI
JMES Path parser
JQ
Bash

This combination may mean that some examples below may not map exactly to your particular use-case or configuration, so please bear that in mind.

Basic JSON Formatting and Filtering

I looked at basic formatting and filtering in this post about essential AWS CLI skills. As a quick recap, you can use the --query command-line parameter to pass a string specifying a query expression on the JSON results, and sometimes these can get quite complex.

If you want a bit more power in your querying, it is worth looking at the JQ tool, which also lets you process JSON structures. The good thing about using JQ is that if can operate on files as well as STDIN, so you can save your JSON output into a file and run JQ over and over. This will definitely be faster when writing your queries and may also save you some AWS processing cost.

As a start, JQ is great for simply pretty-formatting JSON output from something like a lambda function, by just piping it into jq '.'. This isn't necessary with the AWS CLI because it already formats the JSON output.

AWS CLI examples

The simplest use of the CLI filter is to print a reduced amount of data so that it is more manageable. This uses JMES Path expressions to process the JSON data in the output.

Here are some basic examples. Note that the query expression is added using the --query argument. In this case we will use CloudFormation output:

aws cloudformation describe-stacks --query "<filter-goes-here>"

Use Case	Command
Show only stack ID, name and update time	`'Stacks[*].[StackId, StackName, LastUpdatedTime]'`
Show stack name/ID for stacks whose name contains 'foo'	`'Stacks[?StackId.contains(@, 'foo')].[StackId, StackName]'`
Show stack name/ID for stacks which have outputs exported and have been updated since Nov 2022	`"Stacks[?Outputs && LastUpdatedTime>'2022-11'].[StackId,StackName,LastUpdatedTime]"`

(Note the final example only works because the date format in LastUpdatedTime is able to be compared as a string - more on that later.)

JQ

If you want to do serious local-processing of AWS or any other JSON output, you need to get familiar with JQ, an awesome tool which lets you process JSON structures, not only filtering it like the AWS CLI, but also restructuring it.

Here are a few basic ways to use it. Note that you can either pipe input into JQ or provide a filename which contains your JSON.

Use Case	Command
Simply pretty-print JSON output	`jq '.'`
Get all the sort keys from a dynamo query response (assuming they are strings)	`jq '.Items[].SortKey.S'`
Put the result into a list	`jq '[.Items[].SortKey.S]'`
Only print keys, not values	`jq 'keys[]'`

An Important Note

It should be stressed here that QUOTING IS IMPORTANT. You may have noticed above that for the AWS --query I used double-quotes to enclose the whole expressions, and single-quotes when quoting literals within the expression. But for JQ, you use the opposite, which is made clear in the manual. At least within the environment I am using (bash), not following this will only lead to endless miserable debugging.

Comparing The Two

In general, the --query filters using JMES are a little more concise than their JQ alternatives, but in my opinion the sequential nature of JQ using pipes (|) is more readable than JMES. However, there are some other important differences to consider:

	CLI --query expression	JQ Expression
Usage	Only with AWS CLI commands	With any output or file
Output	Only outputs filtered results	Can restructure into new JSON
Types	Does not handle dates natively	Handles date conversions
Scripting	Expression must be entered on command-line	Expression can be store in file with comments

Below are some common queries I've used, with the CLI query and JQ query side-by-side. You should note that all the JQ expressions are wrapped in [], because by default JQ does not output a list. The AWS CLI query function does output a list, so the additional [] are used to match the outputs. For these examples, I am using DynamoDB output which looks something like this:

{
    "Items": [
        {
            "PartitionKey": { "S": "blog/books/2023-01-drive-daniel-pink/" },
            "SortKey": { "S": "1675071762" },
            "SomeField": { "N": "54" },
            // Other Fields...
        },
        {
            "PartitionKey": { "S": "blog/books/2023-01-drive-daniel-pink/" },
            "SortKey": { "S": "1675071862" },
            "SomeField": { "N": "44" },
            // Other Fields...
        },
        // More items...
    ]
}

Example	AWS CLI query expression	JQ Expression
Show all sort keys from dynamo output	`Items[*].SortKey.S`	`[.Items[].SortKey.S]`
Particular attributes from dynamo output	`Items[*].[SortKey.S,SomeField.N]`	`[.Items[] \| [.SortKey.S,.SomeField.N]]`
Filter by field value	`Items[?SortKey.S>'1674000000'].SomeField.N`	`[.Items[] \| select(.SortKey.S>"1674000000").SomeField.N]`
Filter on string prefix	`Items[?starts_with(SortKey.S, 'TEXT')].SomeField.N`	`[.Items[] \| select(.SortKey.S \| startswith("TEXT")).SomeField.N]`

If you want an example not using the data above, here is one you can run on your CloudFormation stacks right now. Each command (one JMES and one JQ) will show the last updated time of only your 'Dev'-stage stacks:

bash-5.1$ aws cloudformation describe-stacks --query "Stacks[?contains(Tags[], {Key: 'STAGE', Value: 'dev'})].[StackName,LastUpdatedTime]"
bash-5.1$ aws cloudformation describe-stacks | jq '[.Stacks[] | select(.Tags[] | contains({Key: "STAGE", Value: "dev"})) | [.StackName,.LastUpdatedTime]]'
[
    [
        "my-sls-stack-dev",
        "2023-02-03T07:56:49.848Z"
    ],
    [
        "my-www-stack-dev",
        "2022-12-20T11:07:21.155Z"
    ]
]

Getting more complex

Let's do some sorting. Yes, they can do that, and have many other functions built in!

Example	AWS CLI query expression	JQ Expression
Sort output numerically by field values*	`sort_by(Items[], &to_number(SomeField.N))[][SortKey.S,SomeField.N]`	`[.Items[] \| [.SortKey.S,.ServiceTime.N]] \| sort_by(.[1] \| tonumber)`
Sum fields (e.g. get total page access time )	`sum(map(&to_number(ServiceTime.N), Items[*]))`	`[.Items[].ServiceTime.N \| tonumber] \| add`
Perform counting e.g. sum of pages accessed by Mozilla	`length(Items[?AgentString && starts_with(AgentString.S, 'Mozilla')])`	`[.Items[] \| select(.AgentString.S \| startswith("Mozilla"))] \| length`

* Note the expressions convert the fields to numbers here so as to sort numerically rather than textually

Wrapping Up

This post shows that it is possible to perform some complicated transformations on JSON output data. I think most people could see that the above commands, which you can almost call 1-liners, can replace a whole JavaScript or Python function and allow you to perform complicated ad-hoc and maybe even regular tasks, with much less development overhead.

In fact, because JQ can read your filter expression from a file (which can also contain comments), complex filters can turn into 1-liners, with JQ as your script interpreter. This also means that you can version-control and track your JQ scripts.

The specifications of these tools are quite similar and are available in library form in JavaScript, Python, Go and many other languages.

More Resources

Filtering output from the AWS CLI

This post was adapted from a larger post on my blog. See the full post here