DEV Community: Paul Bradley

Using the Go AWS API to get AWS CloudWatch Metrics for an S3 bucket.

Paul Bradley — Sat, 27 May 2023 09:14:25 +0000

Earlier this year, one of the largest web applications I've ever developed went live. As part of supporting the organisations using the application I'd built an operations dashboard that lets me see all the stats and logs from the application in one place.

The application uses a private S3 bucket to store cached files. The S3 bucket has a life cycle policy to delete files twenty hours after being created. I wanted the dashboard to show the count of items in this bucket and the total size of the bucket to keep track of consumption.

This article explains how I used the Go API (Application Programming Interface) to extract the metrics from CloudWatch

Amazon CloudWatch metrics for Amazon S3 can help you understand and improve the performance of applications that use Amazon S3. There are several ways that you can use CloudWatch with Amazon S3.

Amazon CloudWatch Daily Storage Metrics for Buckets

Monitor bucket storage using CloudWatch, which collects and processes storage data from Amazon S3 into readable, daily metrics. These storage metrics for Amazon S3 are reported once per day and are provided to all customers at no additional cost.

Amazon S3 CloudWatch Daily Storage Metrics for Buckets

The CloudWatch AWS/S3 namespace includes the following daily storage metrics for buckets.

BucketSizeBytes

The amount of data in bytes stored in a bucket in the STANDARD storage class, INTELLIGENT_TIERING storage class, Standard - Infrequent Access (STANDARD_IA) storage class, OneZone - Infrequent Access (ONEZONE_IA), Reduced Redundancy Storage (RRS) class, Deep Archive Storage (S3 Glacier Deep Archive) class or, Glacier (GLACIER) storage class.

This value is calculated by summing the size of all objects in the bucket (both current and non-current objects), including the size of all parts for all incomplete multipart uploads to the bucket.

Valid storage type filters: StandardStorage, IntelligentTieringFAStorage, IntelligentTieringIAStorage, IntelligentTieringAAStorage, IntelligentTieringDAAStorage, StandardIAStorage, StandardIASizeOverhead, StandardIAObjectOverhead, OneZoneIAStorage, OneZoneIASizeOverhead, ReducedRedundancyStorage, GlacierStorage, GlacierStagingStorage, GlacierObjectOverhead, GlacierS3ObjectOverhead, DeepArchiveStorage, DeepArchiveObjectOverhead, DeepArchiveS3ObjectOverhead and, DeepArchiveStagingStorage (see the StorageType dimension) Units: Bytes Valid statistics: Average

NumberOfObjects

The total number of objects stored in a bucket for all storage classes except for the GLACIER storage class. This value is calculated by counting all objects in the bucket (both current and noncurrent objects) and the total number of parts for all incomplete multipart uploads to the bucket. Valid storage type filters: AllStorageTypes (see the StorageType dimension) Units: Count Valid statistics: Average

Using Go to Obtain S3 Bucket Metrics

The first step is to install the AWS software development kit (SDK) for GoLang. This is done by using the following Go get command issued at the terminal or command prompt.

go get github.com/aws/aws-sdk-go/...

Once the AWS SDK has been installed, you'll need to import the relevant sections into your program to interact with CloudWatch.

package main

import (
    "fmt"
    "log"
    "strconv"
    "strings"
    "time"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/credentials"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/cloudwatch"
)

Within the main part of your program, you need to create an AWS session using the NewSession function. In the example below the newly created session is assigned to the awsSession variable. The session object gets created by supplying the AWS region identifier and your AWS credentials for Access Key and Secret Key. You can obtain these keys from your AWS account. The NewSession function returns a pointer to the session object, and if there was an error, i.e. you specified an invalid region, then an error struct is returned.

You should, therefore, check the status of the error variable and handle it appropriately for your use case. In this example, we log the error to the console as a fatal message. Logfatal equivalent to Print() followed by a call to os.Exit(1); so the program will terminate.

var err error
var awsSession *session.Session

func main() {

    awsSession, err = session.NewSession(&aws.Config{
        Region: aws.String("eu-west-2"),
        Credentials: credentials.NewStaticCredentials(
            os.Getenv("AccessKeyID"),
            os.Getenv("SecretAccessKey"),
            ""),
    })

    if err != nil {
        log.Fatal(err)
    }
}

Once you've got a pointer to a valid AWS session you can reuse that session to make multiple calls against the CloudWatch API.

Obtaining the S3 Bucket Size Metric

The getS3BucketSize function shown below takes one parameter, which is the bucket name for which we want to get the CloudWatch metrics. The function returns three values. The first value is the average data point which contains the bucket size. Secondly, the date/time that CloudWatch logged the value. Finally, the third parameter returns any errors within the function.

We're requesting the metric for S3 objects stored under the StandardStorage tier in the example below. If you want the same function to query different storage types, then you'd need to make this an input parameter to the function.

After getting the result from the GetMetricStatistics function call, we iterate over the result.Datapoints getting the results to return in the return values.

func getS3BucketSize(bucketName string) (float64, time.Time, error) {
    var bucketSize float64
    var metricDateTime time.Time

    svc := cloudwatch.New(awsSession)

    result, err := svc.GetMetricStatistics(&cloudwatch.GetMetricStatisticsInput{
        MetricName: aws.String("BucketSizeBytes"),
        Namespace:  aws.String("AWS/S3"),
        StartTime:  aws.Time(time.Now().Add(-48 * time.Hour)),
        EndTime:    aws.Time(time.Now()),
        Period:     aws.Int64(3600),
        Statistics: []*string{aws.String("Average")},
        Dimensions: []*cloudwatch.Dimension{
            &cloudwatch.Dimension{
                Name:  aws.String("BucketName"),
                Value: aws.String(bucketName),
            },
            &cloudwatch.Dimension{
                Name:  aws.String("StorageType"),
                Value: aws.String("StandardStorage"),
            },
        },
    })

    if err != nil {
        return 0, time.Now(), err
    }

    for _, dataPoint := range result.Datapoints {
        bucketSize = *dataPoint.Average
        metricDateTime = *dataPoint.Timestamp
    }

    return bucketSize, metricDateTime, err
}

Obtaining the S3 Bucket Count Metric

To get the count of files/objects stored in the S3 bucket, you'd need to change the MetricName from BucketSizeBytes to NumberOfObjects and change the storage type to AllStorageTypes as shown in the snippet below.

    MetricName: aws.String("NumberOfObjects"),
    Namespace:  aws.String("AWS/S3"),
    StartTime:  aws.Time(time.Now().Add(-48 * time.Hour)),
    EndTime:    aws.Time(time.Now()),
    Period:     aws.Int64(3600),
    Statistics: []*string{aws.String("Average")},

    Dimensions: []*cloudwatch.Dimension{
        &cloudwatch.Dimension{
            Name:  aws.String("BucketName"),
            Value: aws.String(bucketName),
        },
        &cloudwatch.Dimension{
            Name:  aws.String("StorageType"),
            Value: aws.String("AllStorageTypes"),
        },
    },

Conclusion

I'm using these functions within a Linux command-line utility which gets invoked from a cron job daily. The results from the last thirty days are stored in a database so that I can present the values on my dashboard as a Sparkline chart. A Sparkline is a small line chart, typically drawn without axes or coordinates. It presents the general shape of the variation and lets me quickly see the growth trends.

Improving web application performance by using session storage with JavaScript

Paul Bradley — Wed, 14 Sep 2022 10:21:30 +0000

Introduction

In my day job, I work on a web application that consolidates clinical information from multiple sources. One of those sources is a Health Information Exchange. Depending on the patient, it can take several seconds to return the data for a patient with a complex medical history.

As such, the data is returned via an AJAX request so that I can present the user with a progress bar while the health information exchange responds. This article demonstrates how I use sessionStorage to cache the results and make the web application feel more responsive to the user.

The Web Storage API provides mechanisms by which browsers can store key/value pairs in a much more intuitive fashion than using cookies. The Web Storage API provides two mechanisms for caching data, localStorage and sessionStorage.

The localStorage API persists data even when the browser is closed and reopened. It stores data with no expiration date and gets cleared only through JavaScript or clearing the browser cache. As my web application holds clinical data, I didn't want to cache that data between browsing sessions.

As such, I choose sessionStorage as it maintains a separate storage area for each given origin. It stores data only for a session, meaning that the information is only stored until the browser (or tab) is closed. The data is never transferred to the server. It also has a much larger storage limit than a cookie, allowing up to 5MB of data to be stored.

sessionStorage.getItem

First, we need to use getItem function to see if anything is already stored in the cache. The getItem function has one parameter, the key name,
a string containing the name of the key you want to retrieve.

If the cache is empty, we perform the AJAX request by checking if it equals null. If the cache isn't empty, we load the HTML code from the cache by using the getItem function and assign it the HTML element with an id attribute of LloyedGeorge.

Lloyed George was the person known for inventing the paper patient records and as this is a medical application the use of his name as a variable is a hat tip to the man. The placeholder .Patient.LloyedGeorge in the sample below is replaced with a unique value for each patient. This ensures that if a user is viewing multiple patient records, the cached data for another patient isn't retrieved and displayed incorrectly.

It's a good idea to use a unique key name that has a meaning within your application.

if (sessionStorage.getItem(".Patient.LloyedGeorge") === null) {
    xds_ajax_request();
} else {
    document.getElementById("LloyedGeorge").innerHTML = sessionStorage.getItem(".Patient.LloyedGeorge");
}

sessionStorage.setItem

When our AJAX call responds with a valid result, we assign responseText to the HTML element with an id attribute of LloyedGeorge.

We also save a copy of the responseText into the session storage using the setItem function call. The setItem function accepts two parameters. The first is a key name, a string containing the name of the key you want to create/update. The second is the key value, a string containing the value you want to give the key you are creating or updating.

Once we've saved the AJAX response into the session storage, we can refresh the web page, and the AJAX call won't be triggered, as the code example above will load the data directly from the session storage. Again, this makes your application feel fast, as the data will appear immediately.

if (xmlHttp.readyState == 4 && xmlHttp.status == 200) {
    document.getElementById("LloyedGeorge").innerHTML = xmlHttp.responseText;
    sessionStorage.setItem(".Patient.LloyedGeorge", xmlHttp.responseText);
}

sessionStorage.removeItem

If you decide to use the sessionStorage API's, you need a way for your users to invalidate the cache, so your application can re-query the AJAX endpoint/service. On a mobile application, this might be a pull to refresh gesture, or it could be a button the user needs to press. Either way, the refresh event should call the removeItem function to invalidate the cache.

The removeItem function has one parameter: the key name for the key you want to remove.

function force_refresh() {
    sessionStorage.removeItem('.Patient.LloyedGeorge');
}

Need Help or Advice

If you need help or advice on improving your web application performance, don't hesitate to get in touch with me via Twitter. Send me a direct message to @nhsdeveloper.

Provisioning an AWS PostgreSQL Aurora RDS Cluster with Babelfish using Terraform

Paul Bradley — Mon, 05 Sep 2022 07:07:34 +0000

Introduction to Babelfish

Migrating an SQL Server database to Amazon Web Services, while possible, can be expensive. Microsoft's SQL Server licensing model is expensive when compared to an open source alternative like PostgreSQL. Thanks to an open-source project called Babelfish for PostgreSQL you can move your SQL Server workloads to the cloud and benefit from the cost efficiencies that PostgreSQL offer.

Babelfish for PostgreSQL adds a Microsoft SQL Server-compatible end-point to PostgreSQL. Babelfish allows PostgreSQL to understand T-SQL, SQL Server's proprietary SQL dialect, and supports the TDS communication protocol, so applications originally written for SQL Server may work with PostgreSQL with fewer code changes and without changing database drivers.

Software suppliers will need to test their software using a Babelfish endpoint, but most applications will work with minor changes.

Provisioning an Aurora PostgreSQL Cluster with Babelfish Enabled Using Terraform

Below is a sample Terraform script to provision an RDS Aurora Cluster with a Babelfish endpoint enabled.
I hope you find this useful.
The code is commented, and you can change elements to suit your requirements.

resource "aws_rds_cluster" "rds_cluster_maywoods_audit_tool" {

    cluster_identifier = "maywoods-audit-tool"
    availability_zones = ["eu-west-2a", "eu-west-2b", "eu-west-2c"]

    // database engine type, mode and version
    engine          = "aurora-postgresql"
    engine_mode     = "provisioned"
    engine_version  = "13.7"

    // database_nam must begin with a letter and 
    // contain only alphanumeric characters.
    // ensure a strong long password is used.
    database_name   = "MaywoodsAuditToolsQA"
    master_password = "**********************************"
    master_username = "MaywoodsAuditAdmin"

    backup_retention_period = "60"
    copy_tags_to_snapshot = true

    db_cluster_parameter_group_name = aws_rds_cluster_parameter_group.rds_cluster_maywoods_audit_tool_pg.name
    db_subnet_group_name = aws_db_subnet_group.rds_cluster_maywoods_audit_tool_sng.name
    deletion_protection = true
    enabled_cloudwatch_logs_exports = ["postgresql"]

    skip_final_snapshot = flase
    final_snapshot_identifier = "maywoods-audit-tool-cluster-final-snapshot"

    iam_database_authentication_enabled = false
    iam_roles = []

    preferred_backup_window = "23:00-00:00"
    preferred_maintenance_window = "sun:01:00-sun:02:00"

    storage_encrypted = true
    kms_key_id = "aws/rds"

    vpc_security_group_ids = var.sg_id_list

    tags = {
        // override any tags already set within the
        // default_tags block with the providers.tf
        Application = "CORE SERVICE - AUDIT TOOL DATABASE",
        Provider = "MAYWOODS"
    }
}

resource "aws_rds_cluster_instance" "rds_cluster_maywoods_audit_tool_instance" {

    cluster_identifier = aws_rds_cluster.rds_cluster_maywoods_audit_tool.id

    // change instance type/size and count
    // based on the workloads requirements
    instance_class = "db.r5.large"
    count = 1
    identifier = "maywoods-audit-tool-inst-0"

    // use the same engine/version as define in the cluster
    engine = aws_rds_cluster.rds_cluster_maywoods_audit_tool.engine
    engine_version = aws_rds_cluster.rds_cluster_maywoods_audit_tool.engine_version

    auto_minor_version_upgrade = true
    copy_tags_to_snapshot = true
    db_parameter_group_name = "maywoods-audit-tool-pg"
    db_subnet_group_name = aws_db_subnet_group.rds_cluster_maywoods_audit_tool_sng.name
    monitoring_interval = "0"
    monitoring_role_arn = ""
    performance_insights_enabled = true
    preferred_maintenance_window = "sun:01:00-sun:02:00"
    publicly_accessible = false

    tags = {
        // override any tags already set within the
        // default_tags block with the providers.tf
        Application = "CORE SERVICE - AUDIT TOOL DATABASE",
        Provider = "MAYWOODS"
    }
}

resource "aws_rds_cluster_parameter_group" "rds_cluster_maywoods_audit_tool_pg" {
    name = "maywoods-audit-tool-pg"
    family = "aurora-postgresql13"

    # enable babelfish to be active
    parameter {
        name = "rds.babelfish_status"
        value = "on"
        apply_method = "pending-reboot"
    }

    tags = {
        // override any tags already set within the
        // default_tags block with the providers.tf
        Application = "CORE SERVICE - AUDIT TOOL DATABASE",
        Provider = "MAYWOODS"
    }
}

resource "aws_db_subnet_group" "rds_cluster_maywoods_audit_tool_sng" {
    name = "${var.name_prefix}maywoods-db-subnet"
    subnet_ids = [var.private_subnet_zero, var.private_subnet_one]

    tags = {
        Application = "ScR CORE SERVICE - AUDIT TOOL DB",
        Provider = "MAYWOODS"
    }
}

resource "aws_cloudwatch_log_group" "rds_cluster_maywoods_audit_tool_lg" {
    name = "/aws/rds/cluster/${aws_rds_cluster.rds_cluster_maywoods_audit_tool.cluster_identifier}/postgresql"
    retention_in_days = 60
}

Security Groups

To allow servers to connect to the RDS Aurora PostgreSQL Cluster you'll also need to define a security group.
The security group will need to open ports 5432 for native PostgreSQL traffic, and port 1433 for Babelfish traffic.

resource "aws_security_group" "rds_maywoods_audit_tool_sg" {
    name = "${var.name_prefix}rds_maywoods_audit_tool_sg"
    description = "Maywoods Audit Tool Database Access"
    vpc_id = var.vpc_id

    tags = {
        Name = "${var.name_prefix}rds_maywoods_audit_tool_sg",
    }
}

resource "aws_security_group_rule" "rds_maywoods_audit_tool_psql_inbound" {
    security_group_id = aws_security_group.rds_maywoods_audit_tool_sg.id
    description = "Ingress PostgreSQL traffic from hospital servers"
    type = "ingress"
    from_port = 5432
    to_port   = 5432
    protocol  = "tcp"
    cidr_blocks = ["10.164.33.38/32", "10.164.33.37/32"]
}

resource "aws_security_group_rule" "rds_maywoods_audit_tool_psql_outbound" {
    security_group_id = aws_security_group.rds_maywoods_audit_tool_sg.id
    description = "Egress PostgreSQL traffic from hospital servers"
    type = "egress"
    from_port = 5432
    to_port   = 5432
    protocol  = "tcp"
    cidr_blocks = ["10.164.33.38/32", "10.164.33.37/32"]
}

resource "aws_security_group_rule" "rds_maywoods_audit_tool_babelfish_inbound" {
    security_group_id = aws_security_group.rds_maywoods_audit_tool_sg.id
    description = "Ingress Babelfish traffic from hospital servers"
    type = "ingress"
    from_port = 1433
    to_port   = 1433
    protocol  = "tcp"
    cidr_blocks = ["10.164.33.38/32", "10.164.33.37/32"]
}

resource "aws_security_group_rule" "rds_maywoods_audit_tool_babelfish_outbound" {
    security_group_id = aws_security_group.rds_maywoods_audit_tool_sg.id
    description = "Egress Babelfish traffic from hospital servers"
    type = "egress"
    from_port = 1433
    to_port   = 1433
    protocol  = "tcp"
    cidr_blocks = ["10.164.33.38/32", "10.164.33.37/32"]
}

Using SQLite to generate documentation and software configurations

Paul Bradley — Thu, 01 Sep 2022 07:54:41 +0000

Using SQLite to generate documentation and software configurations

Why would you want to use SQLite to maintain documentation? Or use it to manage software configuration files?

In this article I explain the benefits and demonstrate how to use SQLite to generate documentation.

I maintain several documents for a large health project. The documents display the same data in different ways depending on the intended audience of the document.

The first is a list of publishing standards, and the second is a role-based access control matrix. The data set grows and evolves as the project responds to new use cases. Having an automated process for generating the documents from the data is crucial.

Rather than attempting to maintain the two documents separately and potentially getting them out of synchronisation, I decided to use SQLite to hold a central copy of the data and use SQL (Structured Query Language) queries to produce the documentation formatting codes required.

SQLite uses a single terminal command to access all its features. This command, sqlite3, makes it easy to script the steps in your document creation process. The ability to chain SQLite commands together makes the process flexible and repeatable. Also, the output from these commands can be piped into external files. Most documentation build process can include these external files ensuring the latest data is presented within the document.

Leveraging the power of the SQL syntax, you can also control the ordering of the data and the splitting of the large results across multiple pages.

Also, as the whole process is scripted, in my case using make, the commands that generate the documentation can be managed using source code control. Using a source code version system, like Git GNU Make, allows you to track the history of changes made to your documentation data.

Document Types

The data set I maintain is a list of medical document types used in a regional Health Information Exchange.
The data set contains three fields: document name, clinical code, and schema. I maintain the list in a simple comma-separated (CSV) data file, like:

Discharge Letter,823701000000103,SNOMED-CT
Drug/substance use,1064501000000103,SNOMED-CT
End of Life Care Document,861411000000103,SNOMED-CT

Importing the data into SQLite

The CSV file is my master data source. If I need to change an entry, I change it within the CSV file and then re-process the data. If I add a new document type to the health exchange, I will add the entry to the CSV file and then run make load to load the data into SQLite for further processing.

I use GNU Make to run the commands I need.
It's not necessary to use make. If your more familiar with using bash scripts, or on Windows batch files, then use those. The make system, for me, is an easy way to group a collection of commands to together.

The make system reads a makefile within your project folder. The makefile contains the commands you want to run in the order you want to run them. The makefile can have multiple targets; denoted by a keyword followed by a colon character. When you run the make command followed by a target name, just the commands within that section will be executed.

I use two targets within my makefile, making it easy for me to load the data and produce the documentation docs. The code below shows the SQLite commands I use to import the CSV file into a database table.

load:
    sqlite3 -cmd ".read create_tables.sql"
            -cmd ".mode csv"
            -cmd ".import document_types.csv document_types"
            document_types.db < count.sql

The -cmd parameter of the sqlite3 command-line utility allows you to specify which commands need executing. We need to use multiple -cmd sequences to execute the commands we need in order. In the example above, we start by reading and running the SQL statements within the create_tables.sql file, which drops and re-creates the document types table within the database file document_types.db.

We then switch the mode of operation to CSV before importing the CSV file. The .import command has two parameters. The first parameter is the name of the CSV file containing your data. The second parameter instructs SQLite as to which table the data should be imported into. Finally, we read and execute the count.sql file, which counts the number of records in the table and prints it to the terminal screen.

The create table SQL

The SQL code below shows the contents of the create_tables.sql file. It starts by dropping the table document_types to clear out the old data; and then re-creates the table by supplying the column definitions.

DROP TABLE IF EXISTS document_types;

CREATE TABLE document_types (
    "name"      TEXT,
    "code"      TEXT,
    "scheme"    TEXT
);

The contents of the file count.sql is shown below.

SELECT COUNT(*) || ' : Records Imported'
  FROM document_types;

Now we have our data in an SQLite database file; we can start by producing the documentation code.

Creating documentation code

It shouldn't matter which documentation system you use; they all generally support including files. LaTex, Asciidocter and DocBook all support including files. The key is getting SQLite to produce output in the format that your documentation system requires.

Personally, I use LaTeX and the code to display the data within a table is shown below:

\section*{Document Types}
\begin{center}
    \begin{tabular}{p{8cm} p{3cm} p{2.7cm} }
        \bfseries{Value} & \bfseries{Node} & \bfseries{Schema} \\
        \toprule
            \input{doc-types-table.tex}
        \bottomrule
    \end{tabular}
\end{center}

The line of interest in the above code snippet is the line that says input{doc-types-table.tex} This line instructs the LaTeX system to import the commands contained within a file called doc-types-table.tex.
We need to get SQLite to generate that file with the data and the correct formatting codes to render it as part of the table.

The LaTeX format for the table row is shown below.
The ampersand character separates each column within the table. The end of each row is depicted by the two backslash characters. The \midrule command produces a thin line between the table rows.

Discharge Letter & 823701000000103 & SNOMED-CT \\ \midrule

Therefore, we need an SQLite SQL command to output the data in the above format. The SQL I use is shown below. Using the SQLite construct || to concatenate (join) the data fields together, separated by ampersand characters. The SQL statement also sorts the data into name order for me limits the number of records returned to 30, as only 30 entries will fit on a given page.

SELECT name || ' & ' || code || ' & ' || scheme || ' \\ \midrule ' AS LaTeX
  FROM document_types
 ORDER BY name
 LIMIT 0, 30;

If the above SQL is saved into a file called publishing_standards_page.sql then we can update our makefile to include a docs: target so we can issue the command *make docs" every time we want to re-generate the documentation.

The SQLite command is slightly different as we switch to list mode and redirect the output to the file doc-types-table.tex within our LaTeX project folder.

docs:
    sqlite3 -cmd ".mode list"
        document_types.db
        < publishing_standards_page.sql
        > /path/to/latex/doc-types-table.tex

Once you get your head around the idea of your database generating your documentation, you can automate the creation of many different documents for different audiences from the same data set.
For example, we also have a request for a new role document, which users must complete before we define new roles. This document is a PDF that again lists the document types but needs to generate tick boxes so people can express which documents the new role should have access to view.

LaTeX is pretty good as it can create forms with a PDF. The SQL below is saved into a file called roles_request_form.sql and generates the LaTeX code required to produce a PDF form. Each checkbox within a form needs to have a unique name. Therefore, I've used the SQLite inbuilt keyword ROWID, which returns each record's unique internal row number.

SELECT name || ' & \CheckBox[name=' || ROWID ||
    ', width=1em, bordercolor={lightgray}]{Yes} \\ \midrule ' AS LaTeX
  FROM document_types
 ORDER BY name
 LIMIT 0, 30;

The updated makefile is shown below.

docs:
    sqlite3 -cmd ".mode list"
        document_types.db
        < publishing_standards_page.sql
        > /path/to/latex/doc-types-table.tex
    sqlite3 -cmd ".mode list"
        document_types.db
        < roles_request_form.sql
        > /path/to/latex/doc-roles-form.tex

Creating software configurations

The same process can also be used to generate your software configuration files. The health exchange uses XML files to apply restrictive polices to groups of users. For example, a clinical user can view all document types except two specific documents. The database is used to generate the XML configuration files the software needs to configure these polices.

SELECT
  '<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">' ||
  code  || '</AttributeValue>' AS ClinicalRole
 FROM document_types
WHERE code IN ('149741000000107', '723394009')
GROUP BY code
ORDER BY code;

The example SQL above extracts the document clinical codes for the two documents clinical staff aren't allowed to view.

config:
    sqlite3 -cmd ".mode list"
        document_types.db
        < clinical_role_restrictions.sql
        > /path/to/software/configs/roles.xml

Amazon Rekognition using the Go AWS API

Paul Bradley — Sat, 16 Jan 2021 12:03:38 +0000

Amazon Rekognition using the Go AWS API

Amazon Rekognition is a highly scalable, deep learning technology that let's you identify objects, people, and text within images and videos. It also provides highly accurate facial analysis and facial search capabilities. In this short article we'll explore two use cases with the Go API. The first will be to OCR (extract text) from and image and the second will be to identify the contents of an image.

The first step is to install the AWS software development kit (SDK) for Go. This is done by using the following command issued at the terminal or command prompt.

go get github.com/aws/aws-sdk-go/...

Once the AWS SDK has been installed, you'll then need to import the relevant sections into your program to be able to interact with the Rekognition service.

package main

import (
    "bufio"
    "fmt"
    "io/ioutil"
    "log"
    "os"
    "strings"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/credentials"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/rekognition"
)

The first step within our interaction with any AWS service is to initialise a session; by passing in your AWS access key and AWS secret key. For security reasons these aren't included within our program but obtained from environment variables. After calling the NewSession function we need to check the error state. If an error has occurred then we terminate the program and log the error message to the terminal.

    s, err := session.NewSession(&aws.Config{
        Region: aws.String("eu-west-2"),
        Credentials: credentials.NewStaticCredentials(
            os.Getenv("AccessKeyID"),
            os.Getenv("SecretAccessKey"),
            ""),
    })

    if err != nil {
        log.Fatal(err.Error())
    }

Optical Character Recognition

The image below is the later flow packaging for a Covid19 test. Let's assume we want to perform analysis of this image and extract the text into a machine readable format so that we can use the data within our application/database. This would be a good test for the API as the packaging is quite shiny and thus reflects the light in different ways.

We initialise the rekognition API using the DetectTextInput option. We need to pass in the raw bytes for the image we want to process. This can either be a reference to an S3 object (ideal for running this type of analysis as a Lambda function) or like in this example an image loaded from the local file system.

    svc := rekognition.New(s)
    input := &rekognition.DetectTextInput{
        Image: &rekognition.Image{Bytes: loadLocalFile("lfa.jpg")},
    }

The function to load the image from local disk is shown below.

func loadLocalFile(filename string) []byte {
    var err error
    var f *os.File
    var reader *bufio.Reader
    var data []byte

    f, err = os.Open(filename)
    if err != nil {
        log.Fatal(err.Error()
    }

    reader = bufio.NewReader(f)
    data, err = ioutil.ReadAll(reader)
    if err != nil {
        log.Fatal(err.Error())
    }

    return data
}

We can then run the DetectText function and capture the results. Once we have the results we can then loop over the entires and print out the text that has been detected within the image.

    result, err := svc.DetectText(input)
    if err != nil {
        log.Fatal(err.Error())
    }

    for _, entity := range result.TextDetections {
        if *entity.Type == "LINE" {
            fmt.Println(strings.TrimSpace(*entity.DetectedText))
        }
    }

Which produces the following output. As you can see, its done a very good job of detecting the text within the image.

INNOVA
1 Test
Antigen Test Cartridge
LOT X2010008
2020. 10. 30
2022. 10. 29
CE
INNOVA

As with all AWS Rekognition results a confidence score is returned, which you can use to decide if you want to trust a particular result. You can also return the bounding box coordinates for the detected text by accessing the Geometry values. I plan to do a follow up article which demonstrates how to use these values to draw highlighting boxes back onto the original image.

    for _, entity := range result.TextDetections {
        if *entity.Type == "LINE" {
            fmt.Println(strings.TrimSpace(*entity.DetectedText))
            fmt.Println(*entity.Confidence)
            fmt.Println(*entity.Geometry.Polygon[0].X)
            fmt.Println(*entity.Geometry.Polygon[0].Y)
        }
    }

Image Identification

Let's try using the image identification service to see what it makes of this image:

So preparing the input is slightly different. This time we need to use the DetectLabelsInput function which again let's us load the image content from either an S3 bucket or a local file. However, the DetectLabelsInput function has two additional parameters to specify the maximum number of identifications (labels) and the minium confidence score to use.

svc := rekognition.New(s)

input := &rekognition.DetectLabelsInput{
    Image:         &rekognition.Image{Bytes: loadLocalFile("input.jpg")},
    MaxLabels:     aws.Int64(123),
    MinConfidence: aws.Float64(70.000000),
}

result, err := svc.DetectLabels(input)
for _, detectedLabel := range result.Labels {
    fmt.Println(*detectedLabel.Name)
}

Once we've defined the input, then we can call the DetectLabels function and loop over the results printing out the label names. For the above image the following identifications were returned:

Bird
Animal
Jay, Finch
Blue Jay

Costs

At $1.16 for every 1,000 images it would cost around $60 to categorise 50,000 images. So I'm going to be looking at categorising my entire photo albums so that I can search by the generated labels.

Need Help or Advice

If you need any help or advice in using AWS Rekognition with Go then please get in touch I'd be happy to help.

Terminal Image Manipulation

Paul Bradley — Mon, 04 Jan 2021 09:38:21 +0000

As a web developer there are many instances in which you need to interact with images. Maybe you need to rotate or resize an image before posting it to your web site. In most cases, you'll not need a fully blown image editor like Gimp installed on your machine. With ImageMagick and other command line tools, such as jpegtran, you can accomplish a lot from the terminal. Here is a list of the commands I use on a regular basis.

Get dimensions of an image

identify input.jpg

Which will return the dimensions like:

input.jpg JPEG 3264x2448 3264x2448+0+0 8-bit sRGB

Rotate an image

You can rotate an image through 90 or 180 degrees by using the jpegtran command:

jpegtran -rotate 90 input.jpg > output.jpg

Resize an image maintaining aspect ratio

You can use the convert command from the ImageMagic library to resize an image to a specific width while maintaining the aspect ration for the images height. In this example, we resize the input image to a maximum width of 700 pixels. We also reduce the quality of the JPG compression to 80%.

convert -geometry 700x input.jpg -quality 80 output.jpg

Prepare an image for a website

To prepare a JPG image which is as small as possible for website performance we should remove all the EXIF data and remove the embedded thumbnail that many cameras generate. The command below does this, while also converting the JPG to use a progressive encoding. The progressive encoding can make the image appear to load faster over slow internet connections, as it renders the image in stages to make it appear it's being streamed in.

jpegtran -copy none -progressive -optimize input.jpg > output.jpg

Remove colour from an image

To convert a JPG image to a greyscale black and white photo use the following command:

jpegtran -greyscale input.jpg > grey.jpg

Upgrading Ubuntu Server from 19.04 to 20.04 (Focal Fossa)

Paul Bradley — Wed, 29 Apr 2020 06:39:31 +0000

Preparation

Before upgrading your Ubuntu server take a backup or snapshot of your existing server so that you can restore it to a working server if the upgrade fails at any point. Most hosting providers offer a facility via their control panel to make a snap shot backup. It's probably best practice to take this snap shot while the server is powered down.

Upgrading to Ubuntu 20.04 LTS (Focal Fossa) is a good idea as it's a long term support release which means you'll get security updates for the next five years. Focal Fossa comes with a lot of new packages and major software upgrades, including the latest versions of Java, Python, Ruby, PHP and many more. This release is based on the Linux 5.4 LTS kernel and adds support for new hardware and file systems.

Check for held packages & update

Start by checking if you have any held packages by using the the showhold command. If you do have any then mark them as un-held by issuing an unhold command followed by the package names listed in the previous command.

sudo apt-mark showhold
sudo apt-mark unhold package_name

Once you have un-held any packages, perform a standard update and upgrade and then reboot the server.

sudo apt update
sudo apt upgrade
sudo systemctl reboot

Purge & remove unwanted packages

After the server restarts then perform a full upgrade and then purge any unwanted or unused packages to recover some disk space.

sudo apt full-upgrade
sudo apt --purge autoremove

Upgrade to 20.04

At this point you're ready to upgrade from 19.04 to 20.04 by performing the release upgrade commands. As the upgrade progresses you'll be prompted to answer various questions in relation to your setup. These questions will vary depending on what software you've already installed and configured on your Ubuntu server.

sudo apt install update-manager-core
sudo do-release-upgrade -d

Verifying the version number after the upgrade

After the upgrade has completed and the server has rebooted you can check the current version of your Ubuntu server by checking the release version.

lsb_release -a

Which should output something like.

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:    20.04
Codename:   focal

Using Google Chrome's Socks5 Proxy With SSH Tunnels

Paul Bradley — Mon, 04 Nov 2019 08:06:38 +0000

Using Google Chrome's Socks5 Proxy With SSH Tunnels

At work we're migrating our core infrastructure to Amazon's AWS service. We have a pretty standard set-up with an infrastructure account hosting our bastion servers, allowing suppliers to connect to our different environments. Currently we have three Virtual Private Clouds (VPC) - production, pre-production and development.

On premise access to these environments is provided via a Direct Connect pipe provided by our WAN supplier.

   INTERNET
      |
+-------------+       +----------------------+
|     |       |       |                      |
|  +--v-----+ |       | +--------+ +-------+ |
|  | Bastion| |       | | App    | |DB     | |
|  | Server |-----------> Server | |Server | |
|  +--------+ |       | |        | |       | |
|             |       | +--------+ +-------+ |
+-------------+       +----------------------+

INFRASTRUCTURE                     PRODUCTION
VPC                                       VPC

Accessing the web interfaces of the applications running on our application servers is fine while working on premise. However, when I work from home I need to create an SSH tunnel through the bastion server and then use Chromes socks5 proxy to direct requests through the SSH tunnel.

Creating the SSH tunnel

SSH into your bastion server using the -D option to bind an addressing port from the local machine to the connection. I tend to use port 1080

ssh ec2-user@bastion -D1080

Launching Chrome to use the binding
Launch Chrome and pass in the --proxy-server option to use the socks5 proxy on your localhost. All traffic/requests will be directed to the port number you used in the SSH connection.

"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
    --user-data-dir="$HOMEproxy-profile"
    --proxy-server="socks5://localhost:1080"

Using this Chrome session you should be able to connect to the web interfaces or applications running on servers within your VPC's

Also posted @ https://paulbradley.org/chrome-socks5/

Building a digital wingman to assist busy clinical staff

Paul Bradley — Mon, 08 Apr 2019 14:20:44 +0000

Building a digital wingman to assist busy clinical staff

I've been working on creating a "digital wingman" to assist busy clinical staff, by detecting if particular drugs have been missed off a discharge letter.

Sorry, the video doesn't have any voice over so this is what you're seeing:

An unstructured discharge letter is sent to AWS's Medical Comprehend AI
A list of medications, dosage and route information is detected within the text, along with forty items of interest. These include medical procedurals and diagnosis.
A visual model of the medical terms and their relationship with each other is then produced and displayed.
We then remove the drug that treats hypertension (high blood pressure) and rerun the program. As hypertension is mentioned within the clinical narrative the system then checks all possible drugs that could be prescribed to treat hypertension.
Not finding one, it then sends an alert to the caregiver for investigation.

AWS medical comprehend

The medical comprehend API is a recent offering from AWS which accepts unstructured medical text and returns a JSON object outlining all the medical, procedural and diagnosis terms found within the text. For medications, it also returns the dose, route and frequency if found within the text. For each medical term found a confidence score is returned to help you decided if you should trust the value. An offset position is also returned indicating the starting point for the medical term found.

Example using Go

The first step is to install the AWS software development kit (SDK) for Go. This is done by using the following Go get command issued at the terminal or command prompt.

go get github.com/aws/aws-sdk-go/...

Once the AWS SDK has been installed, you’ll then need to import the relevant sections into your program to be able to interact with the medical comprehend API.

package main

import (
    "bytes"
    "fmt"
    "log"
    "net/http"
    "os"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/credentials"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/comprehendmedical"
)

You’ll need to create an AWS session select a region which supports the medical comprehend API (not all do). In the example below the newly created session is assigned to the s variable. The session object is created by passing in the region identifier and your AWS credentials for id and secret key. Your id and secret key can be obtained from your AWS account.

func main() {

    // create an AWS session which can be
    // reused if we're uploading many files
    s, err := session.NewSession(&aws.Config{
        Region: aws.String("us-east-1"),
        Credentials: credentials.NewStaticCredentials(
            "XXX",
            "YYY",
            ""),
    })

    if err != nil {
        log.Fatal(err)
    }

    client := comprehendmedical.New(s)
    input  := comprehendmedical.DetectEntitiesInput{}

    input.SetText(`
        Pt is 40yo mother, highschool teacher
        HPI   : Sleeping trouble on present dosage of Clonidine.
                Severe Rash  on face and leg, slightly itchy
        Meds  : Vyvanse 50 mgs po at breakfast daily,
                Clonidine 0.2 mgs -- 1 and 1 / 2 tabs po qhs
        HEENT : Boggy inferior turbinates, No oropharyngeal lesion
        Lungs : clear
        Heart : Regular rhythm
        Skin  : Mild erythematous eruption to hairline
    `)

    result, err := client.DetectEntities(&input)

    if err != nil {
        log.Fatal(err)
    }
}

Start by defining a medical comprehend client using the New function passing in your AWS session variable s. Then define an input object and use the SetText function to define the medical text you want to process. The processing of the medical text is done when the DetectEntities function is called and the results are stored in the results variable.

To get the JSON string returned you can use the GoString function as shown below. This is useful if you want to store the JSON in a Postgres JSONB database field to perform search queries across many patients.

fmt.Println(result.GoString())

You can loop over the detected terms and only print out the medications found by using the following pattern:

for _, entity := range result.Entities {
    if *entity.Category == "MEDICATION" {
        fmt.Println(*entity.Text)
        fmt.Println(*entity.Type)
        fmt.Println(*entity.Score)
        fmt.Println("-----------")
    }
}

Which will output the following results:

Clonidine
GENERIC_NAME
0.9948381781578064
-----------
Vyvanse
BRAND_NAME
0.9995348453521729
-----------
Clonidine
GENERIC_NAME
0.997945249080658
-----------

Uploading file(s) to AWS S3 using Go

Paul Bradley — Thu, 10 Jan 2019 11:02:04 +0000

Uploading file(s) to AWS S3 using Go

My current side project is Glacial, a secure cloud-based document storage & viewing solution for long term archiving of medical records & documents. Clinical documents are encrypted locally before leaving the hospital network and are encrypted again using Amazon’s S3 server-side AES256 encryption.

This quick tutorial demonstrates how to upload your file(s) to an AWS S3 bucket using the Go programming language.

The first step is to install the AWS software development kit (SDK) for Go. This is done by using the following go get command issued at the terminal or command prompt.

go get github.com/aws/aws-sdk-go/...

Once the AWS SDK has been installed, you’ll then need to import the relevant sections into your program to be able to interact with S3.

package main

import (
    "bytes"
    "fmt"
    "log"
    "net/http"
    "os"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/credentials"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3"
)

You’ll need to assign a couple of variables. These will hold the AWS region identifier and the S3 bucket name that you wish to upload files to.

const (
    S3_REGION = "eu-west-1"
    S3_BUCKET = "glacial-io"
)

Within the main part of your program, you need to create an AWS session using the NewSession function. In the example below the newly created session is assigned to the s variable. The session object is created by passing in the region identifier and your AWS credentials for id and secret key. Your id and secret key can be obtained from your AWS account. The NewSession function returns a pointer to the session object and if there was an error, i.e. you specified an invalid region, then an error struct is returned.

You should, therefore, check the status of the error variable and handle appropriately for your use case. In this example, we simply log the error to the console as a fatal message.

func main() {

    // create an AWS session which can be
    // reused if we're uploading many files
    s, err := session.NewSession(&aws.Config{
        Region: aws.String(S3_REGION),
        Credentials: credentials.NewStaticCredentials(
            "XXX",
            "YYY",
            ""),
    })

    if err != nil {
        log.Fatal(err)
    }

    err = uploadFileToS3(s, "discharge-letter-787653.pdf")

    if err != nil {
        log.Fatal(err)
    }
}

Once you’ve got a pointer to a valid AWS session you can reuse that session to upload multiple files. In the above example, we’re only using it once as we pass it to the upload function to upload a discharge letter.

The upload function

The upload function takes two parameters, the pointer to the session object and the name of the file to upload. Firstly, we attempt to open the file. If that fails, because the filename is invalid, then we return the error and exit the function. If the file does exist, then we proceed to calculate the size of the file and read the file's bytes into a buffer.

We use the PutObject function of the SDK to upload the file. Here we can specify various S3 object parameters. For example, we specify the bucket name and the access control list (ACL) in this case we’re flagging the file as private. We also specify which S3 storage classification should be used and in this example, we implement server-side encryption.

func uploadFileToS3(s *session.Session, fileName string) error {

    // open the file for use
    file, err := os.Open(fileName)
    if err != nil {
        return err
    }
    defer file.Close()

    // get the file size and read
    // the file content into a buffer
    fileInfo, _ := file.Stat()
    var size = fileInfo.Size()
    buffer := make([]byte, size)
    file.Read(buffer)

    // config settings: this is where you choose the bucket,
    // filename, content-type and storage class of the file
    // you're uploading
    e, s3err := s3.New(s).PutObject(&s3.PutObjectInput{
        Bucket:               aws.String(S3_BUCKET),
        Key:                  aws.String(fileName),
        ACL:                  aws.String("private"),
        Body:                 bytes.NewReader(buffer),
        ContentLength:        aws.Int64(size),
        ContentType:          aws.String(http.DetectContentType(buffer)),
        ContentDisposition:   aws.String("attachment"),
        ServerSideEncryption: aws.String("AES256"),
        StorageClass:         aws.String("INTELLIGENT_TIERING"),
    })

    return s3err
}

Storage Classifications

Besides the standard storage class STANDARD and the infrequent access class STANDARD_IA, there is now a new (as of Nov 2018) storage class called intelligent tiering. This is ideal for use cases where the file access patterns aren’t known and you’re unsure which class to use. The intelligent tiering option attempts to save you money by automatically moving files between standard and infrequent access based on usage patterns.

You can also now upload files directly to the Glacier storage engine using the value GLACIER.

If cost is an issue, and you don’t need the level of redundancy that the other storage classes offer then you can use REDUCED_REDUNDANCY to lower your monthly fees.

Also posted @ https://paulbradley.org/gos3/

Typesetting Musical Scores Using Code

Paul Bradley — Thu, 06 Dec 2018 10:56:43 +0000

An introductory guide to typesetting musical scores using LilyPond. In this guide, we'll learn how to typeset the music for the lullaby “Hush, Little Baby”. Besides adding musical notes we'll also cover adding song lyrics.

Introduction

For the last couple of years, I've been teaching an after-school coding club at several schools in my region. For one term each year we focus on coding music using the excellent programmable synth Sonic Pi.

Music is a fun way to learn the concepts of computer programming as many songs learnt as children use structures which are commonly used in computer programming. Note durations can be defined using variables and songs typically have repeated sections or rhythms which can be easily coded using simple loops and functions.

To create the worksheets which the children used during class I needed to typeset sections of musical scores to demonstrate a key concept or learning objective.

In the photograph below we’re peer programming the Star Wars theme tune :-)

LilyPond

When I started looking for a suitable program to create the musical scores I wanted something that was free and preferably something that would work from the command line. I'm a big fan of the document preparation system LaTeX, so I was looking for something similar for music.

Then I found the perfect fit for my needs, LillyPond. LilyPond is a music engraving program, devoted to producing the highest-quality sheet music possible. It brings the aesthetics of traditionally engraved music to computer printouts.

LilyPond is similar to a programming language, you don't write music by dragging notes from a graphical toolbar and placing them on a dynamically refreshing score; you write music by typing text.

Yes, that's right, you type LilyPond commands into a simple text file and this text is then interpreted by LilyPond producing beautifully engraved sheet music. Once you get a basic understanding of the various LilyPond commands, it's quite easy to produce the music you need.

Getting Started

In this tutorial, we're going create the musical score shown below. Start by grabbing a copy of LilyPond from the projects download page. Choose the appropriate link for your operating system.

Start by creating a text file called mocking.ly in your favourite text editor program. To get started, type in the commands shown below into your text editor.

\version "2.14.2"
\header  {title = "Mocking Bird"}

{
    \time 4/4
    \clef treble

    \relative c' {
        d4 b'8 b8 b4 b4
    }
}

Every LilyPond file should start with a version statement. The version statement is a line which describes which version of LilyPond the source file is intended for. The second command sets up a header with an appropriate title for the score.

The time statement sets the time signature for the piece. In this example, we're using common time or four-four time. Other time signatures could be 3/4 for a waltz or 2/2 for a march.

The clef statement instructs LilyPond to output the treble clef at the start of the score. Other styles are available, such as bass, baritone, tenor and soprano.

The relative c' statement instructs LilyPond to typeset all the notes enclosed within the curly brackets relative to the middle C note on a piano keyboard. The letters and numbers within the curly brackets represent the notes for the first bar of music. That’s one d followed by four b notes.

The numbers after the letters depict the duration of the note to typeset. As we've already set the time signature to 4/4 (or common time) then to depict a crochet we use the number four, as there are four crochets per bar in common time.

So to depict a minim we would follow the letter with a number two. To depict a quaver we use the number eight, and a semi-quaver would, therefore, be represented with a number sixteen. Lilypond is clever enough to join the stems of multiple quavers together.

Generating the score

When you've finished typing in the code then save the text file mocking.ly and shell out to a command line or terminal session depending on the operating system you're using. You then run the LilyPond program against your text file to generate an image which will contain the musical score. The command syntax is:

lilypond --png mocking.ly

GNU LilyPond 2.14.2
Processing `mocking.ly'
Preprocessing graphical objects...
Layout output to `mocking.ps'...
Converting to PNG...
success: Compilation successfully completed

Assuming there are no errors in your mocking.ly file, you should find a new image file in your folder called mocking.png. If you view this file you should have something which looks like:

Yeah!, we've typeset our first musical score. Now we just need to continue with the remaining bars of music. I'd recommend putting each subsequent bar of music on a separate line within your text file as it'll make it easier to keep track of where in the score you're up to.

\version "2.14.2"
\header  {title = "Mocking Bird"}

{
    \time 4/4
    \clef treble

    \relative c' {
        d4 b'8 b8 b4 b4
        b4 a8 a8 a2
        d,8 d8 a'8 a8 a4 a8 b8
        a4 g4 g2
        d4 b'4 b4 b4
        b4 a4 a2
        d,8 d8 a'8 a8 a4 a8 b8
        a4 g4 g2
    }
}

Removing the indent on the first stave

When you re-run the text file through LilyPond you should get a png file which shows two staves of music. Notice how each bar line is automatically numbered for you. This is a great help when you're trying to explain something to children as you can refer to these numbers.

Notice how the first stave is indented slightly. This is the default behaviour of LilyPond, however, it's very easy to change this behaviour by inserting a layout statement to your source file and specifying an indent value of zero. See the code below which has the layout statement added.

\version "2.14.2"
\header  {title  = "Mocking Bird"}
\layout  {indent = 0}

{
    \time 4/4
    \clef treble

    \relative c' {
        d4 b'8 b8 b4 b4
        b4 a8 a8 a2
        d,8 d8 a'8 a8 a4 a8 b8
        a4 g4 g2
        d4 b'4 b4 b4
        b4 a4 a2
        d,8 d8 a'8 a8 a4 a8 b8
        a4 g4 g2
    }
}

Adding lyrics

The next step is to add the lyrics so that they appear under the correct notes. We do this by including an addlyrics statement with an opening and closing curly brackets. Again, everything that is included within the curly brackets is applied to the lyrics command. By default I found the text size to be too big for lyrics, so I use the set statement to reduce the default font size by three points.

When typing out the lyrics it's a good idea to keep each bar to a single line for easier reading. The key to getting the lyrics to line up with the correct notes is to add the correct duration numbers after the words or part of words.

\version "2.14.2"
\header  {title  = "Mocking Bird"}
\layout  {indent = 0}

{
    \time 4/4
    \clef treble

    \relative c' {
        d4 b'8 b8 b4 b4
        b4 a8 a8 a2
        d,8 d8 a'8 a8 a4 a8 b8
        a4 g4 g2
        d4 b'4 b4 b4
        b4 a4 a2
        d,8 d8 a'8 a8 a4 a8 b8
        a4 g4 g2
    }

    \addlyrics {
        \set fontSize = #-3

        Hush4 lit-8 tle8 ba4 by4
        don't4 say8 a8 word,2
        Mam8 my's8 going8 to8 buy4 you8 a8
        mock4 ing4 bird2

        If4 that4 mock4 ing4
        bird4 won't4 sing2
        Mam8 my's8 going8 to8 buy4 you8 a8
        dia mond4 ring2
    }
}

Adding comments

If you re-look at my finished score, you'll notice that above the first stave the first two notes have their value above, for example, D and B. Every time a note appears for the first time in the music I wanted to add its notation above the stave to help those children who weren’t confident at reading music.

To achieve this you can use the markup statement enclosing the text you want to appear above the stave within the curly brackets. The tiny statement sets the font size to a smaller font so that the text doesn't look too big and out of place.

\version "2.14.2"
\header  {title  = "Mocking Bird"}
\layout  {indent = 0}

{
    \time 4/4
    \clef treble

    \relative c' {
        d^\markup{\tiny D} b'8^\markup{\tiny B} b8 b4 b4
        b4 a8^\markup{\tiny A} a8 a2
        d,8 d8 a'8 a8 a4 a8 b8
        a4 g4 g2^\markup{\tiny G}
        d4 b'4 b4 b4
        b4 a4 a2
        d,8 d8 a'8 a8 a4 a8 b8
        a4 g4 g2
    }

    \addlyrics {
        \set fontSize = #-3

        Hush4 lit-8 tle8 ba4 by4
        don't4 say8 a8 word,2
        Mam8 my's8 going8 to8 buy4 you8 a8
        mock4 ing4 bird2

        If4 that4 mock4 ing4
        bird4 won't4 sing2
        Mam8 my's8 going8 to8 buy4 you8 a8
        dia mond4 ring2
    }
}

I hope you found this introduction to Lilypond useful. Please check out the official manuals for LilyPond as there are lots of commands and statements you can use to tweak and configure the output of your musical scores.

If you don't have the need to typeset music at least give Sonic Pi a try, it really is great fun.

DEV Community: Paul Bradley

Using the Go AWS API to get AWS CloudWatch Metrics for an S3 bucket.

Amazon CloudWatch Daily Storage Metrics for Buckets

Amazon S3 CloudWatch Daily Storage Metrics for Buckets

BucketSizeBytes

NumberOfObjects

Using Go to Obtain S3 Bucket Metrics

Obtaining the S3 Bucket Size Metric

Obtaining the S3 Bucket Count Metric

Conclusion

Improving web application performance by using session storage with JavaScript

Introduction

sessionStorage.getItem

sessionStorage.setItem

sessionStorage.removeItem

Need Help or Advice

Provisioning an AWS PostgreSQL Aurora RDS Cluster with Babelfish using Terraform

Introduction to Babelfish

Provisioning an Aurora PostgreSQL Cluster with Babelfish Enabled Using Terraform

Security Groups

Using SQLite to generate documentation and software configurations

Using SQLite to generate documentation and software configurations

Document Types

Importing the data into SQLite

The create table SQL

Creating documentation code

Creating software configurations

Amazon Rekognition using the Go AWS API

Amazon Rekognition using the Go AWS API

Optical Character Recognition

Image Identification

Costs

Need Help or Advice

Terminal Image Manipulation

Get dimensions of an image

Rotate an image

Resize an image maintaining aspect ratio

Prepare an image for a website

Remove colour from an image

Upgrading Ubuntu Server from 19.04 to 20.04 (Focal Fossa)

Preparation

Check for held packages & update

Purge & remove unwanted packages

Upgrade to 20.04

Verifying the version number after the upgrade

Using Google Chrome's Socks5 Proxy With SSH Tunnels

Using Google Chrome's Socks5 Proxy With SSH Tunnels

Creating the SSH tunnel

Building a digital wingman to assist busy clinical staff

Building a digital wingman to assist busy clinical staff

AWS medical comprehend

Example using Go

Categories

Uploading file(s) to AWS S3 using Go

Uploading file(s) to AWS S3 using Go

The upload function

Storage Classifications

Typesetting Musical Scores Using Code

Introduction

LilyPond

Getting Started

Generating the score

Removing the indent on the first stave

Adding lyrics

Adding comments