DEV Community: Pete Wilcock

GitHub Stargazers over time using Google Charts

Pete Wilcock — Sat, 26 Jun 2021 20:31:00 +0000

Announcing… Planetarium! A way to generate a chart of GitHub Stargazers over time using Google Charts.

Tracking the popularity of your GitHub repository over time is something that might appeal to you. ‘Stargazers’ are a public list of GitHub users that have starred (essentially bookmarked) your repository and is a rough indicator of engagement. People either want to use your code, or at least not forget the link to follow up on it another time.

Charting your Stargazers

Whilst it’s not public, the GitHub API exposes the date and time of when each person starred your repository. This then makes it possible to aggregate these stars into a pretty graph which is a cool way to see how your code’s popularity has increased over time.

Whilst ruminating over the surprising popularity of our recent Serverless Static WordPress on AWS module, I was wondering how to get my own graph like others I’d seen elsewhere. The source of these seemed to be this Starcharts repo written in Go. Whilst this was pretty cool, it seemed to have a few limitations and lacked configurability. Plus it’s written in Go, which I’m definitely never going to learn.

My first thought was to get the Stargazing data myself and plot it in a Google sheet. But how? Wrestling with the output of an API to get data into your desired format is always a bit frustrating, but my previous experience with Steampipe, which lets you query a variety of APIs, including AWS and GitHub, in glorious SQL code, made this extremely easy. After writing up a simple query, I was able to get the data I wanted, aggregated over time in a single result. Awesome!

So I put it in my Google sheet, and I generate a graph manually, and that looks fine – but then suddenly I realise that Google Charts is actually a service and I could potentially generate this dynamically. Was it possible?

A mad quest begins

Well what was initially a bit of curious enquiry turned into a delirious 2am mission to not sleep until I could get dynamic GitHub query + chart generation as a single PNG file working. Man was it not easy!

Google Charts is nice enough, but you can only render charts from an HTML page with embedded Javascript with your data encoded within the page. None of this is dynamic and at first glance this doesn’t seem particularly easy to automate. It is possible to get your data imported using an external data source, but it needs to be available as a queriable URL. That means I’d have to build my own API to return some data. It’s possible, but I just want to plug in some basic CSV data here!

So down another rabbit hole I go… Steampipe to query GitHub and deposit a nicely-formatted CSV file into my local workspace. I then use jquery.csv.js to load the CSV file into Google Charts with Javascript. But… you can’t do this using a file:// path in a local browser due to CORS security controls, so the HTML has to be loaded from a real webserver….

I want this to be a quick end-to-end file generation process, so I create an extremely minimal not-even webserver in Python that runs just long enough to process 3 requests before shutting itself down. But loading the page in the browser just renders the image as an Image Data URI… this means getting the image I want is not as simple as just curling a URL, because there is no URL!

Various server-side methods of loading the page won’t work, because they critically need a Javascript interpreter, just like a real browser…. hmmm. What about headless Chrome and Selenium? I don’t want to install any of this locally, so I’ll need that in a Docker container please – luckily one exists.

So after a couple of hours of punching some Python code and working out the worst of the bugs, I finally compress it all into a single shell script that’ll grab the latest stargazer source data, then run a docker container with selenium so I can use a Python library to load the page and automatically save the image before depositing it back into the local working directory.

Conclusion

It might be a terrible solution because I’ve missed a much more obvious, easier way of doing this, but in any case I remain amazed at how you can set your sights on a silly, pointless goal and then spend a few hours grinding away in code and many, many searches to finally get it working in the way you imaged.

My next iteration of this will be to bundle it into a Lambda function and have it run on a schedule, to update an image dynamically loaded into the README of my desired GitHub repo! Watch this space for that.

Full instructions and more details are available on the Planetarium GitHub page. I hope you find it useful!

--
Pete Wilcock is a 9x AWS Certified DevOps Consultant, AWS Community Builder, and Technical Writer for TechToSpeech. If I’m not possibly losing my mind and definitely my social life buried in some project, you can find me on LinkedIn, Twitter, GitHub, or my personal site.

Serverless Static Wordpress on AWS for $0.01 a day

Pete Wilcock — Sat, 19 Jun 2021 23:21:14 +0000

You’ll think this article is clickbait, but it’s not. I’ve built a fully-functional static serverless Wordpress solution on AWS, with Global CDN, WAF and A-Grade SSL for literally one cent per day. It’s fast, resilient, scalable, and unlike many Wordpress sites, not susceptible to brute-force login attacks.

What’s more: You can do it too. It’s wrapped up in an open-source Terraform module and I’m kind of hoping it’ll break the internet with its accessible simplicity. You can set this up from a standing start in less than 30 minutes.

What does Serverless Static Wordpress do?

Serverless Static Wordpress is a Community Terraform Module from TechToSpeech that needs nothing more than a registered domain name with its DNS pointed at AWS.

It creates a complete infrastructure framework that allows you to launch a temporary, transient Wordpress container. You then log in and customise it like any Wordpress site, and finally publish it as a static site fronted by a global CloudFront CDN and S3 Origin. When you’re done you shut down the Wordpress container and it costs you almost nothing.

The emphasis is on extremely minimal configuration as the majority of everything you’d need is pre-installed and pre-configured in line with industry best practices and highly efficient running costs.

Have an existing site that you'd like to convert to Serverless Static Wordpress? No problem.

There’s a lot to get through here so this story is broken up into two different articles; The Short Version and The Long Version. Pick the first one if you want to grab this thing and get started immediately, and the second one if you want to know exactly how this solution works under the hood and learn about all of the work it took to put it together.

If you are familiar with AWS this will go a lot faster, but even if this is your first time The Short Version steps should get you up and running in no time.

The Short Version

For all of these steps we’ll use the example domain www.peter.cloud.

Step 0 - Create an AWS account.

Step 1 - The domain

In your AWS account, create a Route53 Hosted Zone for your domain name. Once created, this will display four DNS nameservers for your new website.

Go to wherever your domain name is registered (it could be AWS itself, or anywhere else), and update the DNS Servers to the four just created.

Step 2 - The tools and credentials

Ensure you have Terraform, AWS-CLI (with a user and credentials) and optionally (but preferably) Docker with the service started.

Step 3 - The code and the plan

Check the GitHub README for the project for extensive code examples to get set up and deploying quickly.

The module does _almost_ everything by itself, but there are a couple of extra resources specified here that’ll make it a completely hands-off end-to-end process. These are the parts that need docker installed and the AWS CLI configured properly with a default region and profile.

Terraform init, plan, and apply, then sit back and get a drink. The full end-to-end creation process here should take about 10 minutes. Unless you did something wrong, this should complete without errors - but just in case it does try running the plan and apply steps again.

Step 4 - Launch Wordpress

What you did in Step 3 is create everything you need to launch Wordpress and publish your static site - but it’s not launched yet. If you didn’t add the optional Terraform resource to trigger the CodeBuild job, you’ll need to head to the CodeBuild console in your chosen region and start the ‘-serverless-wordpress-docker-build’ job and wait for it to complete (it’ll take about 2 minutes).

Modify the ‘launch’ attribute of your module from 0 to 1, and then run ‘terraform apply’ again. Wordpress will now launch. The first-time set-up will take roughly 5 minutes while the initial site is created, configured, and the necessary plugins are installed.

By default, your Wordpress installation will become available at http://wordpress.yourdomain.com (Why no SSL for this part? Check The Long Version for details)

The default username is ‘supervisor’ and default password is ‘techtospeech.com’. You can either override these, along with the default subdomain, in the module’s configuration, or modify them after you log into Wordpress. Please make sure you do!

Step 5 - Post and Publish

This is now a Wordpress installation like any other. You can install most plugins (although any with explicit server-side functionality won’t work in a static site), any themes you like, and even import an existing site backup. SEO plugins like Yoast work just fine. If the site is idle for more than 5 minutes the backend database will pause (to save money), but simply refreshing the admin console will make it start up again after a few seconds of delay.

Whenever you want to stop the container, toggle the ‘launch’ attribute of your Terraform configuration back to 0 and re-run ‘terraform apply’. You can also manually scale the ECS Service to 0 tasks in the AWS console. A better method of one-tap launching and stopping the site will be coming in future! Stopping the container doesn’t lose any data - the database will be saved in the background, along with any files, images, plugins or themes you installed. Its state is perfectly preserved until the next time you launch it.

But - it is critically important that you remember to shut the container down when you’re done. Otherwise it’ll keep on running - it won’t cost you too much if it does but the whole point of this solution is to save money, not waste it! (Another future update will have an auto-shutdown-when-idle feature)

When you’re ready it’s time to publish. Check out the WP2Static plugin that has been automatically installed for you, along with the WP2Static S3 Addon. Leon Stafford is the creator of these plugins. He’s the open-source static Wordpress guru without whom this project would not have a deployment mechanism. I talk more about Leon in The Long Version.

You can check out all of the WP2Static options you can tweak to configure your deployment, but the auto-setup has pre-filled everything you need to publish. Hit the ‘Generate Static Site’ button and periodically refresh the log to check on status. This will now crawl and rewrite every URL to a neat static version that then gets pushed directly into the S3 bucket backing the static version of the site.

This process can be as fast as 3-5 minutes or much longer if you have a larger site. We’re working on faster methods for these different steps (and help is gratefully received!) but at the moment the version bundled with this set-up is stable and reliable with a little patience.

Once the process completes…. You’re done! Head to the https version of your site (in this example, https://www.peter.cloud) and be amazed at how your website looks identical to how it did before, except now it’s serverless, static, cached with a Global CDN and won’t go down to some feeble DDoS probing or wp-login brute-forcing. In fact, try to go to https://www.peter.cloud/wp-admin - ha! Doesn’t exist!

Is it really only $0.01 a day?

Yes, but also no. It depends.

Consider the typical use case. You’re a nerdy tech person with a personal blogging site (and I am, check out petewilcock.com), you’ve got a few articles, you probably get between 50 and 200 hits a day. With that kind of traffic the site will run in its static form without any intervention for around $0.01 a day - completely legitimately.

On the other hand, if you’re a globally-popular massive content-generation business with thousands of articles and millions of hits a month… it’ll cost more. Your only variable cost with traffic is CloudFront distribution charges, and you can mitigate these by either getting a discount with a CloudFront Security Savings Bundle, or if you’re a particularly big player you can enquire about custom CloudFront pricing directly to AWS. WAF has some unavoidable fixed costs that completely falsify my clickbait title and will add at least $0.60 a day to running costs. But if you're suspeptible enough to need the WAF, I'm guessing you can afford it.

As I run several static Wordpress sites using this set-up, I’ve paid literally $1 a month for a Savings Bundle and that covers a lot of my usage. Previously I was running a T3 web hosting server with CPanel licencing that cost around $600 a year. Now it’s not a strictly fair comparison for a few reasons (this set-up doesn’t handle email at the moment for example - but look out for a future update!), but you start to get an idea of the difference between this and a ‘conventional’ hosting set-up for a basic website. Even other providers of ‘Static’ Wordpress sites can’t compete with this, because they need a profit-margin and you don’t.

The only other costs to be aware of are the ECS Fargate container running costs that backs Wordpress (and it runs in Spot mode so it’s very cheap), and the RDS Aurora Serverless v1 database that backs the Wordpress database. This is actually the most expensive bit (and you guessed it, a future version will offer a cheaper alternative if you want to sacrifice the features and convenience), but you’re only charged for the time you’re actively modifying the site. As an example, if I’m editing the site for several hours, this might cost around $0.30. The key part is that the vast majority of the time when you’re not adding content, it costs practically nothing.

Troubleshooting

No set-up is ever perfect, so if you have any issues with the Terraform module please report them on our GitHub and we’ll take a look. Contributions are also welcome!

If you experience an issue with the WP2Static plugin, check out their issues page for common issues and solutions and the chances are anything you encounter can be tweaked to work as you’d like. You will be free to upgrade the plugin, or Wordpress itself, whenever you like - but always take a backup first! I recommend UpdraftPlus as a great free Wordpress backup plugin.

In Conclusion

Firstly another reminder. Don’t forget to turn off your Wordpress container when you’re done!

This module is the culmination of many months of painful iterations both to get it working for myself, and then extensively bashing it into shape so I could publish it publicly without embarrassing myself. It leverages all of my experience of AWS since I started with it back in 2014 and compresses every element of DevOps and web development I’ve ever learned into a tight solution that can work for anybody. I hope it works for you!

To learn more about this mad journey, check out The Long Version article.

-- Pete Wilcock is a 9x AWS Certified DevOps Consultant, AWS Community Builder, and Technical Writer for TechToSpeech. If I’m not possibly losing my mind and definitely my social life buried in some project, you can find me on LinkedIn, Twitter, GitHub, or my personal site.

AWS Transfer Family – FTP for EFS and S3

Pete Wilcock — Wed, 19 May 2021 09:36:25 +0000

If you were curious what AWS Transfer Family is, I’ve already spoiled it in the title. But what is it really, why does it exist, and when would you use it?

A Historical Gap

AWS used to have this small problem – it offered some awesome and powerful storage solutions like EBS (which needs to be mounted to an EC2 instance) and S3 (which you could only interact with via the console, SDK, or CLI). When I started using AWS, one of the first things I can remember searching for was “Can I FTP files to S3?”, and that’s a no.

Naturally the community responded and s3fs-fuse is a utility that allows you to mount S3 as a filesystem to your EC2 instance. With a development history going all the way back to 2008, it’s been impressively and consistently developed, adding in new features and bug fixes.

What it can’t do however is get away from the fact that S3 is not a filesystem, and not all POSIX commands are supported. Those that are are a cunning kind of alias to S3 operations that emulate equivalent behaviour. There’s also the risk of ramping up significant costs – filesystem operations translate to S3 API requests which have a cost of $0.005 per 1000 PUT, POST, and LIST requests, and $0.0004 per 1000 GET, SELECT, and other requests. This might not sound like much, but if you inadvertently trigger or have running some background processes that are read/write intensive, you wouldn’t normally think about it. With s3fs, all of those requests can start to mount up. Maybe not a dealbreaker, but be aware.

But with it mounted to an EC2 instance, you could install your favourite FTP server and upload ‘directly’ into S3.

Now I need to point out I’m being glib with my casual use of the term ‘FTP’. Unsecured FTP on port 21 has been a big no-no for a very long time, so really when I say FTP I mean some secured method which is one of: SFTP (FTP extension of SSH) or FTPS (FTP with SSL certificate).

EFS has entered the chat

Then along comes EFS in June 2016 – AWS’ version of the classic Network File System (NFS) for Linux only (look for Amazon FSx for a Windows solution). It’s more expensive than EBS (which is $0.08 per gb/month with the new gp3 volume) at a whopping $0.30 per gb/month. But where EBS charges you for storage provisioned (regardless of whether you fill it or not), EFS only charges you for what you use and could end up being far cheaper depending on how much headroom you’re putting into your EBS volumes – and of course EFS can be mounted to multiple different instances so overall you’ll get some good mileage out of it for the cost.

Still the same issue though, you can’t FTP into it directly. It still needs to be mounted to an EC2 instance (or latterly, an ECS container or Lambda function), but it at least is a fully POSIX-compliant file system. At the time of writing EFS also doesn’t have a front-end file manager in the AWS Console. I’m sure it’ll be a feature that comes along eventually, but in the meantime I feel a bit blind not knowing what’s on my EFS unless I mount it somewhere to inspect it!

So clearly, we needed more…

AWS Transfer… Orphan?

AWS Transfer Family started out as AWS Transfer for SFTP in November 2018, and rebranded and expanded to include FTPS and plain FTP in April 2020.

The offering is essentially a managed FTP service with S3 as the endpoint for the data. At the time of writing in January 2021, AWS Transfer Family for EFS is red hot off the press as the latest endpoint available, and finally we have a way to examine and transfer files into our EFS volumes without having to mount them somewhere first!

But, as with all managed services this comes at a price. What is it?

Price Comparison

Firstly, let’s remind ourselves about S3 and EFS. Uploading into S3 from the internet is free. As mentioned EFS needs to be mounted onto something, but assuming that’s an EC2 instance then uploading data into EC2 from the internet is also free.

At present the price for AWS Transfer Family for SFTP, FTPS, and FTP is the same for all protocols:

Feature	Cost
Time protocol is enabled on your endpoint	$0.30 per hour (and charged by hour)
Data uploads	$0.04 per gigabyte (GB) transferred
Data downloads	$0.04 per gigabyte (GB) transferred

Ouch – just turning it on and not using it will cost you around $219 a month. That’s roughly the same as an r5.xlarge EC2 instance on demand, a pretty chunky beast with 4 vCPUs and 32 GiB of RAM and 10Gbit of Networking. BIG CAVEAT: Stopping an AWS Transfer Family endpoint does not affect billing. So unlike EC2, you will be charged for the service even if it is stopped – the only way to stop being charged is to delete it completely.

Then for the data transfer – that actually compares quite favourably for downloads set against S3 and EC2, which starts at $0.09 per GB and only goes as low as $0.05 if you’re downloading more than 150TB a month.

Let’s do a full cost comparison with the following assumption: You upload data into your FTP server, and download the same data again – a full up/down transfer cycle.

AWS Transfer Family Data Transfer Cost Comparison

GiB	S3/EC2 Up/Down	Transfer Family Up/Down
1	$0.00	$0.08
100	$8.91	$8.00
1024	$92.07	$81.92
TiB
5	$460.71	$409.60
10	$921.51	$1,228.80
50	$4,403.11	$4,096.00
100	$7,987.11	$8,192.00
150	$11,571.11	$12,288.00
300	$19,251.11	$24,576.00

Then I decided I wanted a more granular comparison, so I spent far too long making this graph:

Amazon Transfer Family vs S3/EC2 Transfer Costs

The point at which EC2/S3 data transfer becomes cheaper than AWS Transfer Family (ATF) is exactly 80TB. The lack of bulk data pricing on ATF starts to hurt it when the volume is high enough!

But I stress this is purely data transfer. If we assume you’ve mounted S3 as a filesystem on your EC2 you’re still paying for those S3 API operations too. There are a few commercial offerings for an “SFTP to S3” product that would seem to do exactly this.

When would you use it?

Quite simply, when you need to use FTP and have no other choice:

Legacy systems – and we know there’s still plenty of those in wide operation. They will have no notion of cloud and can only do exports and data transfer with something as simplistic as FTP. Giving them a familiar protocol to speak to will in some cases be the only way to integrate something ancient with the Cloud.

Client Preference – you’ve got a client you want to exchange data with and setting them up with an IAM user or role and walking them through how to do a cross-account S3 upload just isn’t going to fly a lot of the time. Some clients will want the familiarity and good old FTP, particularly if they’re used to dealing with data transfer in this way. Many of them will have performed expensive and time-consuming risk and compliance analysis of SFTP or FTPS and won’t want to go through the process for another methodology if they don’t absolutely have to. Having said that, popular FTP clients such as Filezilla do support S3 protocol, but it’s still so unfamiliar to many that they won’t even consider it.

Looking at the AWS Customer stories, this appears to be the two main cases where companies are delighted that AWS Transfer Family exists. Then you have my own case – I want to poke around my EFS drives without the hassle of spinning up my own instance to mount it!

Other Features of AWS Transfer Family

Use your own identity provider. A big one – being able to connect up Activity Directory or similar to grant users access to FTP comes with all the benefits of a single source of truth for identity.
Use your own domain name – a simple CNAME to the service endpoint will let you brand your FTP endpoint as desired.
Fixed IP (including BYO IP) – allowing you to have external parties whitelist your FTP endpoint, in line with their own security policies. Naturally, you can whitelist incoming connections yourself via Security Groups. FTPS integrates with Amazon Certificate Manager – keep all your SSL management in one place.
Cross-account support. You can allow access to the service across AWS accounts with cross-account IAM roles.
CloudTrail and Cloudwatch support – monitor user activity with all of the possibilities of integration with GuardDuty.
Rock-solid Compliance: AWS Transfer Family is PCI-DSS and GDPR compliant, and HIPAA eligible. The service is also SOC 1, 2, and 3 compliant. Are you going to get that with your self-hosted FTP on EC2? I doubt it! ##File Exchange Protocol (FXP) Does AWS Transfer Family support FXP? Well this is an interesting one, because I couldn’t find it covered in the documentation anywhere. If you don’t know what this is, I’ll borrow from Wikipedia’s definition:

File eXchange Protocol (FXP or FXSP) is a method of data transfer which uses FTP to transfer data from one remote server to another (inter-server) without routing this data through the client’s connection.

Or in other words, FTP to FTP transfers. This is really convenient if you have two remote systems and don’t want to have to pass the data through yourself as the slow proxy in the middle.

So I tested this out for myself – using an FTP set up on an EC2 instance, I set up AWS Transfer Family SFTP endpoint and attempted an FXP transfer between the two. So does AWS Transfer Family support FXP? Yes! I was able to connect and FXP my files across using FXP no problem.

Helping you to not be stupid

As we’ve seen often with S3, the fact you have the ability to control access permissions with a high degree of granularity doesn’t mean you’ll use them properly. Whilst AWS maintains the precepts of the Shared Responsibility Model, they’re also adding in more features to services to stop you being stupid.

Consequently you can deploy AWS Transfer Family as the plain, old, horribly insecure vanilla FTP – but only within a VPC. You can’t make it internet-facing and public by default. You can’t connect it to your Activity Directory identity provider (too insecure), and if you’re absolutely determined to expose it on the public internet you need to put a Network Load Balancer (NLB) in front of it.

Basically you’re not going to trip and do any of that by accident, but if you absolutely want to, you still can and the risks are shouted in your face at every step.

Summary

AWS Transfer Family is in line with most other AWS managed services. Nice features, nice integrations with the rest of the platform and other software. But like most managed offerings it’s going to cost you a pretty reasonable premium over rolling your own cheaper albeit less-elegant solution.

So you need to be sure your requirement is great enough to justify the running and transfer costs, particularly in consideration to the fact there’s no way to power the service down out of hours to save money as you would with EC2 – and that’s a grave disappointment. Hourly billing is also a tad regressive – we’re getting quite used to per-minute and per-second charging these days and this feels like an obvious improvement.

Have you used ATF for anything? If so I’d be interested in your comments below!