DEV Community: t3chflicks

Emoji Search 🔮 Creating, Deploying and Evaluating a Machine Learning Service

t3chflicks — Wed, 14 Jul 2021 13:02:28 +0000

Computers can model language in such a way that they can perform well on tasks such as text similarity. In this article, I attempt to create an Emoji suggestion tool which uses vector word embeddings to determine the relevant Emoji recommendation. I also deploy the machine learning model as a service on AWS and dive into cost estimations. A live example of the service can be found on the T3chFlicks site; all the code is open sourced.

🔗 Find All the Emoji Search Code On Github📔

Emoji 😂 ❤️

Emojis are common-place in internet communication, the tiny intricate pictograms have opened the door to quick thumb tap summarisation.

Computers represent Emojis using the Unicode Standard:

Code: U+1F600 (Unicode)
      \xF0\x9F\x98\x83 (UTF-8 Bytes) 
Emoji: 😀           
Description: grinning face

The popularity of Emojis differs massively across societies and cultures, but Unicode publish some stats on their proportional use:

https://home.unicode.org/emoji/emoji-frequency/*

What do we want?

During project conception we decided that we wanted the following:

To enter a search text and in response get a list of similar Emojis
To run a machine learning model in AWS Lambda
To use this service to improve the efficiency of adding Emojis to our articles.

Build — Emoji Search

To be able to search text and be suggested the most similar Emojis, we need to find the similarity of the search text with the Emojis.

First we must try to compare apples with apples, and luckily Emojis in the Unicode Standard have a short description of the image, using these we can compare the two strings search_text vs. emoji_description .

One current technique for modelling language is using word embeddings — a vector space where the dimensions and relations represent meaning between words.

We really recommend the following video for understanding the concept of embeddings:

Searching Emojis using Word Embeddings

We’re using Google News Vectors from 2019 as a language model, this isn’t necessarily the best choice (newer models exist such as Bert), but it will definitely be up for the task. A condensed version of the model can be found here.

A simple way to investigate the learned representations is to find the closest words for a user-specified word. [source]

We first load the model in the *word2vec* format:

import os
import gensim
GOOGLE_NEWS = os.path.join(FIXTURES_FOLDER, "GoogleNews-vectors-negative300-SLIM.bin.gz")

model = gensim.models.KeyedVectors.load_word2vec_format(GOOGLE_NEWS, binary=True)

This model is used to process Emoji data. Luckily, all the information we need is found in a Python library called emoji_data. We start by extracting the Emoji text into a Pandas data frame. For simplicity, we only keep the standard yellow Emoji set.

from emoji_data import EmojiSequence
d = {'emoji': [], 'description': []}
for (emoji, emoji_meta) in EmojiSequence:
    d['emoji'].append(emoji)
    d['description'].append(emoji_meta.description)
df = pd.DataFrame(d)
df['description'] = df['description'].str.split(' skin tone').str[0].str.replace(':', '').str.replace(',', '')
df = df.drop_duplicates(subset=['description'])

---------------------------------

emoji   description
👨‍❤️‍👨      [couple with heart man man]

The vector of each word in the description text for the Emoji is averaged into a vector for that Emoji:

def process_words(text):
    words = text.split(" ")
    word_vectors = [ model.syn0norm[model.vocab[word].index] for word in words if word in model.vocab]
    if len(word_vectors) > 0:
        mean_vector = np.array(word_vectors).mean(axis=0)
        unit_vector = gensim.matutils.unitvec(mean_vector).astype(np.float32).tolist()
    else:
        unit_vector = np.zeros(model.vector_size, ).tolist()
    return unit_vector

df['vector'] = df['description'].apply(process_words)

For example, using the search text ‘dog’ and cosine similarity of emoji_vector vs. search_vector we can establish the similarity of all the Emojis. We can then select the top two.

from sklearn.metrics.pairwise import cosine_similarity

def find_similarity_to_search(search_vector):
    def func(emoji_vector):
        b_emoji = np.array(emoji_vector)
        cos_sim = dot(search_vector, b_emoji) / (norm(search_vector) * norm(b_emoji))
        return cos_sim
    return func

search_text = "dog"
search_vector = np.array(process_words(search_text))
find_similarity_to_search = find_similarity_to_search(search_vector)
df['similarity'] = df['vector'].apply(find_similarity_to_search)
df.nlargest(2, 'similarity')

--------------------

emoji similarity description
🐕 1.000000 dog
🌭 0.775372 hot dog

Whilst that example is rather plain, here is another that made us chuckle and think the service works well:

search: "giant obstacle"

-------------------
emoji similarity description
🦣 0.416749 mammoth
🤘🏽 0.352729 sign of the horns medium
🧗🏽‍♂️ 0.351723 man climbing medium
🐉 0.330775 dragon
🗺️ 0.330681 world map

Architecture of the Emoji Search API

The Python script above can be run on the Cloud and made accessible via the internet as an API. We use AWS as our Cloud provider with CloudFormation to write infrastructure as code.

The architecture uses an Application Load Balancer with Lambda as an API, we have already discussed this in a previous article:
Cheaper than API Gateway — ALB with Lambda using CloudFormation
*An alternative to API gateway is Application Load Balancer. ALB can be connected with Lambda to produce a highly…*t3chflicks.medium.com

AWS Lambda supports using docker containers, this allows us to run the Lambda locally for development and testing. The script is packaged into the docker container with the Google News Vectors as this makes it accessible at runtime without requiring downloading from S3 or EFS. The Lambda required 1.5GB memory to run the model.

Deployment of the Service

Deployment pipelines can be created on AWS using CodePipeline. This process is triggered by commits to a Github repository. It follows a sequence of defined steps, beginning with provisioning the infrastructure for the service, then testing the code, building a docker container and finally updating the Lambda:

Thanks to the super developer James Turner for showing us how to create CI/CD pipelines on AWS that are triggered by Github commits:
From GitHub to Continuous Deployment in 5 Minutes
*How to go from having a Github repository to having a CD pipeline in AWS where you can run tests, and continually…*aws.plainenglish.io

The Interface

We made a widget that allows users to easily play with the API and view the JSON schemas which define the input and output:

This widget allows you to interface with the API. Visit https://t3chflicks.org/services/emoji-search to play!*

The programmatic use of the API is as follows:

url '[https://api.t3chflicks.org./emoji-search/recommendations'](https://api.t3chflicks.org./emoji-search/recommendations') \
  -H 'content-type: application/json' \
  --data-raw '{"key":"4aac6db6d8dfc3fa7693b8b5918b3d548e3c92ebd70832447bab247a1c0b0b477d9107ecfe05f8153ef6d3d848614de3cb8bcf42e256aac1444eb828be1f4083","search":"Dog","quantity":5}' \
  --compressed

>>> {"most_similar": [{"score": 0.6978708999734073, "emoji": "\ud83d\udc15", "description": "dog"}, {"score": 0.5574100960192504, "emoji": "\ud83c\udf2d", "description": "hot dog"}, {"score": 0.5498123048808492, "emoji": "\ud83e\uddae", "description": "guide dog"}, {"score": 0.5273177453464898, "emoji": "\ud83d\udc08", "description": "cat"}, {"score": 0.4822364083781162, "emoji": "\ud83d\udc15\u200d\ud83e\uddba", "description": "service dog"}]}

🤖 The widget can be found on our site 🤖

The Cost Estimation

Determining the cost of running the Emoji Search service is similar what we have done in a previous article:
Giving away free APIs without going broke.
*I like to write articles on Medium. I also want to share the same articles on our own site without having to rewrite…*t3chflicks.medium.com

For the architecture described the cost equation is:

Total cost = Application Load Balancer cost + Lambda cost

We will continue with the assumptions of **4350 Requests/Month** from 100 users using their 100 free credits.

Lambda Cost

The Emoji Search Lambda has the following response stats:

542 ms on average @ 1536 MB memory (cold start 1362ms)

These numbers can be plugged into the Lambda cost calculator:

A nearly negligible cost at $0.06 / Month. However, when scaling into the millions of requests, the service does begin to become quite costly:

In the scenario that we did receive this sort of request rate, we would undoubtably have to rearchitect.

Application Load Balancer Cost

The Load Balancer will process requests of the following size:

Processed bytes = (167B request + 172B response) * 4350 requests = 1,474,650B** = **1.475MB = 0.001475GB

These numbers can be plugged into the Application Load Balancer cost calculator:

Total ALB Cost = (ALB/month) + (LCU/month)
Total ALB Cost = $0.23 +(730hours/month * $0.0252/hour) = $18.63

Total Cost

At a request rate of 4,350 per month, the service can be estimated to cost:

Total Cost = Application Load Balancer cost + Lambda cost
Total Cost = `$18.63 + $0.06 = $18.69

As many services can share the same ALB, their cost will be shared across projects. A cheaper alternative would be to use API Gateway.

We do not expect this much traffic and simply want to expose the service for fun.

We have created a Cost Budget with alarms that trigger when the service is going to exceed the expected cost.

Conclusion

At T3chFlicks we want to share, this is why you’ll find the code that created this project is open sourced. We’d love to hear from experienced developers / business owners on how they managed to run a successful SaaS, please reach out!

Thanks For Reading

I hope you have enjoyed this article. If you like the style, check out T3chFlicks.org for more tech focused educational content (YouTube, Instagram, Facebook, Twitter, Patreon).

Giving away free APIs without going broke.

t3chflicks — Fri, 11 Jun 2021 14:36:54 +0000

I like to write articles on Medium. I also want to share the same articles on our own site without having to rewrite them in a different format. For this reason, I have created a service which runs something similar to the Medium-To-MarkDown library, a programme which converts Medium posts to MarkDown format. I have offered it to everyone on our site.

To be clear, Medium-To-MarkDown is an example service whilst we work out how best to run an effective Software as a Service (SAAS).

In this article, we:

Break down the hosting costs of the service
Determine how feasible it really is to give away free APIs
Discuss some tactics to avoid going broke

What do we want?

To send an article URL and in response get the MarkDown
To share this service with the world
To avoid T3chFlicks going into financial meltdown

We want to use this service and render the MarkDown on our site for the posts we write. We typically write one article each month, so we can’t really justify the purpose of an API. Instead, a better solution might be to execute a script for the conversion when building our site.

However, at T3chFlicks we want to share - our only opposition is financial. This is a fairly simple service, running on AWS (the largest Cloud Hosting company). Surely this wouldn’t be too much for a UK-based tech company?

The Architecture

The architecture diagram above shows a user making a request to the MediumToMarkDown/Convert endpoint on the load balancer. The Application Load Balancer (ALB) sends this event to the API Lambda, which responds immediately with a URL for the location of the MarkDown article result. The API Lambda also puts the Medium-To-MarkDown job into a SQS queue to be consumed by the Processing Lambda.

The Processing Lambda is where all the work is done, and the result is written to an S3 Bucket which is attached to a CloudFront distribution. This has a lifecycle policy set to delete after one day.

The Medium-To-MarkDown processing script is pretty simple and can be found here.

The Interface

We made a widget that allows users to easily play with the API and view the JSON schemas which define the input and output:

T3chFlicks API-Play Widget

The programmatic use of the API is as follows:

curl --location --request POST 'https://api.t3chflicks.org/medium-to-markdown/convert' \
--header 'Content-Type: application/json' \
--data-raw '{
"mediumURL": "https://t3chflicks.medium.com/how-to-use-amplify-auth-in-nuxt-js-4dbf07da7033",
"key": "52d9204da6c6f8cefd13fb43384c365656dc091473634e3b701892e3cc29abef7af4a8de1f3913235bd4b56a409e09597400da67e358cd62e118d3922563791e"
}'
>>> {"location": "https://document-store.t3chflicks.org/medium-to-markdown/document-title"}

Accessing the MarkDown document:

curl https://document-store.t3chflicks.org/medium-to-markdown/document-title

>>> "# How to use Amplify Auth in Nuxt JS \nVue JS is a frontend framework often extended with Nuxt JS to get great SSO value out of the box. AWS Amplify allows you to add authentication to your Vue web app with ease as shown in a previous article. In this article, I demonstrate how to add Cognito authentication with Amplify to a Nuxt web app..."

🤖 The widget can be found on our site 🤖

The Cost Estimation

When thinking about what our limits are on giving away free services, we can approach the problem from two sides:

How much is the maximum number of requests going to cost?
How many requests can I get for $2/month?

On the T3chFlicks site, we give users 100 free credits per week. At the time of of writing, we have 100 registered users. If we assume all of our users use all of their credits each week for a month on the Medium to Markdown service, then…

Number of requests/month = (100 credits/week * 100 users * 4.35 weeks/month) = 43,500 Requests / Month

That’s a small number of requests, but will it incur a small AWS cost?

The total cost of the service is the sum of the individual component costs:

Total Cost = (ALB cost) + (Lambda cost) + (SQS cost) + (S3 cost) + (CloudFront cost)

These costs are dependent on the amount of data transferred. Apparently, there is an optimum length for a Medium article ~ 1,600 words. At 4.79 characters per word on average in UTF-8 means:

Size of Article = 1600*4.79*1byte = 7,664 bytes ~7.7KB

An example T3chFlicks article has a size of 5.39KB, so this seems legit.

Data transfer/ month = (number of requests / month) * (size of response)
*Data transfer / month = 7.7 KB * 4350 Requests per Month = 33,495 KB ~ 33.5MB

Load Balancing cost

The API is created by an ALB. The cost is broken down into:

$0.0252 per ALB-hour (or partial hour)
$0.008 per Load balancing capacity unit - LCU hour (or partial hour)

LCUs have four usage dimensions and you are charged based on the highest dimension:

Number of newly established connections per second (25/sec)
Number of active connections per minute (3,000/min)
The number of bytes processed for requests and responses (1 GB /hour on EC2, and 0.4GB/hour Lambda)
Number of Rule evaluations (1,000/sec)

Determining this cost is quite difficult, so we made use of the calculator on the ALB page. However, with our estimates for usage of this service, the cost seems to be negligible.

The size of the request is tiny:

Processed bytes = (33Bytes request + 33Bytes response) * 4350 requests = 287,100B = 287KB = 0.287MB = 0.0003GB

The estimated $0.23 for LCU charges is very manageable for our goal of $2. Even at 26 million requests/month, the LCU charge only increases to $2.34.

A cheaper alternative given the predicted rate of requests is the API Gateway service. However, many services can share the same ALB, and since we can share this across many of our projects, so will the cost, meaning we can assume it to be zero.

Total ALB cost = (ALB/month) + (LCU/month)
Total ALB Cost = $0.23 +(730hours/month * $0.0252/hour) = $18.63

Once the traffic has used ~30 LCU, then the cost of traffic will be equal to that of the unused ALB service.

Lambda cost

With AWS Lambda, you pay only for what you use. You are charged based on the number of requests for your functions and the duration, the time it takes for your code to execute — source.

The are two different Lambdas in the service:

API Lambda — immediately returns a response of the result location

197.5 ms on average @ 256 MB memory

Only when getting into the millions of requests does this service begin to cost:

Processing Lambda — performs the transformation of the article and uploads the result

This part of the service depends on the response from many third parties including Medium, YouTube, and Github. This means response times might vary wildly depending on the contents of the Medium blog post. A hard limit of 10 seconds was set on the Lambda runtime.

650ms on average @ 256 MB memory for a standard length blog post

😱 But what if users want to process huge articles 😱

✅ Set a Lambda timeout to avoid long running times ✅

Queue cost

This service does singular batches of items on the SQS queue, but could easily be extended to do larger batches. The pricing page shows that for fewer than a million requests, this service is free:

We can therefore assume the SQS cost to be negligible as we do not expect a million requests per month.

Storage cost

The result of the process is stored in an S3 bucket and users are sent a URL of that object.

At 4350 requests per month, ~135 requests per day, with a bucket lifecycle of one day, the maximum expected to be in the bucket at one point is:

`200 articles, 7.7 KB each, 1,540KB = 1.5MB`

`$0.023 per GB storage costs on S3 for the first 50TB`

`1.5/1000 * 0.0023 = $0.00000345`

Again, this can be considered negligible, probably up until 10 million requests:

7.7KB/1000/1000 * $0.0023/GB * 10,000,000R = $0.1771

CloudFront cost

CloudFront is used in this architecture as a layer around the S3 bucket. The costs for CloudFront origin requests (first time accessing) are free for S3 bucket origins and $0.085 for the first 10TB transferred to users, meaning this cost is negligible up until millions of requests.

Total Cost

For 4,350 requests
Total Cost = (ALB cost) + (Lambda cost) + (SQS cost) + (S3 cost) + (CloudFront cost)

Total = ( $18.396 + $0.23 )+ ($0.00 +$ 0.01) + ($0.00) + ($0.00 ) + ($0.00 )

Total = $18.396 + $0.23 + $0.01

Total = $18.636
Total minus standard ALB charge = $0.24

🙌 I think we can afford 24 cents per month 🙌

Capping Usage

As we previously mentioned, the other way to look at this problem is:

If we put a monthly cap of $2 on this service, how many requests could we support?

For 435,000 requests (100X more)

Lambda costs — $1.27
Lambda costs — $0.45
ALB costs — no change $0.23
Storage costs — Negligible
CloudFront costs — Negligible

Costs = 1.27 + 0.45 + 0.23 = $1.95 ~ $2

100 times more requests would mean 100 times more users:

🌎 We’re happy with spending $2 on 10,000 people 🤗

The Real costs

Cost Tagging

Tagging your resources makes it much easier to retrospectively understand costs, as the AWS billing console can effectively use these tags to split up the bill.

😱 Resource costs are hard to pin down😱

✅ Tag all resources and create a cost budget with project filters✅

AWS Budgets

AWS offers budgets to help solve the problem of capping usage. The service allows you to filter service costs into a group and place alarms on monetary metrics such as expected monthly forecast. You can use these to trigger an email message.

😱 But what if we suddenly get lots of users? 😱

✅ Put a budget alarm on the service✅

We have done this for the Medium-To-MarkDown service by tagging the service resources with a user defined cost allocation tag: Project , and creating a budget which filters using the same tag.

A budget shows a detailed dashboard, which is rather empty at the moment:

Other Considerations

Rate Limiting

ALB does not allow for rate limiting. For that, you must add AWS WAF onto your API which costs $5/month + $1/month per million requests - quite expensive.

How bad would it be if somebody all spent their credits at once?

This could increase the LCU to a high level for a short period of time which could be bad but probably not awful.

Service Status

It is good practice to monitor your service for errors. You can monitor how many users are receiving 4XX and 5XX responses from the ALB, and configure alarms which trigger email alerts.

CI/CD

We used AWS CodePipeline to orchestrate this entire service.

Alarms and a deployment pipeline with tests mean that we minimise the likelihood of updating this service with a breaking change.

Conclusion

At T3chFlicks we want to share, this is why you’ll find that our projects – whether they’re a Smart Buoy for ocean measurement, or a Software as a Service quickstart – are open sourced. This API is due to be followed by many other services and as long as we don’t go broke, we will continue. We’d love to hear from experienced developers / business owners on how they managed to run a successful SaaS, please reach out!

Thanks For Reading

I hope you have enjoyed this article. If you like the style, check out T3chFlicks.org for more tech focused educational content (YouTube, Instagram, Facebook, Twitter, Patreon).

⚡ 🛋️ Wireless Charging Sofa

t3chflicks — Wed, 17 Mar 2021 18:00:50 +0000

Fed up of the wires and hassle of plugging and unplugging your phone as you move around the house? So were we!

We’ve made a wireless charging cover which fits snugly on your sofa arm and blends in seamlessly. This simple make is a great way to upgrade your sofa and is a step on the path to eternal laziness: charging your phone is as simple as putting it down.

Supplies

Faux Leather (Fire retardant) Amazon
Wireless Charge Receiver Amazon
Wireless Charging Module Amazon
Micro-USB Wire Amazon
2A USB Plug Amazon
Sewing Machine Amazon

🔗 Get The Wireless Charging Sofa Files On Github 📔

Tutorial 🤖

Measure Up

Measure the arm of your sofa. You will make the cover in two pieces: one which covers the front of the arm and another piece which wraps around the top of the arm. First, measure the front piece by measuring the width of the arm (at its widest point on curved arms) and how far down the arm you want the cover to go — when doing this, bear in mind you probably want it to sit between the cushion and the arm.

Once you have established how far down the arm you want the cover, measure the second piece which will go across the arm. Measure how deep you want the cover to be.

Measure Wireless Charger Pocket

The wireless charging module will be housed within a pocket inside cover to stop it moving around. We made ours 15.0cm by 9.5cm — based on the size of our wireless charger.

Draw Out Measurements

Add approximately 1/2cm to your measurements (for stitching) and draw the rectangles on the back of your material — there should be 4 in total. Cut them out.

Sew the Pocket

Make the pocket for the wireless charging module. To do so, take the two small rectangles and place them on top of each other so the front of the material is facing inwards on both pieces.

Place the wireless charger between the two pieces and draw a line across the fabric where it ends. Sew along this line to the two pieces of fabric are joined in the middle.

Pin the Cover

Pin the arm cover ready for sewing. With the material face down, lay the rectangle across the top of the arm. Pin the front rectangle onto it, leaving at least 0.5cm seams.

If you have a sofa with a rounded arm, you will need to gather more material where the arm curves to achieve the right shape.

Sew the Cover

Sew the cover together while it is still inside out, removing the pins as you go. If you need guidance on how to hand stitch, or how to thread and use a sewing machine, check out these tutorials (1, 2) from the wonderful world of YouTube!

Test on Arm

Once you’ve finished sewing the cover, turn it inside out and check you’re happy with how it fits on your sofa arm.

Hem the Edges

Hem the edges of the cover by folding approximately 1/2cm of each edge backwards and pinning in place. Sew neatly to create a hemline which will give the cover a clean finish.

Pin the Pocket on the Cover

Place the cover on the sofa arm and put the rectangles you sewed earlier to house the wireless charging module beneath it with the half which fits the wireless charging module facing towards the back of the sofa. The pocket should sit in the middle of the arm in a place you can balance your phone. When you’re happy with the location, pin in place.

Sew in the Pocket

Sew the pocket along the front and two side edges. When sewing along the rear edge, only sew the top layer of the pocket onto the cover. This is so that the wireless charging module can be added or removed but also so that the rectangle shape on the top of the cover, which shows where you put your phone for it to charge properly, is a complete shape.

Plug in Wireless Charger

Put the wireless charger inside the pocket and plug it in.

Enjoy!

Put the cover your sofa arm you’re ready to go!

Thanks for Reading

I hope you have enjoyed this article. If you like the style, check out T3chFlicks.org for more tech focused educational content (YouTube, Instagram, Facebook, Twitter).

🔪 The Ultimate Knife Block

t3chflicks — Wed, 17 Mar 2021 17:59:40 +0000

We’ve all been there, chopping vegetables with a knife so blunt it would be more effective to use a teaspoon. In that moment, you reflect on how you got there: your knives were sharp as razors when you bought them but now, three years down the line, they’re thoroughly inadequate. “I should’ve sharpened my knives” you think to yourself. Shoulda, coulda, woulda but I didn’t.

Most of us don’t bother to sharpen our knives. It’s an extra bit of effort and when you’re just trying to make dinner, faffing around with a sharpener is the last thing you want to do. But what if it wasn’t..?

We decided to make a knife block which incorporates a mechanical knife sharpener. A sharpener right next to your knives — and solar powered so you don’t even need to bother charging it! This build is super straightforward and you end up with a great final product which would be a helpful addition to any kitchen!

Supplies

Rechargeable 18650 battery — https://amzn.to/2XF7cno
Charge controller TP4056- https://amzn.to/2XF7cno
Push button — https://amzn.to/2XF7cno
Small motor — https://amzn.to/2XF7cno
Battery Holder — https://amzn.to/2XF7cno
Glue Gun — https://amzn.to/2XF7cno
Soldering Iron — https://amzn.to/2XF7cno
Wire — https://amzn.to/2XF7cno
Sharpening stone — https://amzn.to/2XF7cno
3 packets of spaghetti — https://amzn.to/2XF7cno
Red PLA — https://amzn.to/2XF7cno
Grinding stone — https://amzn.to/2XF7cno
3X Screws 12mm m3 — https://amzn.to/2XF7cno

🔗 Get The Ultimate Knife Block Files On Github 📔

Tutorial 🤖

Knife Block Design

The basic knife block design is a curvy cuboid with a detachable lid and a space for a solar panel in the front. The lid has slots for the knives. To figure out how big the block needed to be and how wide the knife slots would be, we measured the knives we wanted to put in and designed accordingly.

To power the rotating sharpener, we decided to use a solar panel to keep the design cordless (you don’t want to plug another thing in the kitchen) and remove the hassle of recharging batteries. Also, chances are that unless you’re a serial knife sharpener, a solar panel will provide plenty power.

The electronics are pretty simple to put together. For power, you need one rechargeable battery — preferably 18650 lithium ion. To charge it, you’ll need a solar panel — we used 5V, 500mA because we had one spare, but a smaller one would be perfectly fine. You’ll also need a battery protection circuit and something to put the battery in.

The whole thing will be controlled by a simple button which sits in the top of the knife block. To operate the sharpener, the button needs to be depressed. This is actually a pretty good safety mechanism because it means the sharpener will stop turning as soon as you let go of the button. On the end of the motor, there’s a small grinding stone which I found online.

Print the 3D Case

Firstly, 3D print your knife block shell.

We made the 3D design using Fusion360. To be honest, it was quite a fiddly and time consuming process. If you’d like a tutorial on how to do this, please let us know in the comments below. We’re still learning, too, so if anyone has any tips on the design or good places to learn more about 3D design, please share.

Electronics

We are going to put together following circuit:

Solder Wires Onto the Solar Panel

Take two wires about 10cm long and solder one onto the positive and one onto the negative tab on the solar panel.

Connect the Battery

Put the battery into the holder and solder the positive and negative wires to the B+ and B- inputs on the charge controller.

Connect the Switch and Motor

From the push button terminals, solder one wire from the positive output of the charge controller to the input of the push button. Solder another wire from the output of the push button to the positive of the motor. Solder a wire from the negative output of the charge controller to the negative of the battery.

Check the connections work and note which way the motor turns — you want to put it in the case so it will rotate away from you.

Put in the Motor

Slot the motor into the hole in the knife block. To reduce vibration and help keep the motor in place, you could glue it using a glue gun — this is optional, though, as it fits snugly.

Push the grinding stone onto the end of the motor.

Put on Push Button

Put the push button through the hole in the lid and glue in place.

Put on the Solar Panel

Solder the positive wire from the solar panel to the positive input on the charge controller. Solder the negative wire from the solar panel to the negative input on the charge controller.

Glue around the perimeter of the case and push on the solar panel.

Fill Up With Spaghetti

Fill the large internal cavity with spaghetti. This might sound random, but it gives the block some weight so it doesn’t move around when sharpening and it also helps keep the knives in place.

We used about 3 packs of spaghetti to fill the space. It was a bit long so we trimmed the ends so the lid would fit.

Put on Lid

Screw the lid in place to close the knife block.

Put in Knives

Put your knives into your block and swirl it round in the sun.

We did think that adding little rubber feet would be a good way to reduce vibration, noise and slipperiness, but we didn’t quite get round to that. We also thought that we could add a bottle opener or an electric can opener to make this the ultimate kitchen gadget!

If you have any suggestions for improvements or additions, please let us know in the comments below!

Thanks for reading

I hope you have enjoyed this article. If you like the style, check out T3chFlicks.org for more tech focused educational content (YouTube, Instagram, Facebook, Twitter).

🚢 Home DevOps Pipeline: A junior engineer’s tale (4/4)

t3chflicks — Mon, 15 Mar 2021 15:14:50 +0000

In this series of articles, I will explain how I built my own home development environment using a couple of Raspberry Pis and a lot of software. The code referenced in these articles can be found here. In this article I will cover the monitoring of my pipeline, as well as discuss a few gotchas.

Monitoring

My initial thought for monitoring uptime of my development pipeline was UpTimeRobot, a free service which pings periodically and if failures alerts via email.

Example email*

However, being able to monitor and repair your development pipeline in a web app is just great and portainer have really aced this one.

This application is great and many more features than I want to explain. But it allows for inspection of container logs, ssh into containers, and even deployment of new images — controlling and monitoring multiple hosts.

Portainer running*

To get portainer running I added this to my code docker-compose template:

portainer:

  image: portainer/portainer

  command: -H unix:///var/run/docker.sock

  restart: always

  networks:


homereponet
ports:
9000:9000
8000:9000
volumes:
/var/run/docker.sock:/var/run/docker.sock
/mnt/hdd/portainer:/data
labels:
"traefik.enable=true"
traefik.backend=portainer
traefik.portainer.frontend.rule=Host:portainer.<domain>
traefik.docker.network=homereponet
traefik.portainer.port=9000

The End

This is where this story ends for now. I think that this is a great starter set up for anyone looking to have a industry style development pipeline. As a recap this final setup is:

Gitea Repository for storing code, using git LFS for large media files. Drone CI to build containers, store them in Docker Registry, and deploy using an Ansible plugin to a local set of machines as well as cloudformation plugin for AWS deployment. Portainer for monitoring and control over containers in a UI. Traefik for a reverse proxy with let’s encrypt free https certificate on a free duckdns domain. All storage is done on a hard drive and there are periodic backups to AWS s3.

I hope you have enjoyed this series of articles. I also make many more projects including other hardware, 3D printing, smart-home applications, and machine learning in video form over at T3chFlicks — check us out.

Appendix

Code: https://github.com/sk-t3ch/home-repo

Random Things I learned along the way:

Docker-compose has to be controlled inside the same directory as it was started
policy creation for aws s3 is needed because you only want the pi and the root aws user to be able to access the bucket also add encryption
need to look more into backup techniques
s3 backups to glacier after 10 days meaning storage cost stays low
Raspberry Pi 4 has two usb 3.0 ports and two 2.0 ones.
Using an EXFAT drive instead of HFS+ resulted in failures for the Postgres container
ssh on Gitea cannot use port 22 due to permissions problem that i am yet to solve. to work around this i changed the router forward port to allow for an ssh port in the accepted range (1065+)

Contents: 1, 2, 3, (4)

Thanks for reading

I hope you have enjoyed this article. If you like the style, check out T3chFlicks.org for more tech-focused educational content (YouTube, Instagram, Facebook, Twitter).

🚢 Home DevOps Pipeline: A junior engineer’s tale (3/4)

t3chflicks — Mon, 15 Mar 2021 15:14:16 +0000

Deployment

Ansible

Another great thing about Drone is the plugins available — we will use the Ansible plugin as a tool to run ssh commands on multiple machines. We will as well make use of Drone secrets as a method of securely deploying certain configuration values.

The general methods of deployment I have seen usually consist of publishing to AWS and then magic happens using cloud formation templating and your stack is created.

However, we want to keep the home in home repo. To do this, we want to deploy to specific machines on our local network. For me, this is a load of other Raspberry Pis which do many things, from control my 3D printer, to work as a network of CCTV cameras.

To do this we will be using Ansible like this:

    kind: pipeline
    name: commitPipeline

    trigger:
      event:
         - push

    platform:
      os: linux
      arch: arm

    steps:
      - name: trigger-rebuild-of-image
        image: plugins/ansible:1
        environment:
          repo_username:
            from_secret: repo_username
          repo_password:
            from_secret: repo_password
        settings:
          private_key:
            from_secret: ansible_private_key
          playbook: ansible/playbook.yml
          inventory: ansible/inventory.yml

The actions Ansible performs are found in the playbook.yml and the machines we want to perform on are described in
inventory.yml :


    - hosts: cameras

    tasks:

    - name: .ssh directory
      file:
        path: ~/.ssh
        owner: pi
        group: pi
        state: directory

    - name: put key in
      copy:
        content: "{{ lookup('env','ansible_private_key') }}"
        dest: ~/.ssh/deploy_key
        owner: pi
        group: pi
        mode: 0600
        become: pi

    - name: clone the repo
      git:  
        repo: ssh://git@git.<domain>/<username>/cctv.git
        dest: /home/pi/cctv
        key_file: ~/.ssh/deploy_key
        accept_hostkey: yes
        force: yes

    - name: Rebuild docker image
       shell:
       cmd: docker-compose up --remove-orphans --force-recreate --build -d
      chdir: /home/pi/cctv

    - name: remove private key
       shell:
       cmd: rm ~/.ssh/deploy_key

;

    all:
      hosts:
        drone.<domain>:
      children:
        cameras:
          hosts:
            192.168.0.41:
            192.168.0.42:
          vars:
            ansible_ssh_user: pi

Thanks to ansible my deployment process for updates to my CCTV cameras has been greatly simplified.

Deploying to AWS

Deploying at home is fun and great n all, but the training wheels must come off. Developing in the real world involves the cloud, and at the company I work for, this means AWS.

Deploying your service to an AWS EC2 machine should have a testing step too, however the most common architecture (x86) is very different to the arm architecture on Raspberry Pi. We shouldn’t test our builds using an arm machine if they will be deployed to an x86 machine. We must deploy an x86 Drone runner.

However, the plot twists again and in order to deploy a drone x86 runner using our environment, we need to use the Drone/cloud formation image which is only built for x86, not arm. But we can rebuild it ourselves like I have demonstrated in my code. Like a good boy, I published the image on dockerhub for others to use, too.

Photo by Andreas Wagner on Unsplash*

Using this we can now deploy an AWS machine like this:

    kind: pipeline
    name: commitPipeline

    trigger:
      event:
        - push

    platform:
      os: linux
      arch: arm

    steps:
      - name: validate-seeder-infrastructure
        image: registry.<domain>/aws-cfn
        settings:
          mode: validate
          template: infrastructure.yml
        environment:
          AWS_ACCESS_KEY_ID:
            from_secret: aws_id
          AWS_SECRET_ACCESS_KEY:
            from_secret: aws_key

Once this is deployed, we can run x86 builds with it by either specifying the architecture in the .drone.yml or by removing any architecture declaration because x86 is the default.

I have also rebuilt an AWS-CLI image here.

We have now covered the deployment part of the pipeline and in the next article in this series we will cover the monitoring methods and discuss a few gotchas. Check it out here.

Contents:
1,
2,
(3),
4

Thanks for reading

I hope you have enjoyed this article. If you like the style, check out T3chFlicks.org for more tech-focused educational content (YouTube, Instagram, Facebook, Twitter).

🚢 Home DevOps Pipeline: A junior engineer’s tale (2/4)

t3chflicks — Mon, 15 Mar 2021 15:12:06 +0000

In this series of articles, I will explain how I built my own home development environment using a couple of Raspberry Pis and a lot of software. In this article I will cover the development / CI part of the pipeline, starting with the Git Repository. The code referenced in these articles can be found here.

Gitea

Gitea is a git repository application that can be installed on a machine. This means that the storage of the repositories is done on the machine itself as standard. However, as mistakes do happen, I have upgraded this to an external hard drive with routine backups to AWS S3.

There are major benefits to hosting your own repo at home. If like me, you often edit videos or batches of large images, then downloading stuff from the internet with cheap WiFi is slow (10mb/s) and uploading is just plain daft (500kb/s). Local network storage, however, allows much higher transfer speeds (to be measured).

You may be thinking large files and git doesn’t make any sense. You are absolutely right. Any edits involving large binaries cause your git repo to explode in size. Enter stage left to solve all git/large file problems: the Git LFS (Large File Storage) — a git plugin that stores specified files in a sensible way.

“Git Large File Storage (LFS) replaces large files such as audio samples, videos, datasets, and graphics with text pointers inside Git, while storing the file contents on a remote server like GitHub.com or GitHub Enterprise.”

Git LFS means I have all the benefits of using the beloved git with my necessary large files and enjoy those high speed transfers of local network! Hooray!

Photo by Alexander Sinn on Unsplash*

I’ve introduced Gitea’s use case without giving you much of an explanation of what it is and offers. For that, you should take a look at this comparison between Gitea and other Git repository providers.

To create your own gitea application, use docker-compose to run this template which has a gitea and postgres image (production ready db):

version: “2”

networks:
  appnet:
    external: false

volumes:
  gitea-app:
  gitea-db:

services:

gitea-app:
    image: webhippie/gitea
    env_file:
      - gitea.env
    container_name: gitea-app
    restart: always
    networks
      - appnet
    volumes:
      - ./volumes/gitea_data:/data
      - ./custom:/data/gitea
    ports:
      - "222:22"
      - "3000:3000"
     depends_on:
      - gitea-db

gitea-db:
    image: postgres:alpine
    container_name: gitea-db
    ports:
      - 5432:5432
    restart: always
    volumes:
      - ./volumes/gitea_db:/var/lib/postgresql/data
    environment:
      - POSTGRES_USER=gitea
      - POSTGRES_PASSWORD=gitea
      - POSTGRES_DB=gite
    networks:
      - appnet

In this file, the gitea service references a file gitea.env. This is the configuration file for gitea, which allows us to set up this home repo in such a way that we don’t need to any set up inside the application itself — meaning we can run straight from a backup.

After running docker-compose up , you should be able to access this on your local network under raspberrypi.local:3000 and use it. Happy days.

Photo by Harrison Broadbent on Unsplash*

External Hard Drive

With gitea up and running, I want to add a larger storage device than the Micro SD card. To do so, we need only mount a hard drive like this:

sudo mount /dev/sda1 /media/hdd

Once mounted, you can reference the location /media/hdd just like you would any other folder. Now we want to be able to routinely backup the contents of this drive. Handily, someone has made a docker image which does just that https://github.com/istepanov/docker-backup-to-s3. However, as expected this image isn’t for ARM architecture so I rebuilt the image using a Raspbian base on my Pi.

Adding this to our current docker-compose.yml like:

services:
  backup:
    image: t3chflicks/backuper
    environment:
      - ACCESS_KEY=
      - SECRET_KEY=
      - S3_PATH=
      - CRON_SCHEDULE=0 12 * * *
    volumes:
      - /media/hd:/data:ro
    container_name: backup
    restart: always

gitea-app:
    image: webhippie/gitea
    env_file:
      - gitea.env
    container_name: gitea-app
    restart: always
    networks
      - appnet
    volumes:
      - /media/hd/gitea_data:/data
    ports:
      - "222:22"
      - "3000:3000"
    depends_on:
      - gitea-db

After enabling Git LFS in your Gitea Config, we can also install the git plugin on our repo as simple as git lfs install and subsequently tagging files like this: git lfs track "*.mp4".

Now you can use your git repo to work with large media files and that is awesome.

Traefik

Having a home repository is great. But, since I am not a complete hermit and want to collaborate with others, I then went on to add external access to my repo. I did this using Traefik, the reverse proxy which gives automatic SSL encryption with certificates from Let’s Encrypt and I used a free domain from duckdns.

I found the Traefik docs a little bit confusing but it turned out to be super simple to add into the current docker-compose.

version: “2”
networks:
  appnet:
    external: false
  homereponet:
    external: true

volumes:
  gitea-app:
  gitea-db:
  traefik:

services:
  gitea-app:
    image: webhippie/gitea
    env_file:
      - gitea.env
    container_name: gitea-app
    restart: always
    networks
      - appnet
    volumes:
      - /media/hdd/gitea_data:/data
    ports:
      - "222:22"
      - "3000:3000"
    depends_on:
      - gitea-db
  depends_on:
    - gitea-db
    - traefik
  labels:
    - "traefik.enable=true"
    - traefik.backend=gitea-app
    - traefik.git.frontend.rule=Host:git.<your_domain>
    - traefik.docker.network=homereponet
    - traefik.git.port=3000

gitea-db:
    image: postgres:alpine
    container_name: gitea-db
    ports:
      - 5432:5432
    restart: always
    volumes:
      - /media/hdd/gitea_db:/var/lib/postgresql/data
    environment:
      - POSTGRES_USER=gitea
      - POSTGRES_PASSWORD=gitea
      - POSTGRES_DB=gitea
    networks:
     - appnet

  traefik:
    image: traefik:1.7-alpine
    container_name: “traefik”
    ports:
      - “80:80”
      - “443:443”
    networks:
      - homereponet
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik/traefik.toml:/traefik.toml 
      - ./traefik/acme:/etc/traefik/acme
    restart: always

And this requires some extra steps of creating the acme folder and editing permissions as well as adding this traefik.toml file

debug = false
logLevel = “ERROR”
defaultEntryPoints = [“https”,”http”]

[entryPoints]

[entryPoints.http]
address = “:80”
[entryPoints.http.redirect]
entryPoint = “https”
[entryPoints.https]
address = “:443”
[entryPoints.https.tls]

[retry]

[docker]
endpoint = “unix:///var/run/docker.sock”
domain = “<your domain>”
watch = true
exposedByDefault = false

[acme]
email = “<your email>”
caServer = “https://acme-v02.api.letsencrypt.org/directory"
storage = “acme.json”
entryPoint = “https”
onHostRule = true

[acme.httpChallenge]
entryPoint = “http”

But now as long as you have exposed port 443 and forwarded the traffic to your Pi, you should be able to access your home repository via your duckdns address and it will encrypted on https. How great!

Alt Tea Theme (Dark Mode)*

Drone

We now have our own private Git repo setup in a Docker environment with Postgres, Git LFS, external hard drive storage and external SSL encrypted access. This is great: we can work happily on our project, knowing that we can easily work with large files and work with git from anywhere.

However, this isn’t exactly DevOps. Using git provides you with the continuous integration of your code via pull requests etc., but it does not provide continuous delivery. For that, we look to a software called Drone.

Photo by Iewek Gnos on Unsplash*

We’ll be using Drone to manage pipelines in which we build Docker containers, test and deploy to machines.

Like all the other services discussed so far, Drone has an image on DockerHub. To add this to our current setup, we need to add the following to our docker-compose file:

services:
  drone:
    image: drone/drone:1-linux-arm
    container_name: drone
    volumes:
      - /media/hdd/drone:/var/lib/drone/
    restart: always
    depends_on:
      - gitea-app
    environment:
      - DRONE_OPEN=true
      - DRONE_GITEA_CLIENT_ID=<gitea_auth_id>
      - DRONE_GITEA_CLIENT_SECRET=<gitea_auth_secret>
      - DRONE_GITEA_SERVER=http://git.<your_domain>
      - DRONE_SERVER_HOST=drone.<your_domain>
      - DRONE_SERVER_PROTO=https
      - DRONE_TLS_AUTOCERT=false
      - DRONE_RPC_SECRET=<drone_runner_secret>
      - DRONE_AGENTS_ENABLED=true
    networks:
      - homereponet
      - appnet
    labels:
      - “traefik.enable=true”
      - traefik.backend=drone
      - traefik.drone.frontend.rule=Host:drone.<your_domain>
      - traefik.drone.port=80
      - traefik.docker.network=homereponet

runner:
    container_name: runner
    image: drone/drone-runner-docker:1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - drone
    environment:
      - DRONE_RPC_HOST=drone.<your_domain>
      - DRONE_RPC_PROTO=https
      - DRONE_RPC_SECRET=<drone_runner_secret>
    restart: always
    ports:
      - “3000:3000”
    networks:
      - appnet
      - homereponet

Now, when starting up the system you and visiting https://drone.<domain>you will get an error when trying to login. This is because you need to have Gitea running and create the OAuth application first and then rebuild your containers, setting DRONE_GITEA_CLIENT_SECRET to the new value — this is described here.

You will now be able to login to Drone using your Gitea credentials. Following this, you should sync the repositories of your Gitea by clicking the big button. The next thing you’ll want to try is doing a build, for this you should add a .drone.yml file to your repo and on push, it should trigger a build.

Each step is a build in an isolated environment and executed on the Drone-runner. Drone-runners build for one architecture so we will only build Raspberry Pi (arm v7) images on a Pi runner but for x86 builds we will need to deploy an EC2 on AWS (more about that later). My example drone file looks like this:

kind: pipeline
name: commitPipeline

trigger:
  event:
    - push

platform:
  os: linux
  arch: arm

steps:
  - name: buildsomething
    image: python:alpine
    commands:
      - echo ‘hello world’

Voila, you are now the proud owner of a home repo! We’ll ignore the fact that it probably still can’t do as much as Gitlab out of the box, but it is yours (woo, capitalism!).

Registry

DockerHub is great, just like Github is great. This is because they provide a place for people to share their work. However, just as the argument for the private repo is not wanting to share and for fast local speed, the same applies to Docker registries.

We will be creating our own place to store the images we’ve made using Docker’s own registry image. To add this into our existing Docker-compose file, we just need to add this extra service:

registry:
  image: registry:2
  container_name: registry
  restart: always
  networks:
    - homerepone
  volumes:
    - ./volumes/registry:/var/lib/registr
    - ./auth:/auth
  labels:
    - “traefik.enable=true”
    - traefik.registry.frontend.rule=Host:registry.<your_domain>
    - traefik.backend=registry
    - traefik.docker.network=homereponet
    - traefik.registry.port=5000
  environment:
    - REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/data
    - REGISTRY_AUTH=htpasswd
    - REGISTRY_AUTH_HTPASSWD_REALM=Registry
    - REGISTRY_AUTH_HTPASSWD_PATH=/auth/registry.password

This references an auth/registry.password file which we create using apache2-utils bcrypt hash password like described here.

Running our new docker-compose.yml means you can access the registry by logging in using the docker cli using docker login [https://registry.<](https://registry.homerepo4.duckdns.org)your_domain> .

We have now covered the development part of the pipeline and in the next article in this series we will cover the deployment methods. Check it out here.

Contents: 1, (2), 3, 4

Thanks for reading

I hope you have enjoyed this article. If you like the style, check out T3chFlicks.org for more tech-focused educational content (YouTube, Instagram, Facebook, Twitter).

🚢 Home DevOps Pipeline: A junior engineer’s tale (1/4)

t3chflicks — Mon, 15 Mar 2021 15:11:28 +0000

I’ve been working as a software engineer for just over a year whilst simultaneously working on many of my own side projects such as T3chFlicks (an edutainment channel). During this period, I have picked up many different technologies & skills, however one role I managed to stay away from until the past couple of months is DevOps.

In this series of articles, I will explain how I applied this new-found knowledge by building my own home development environment using a couple of Raspberry Pis.

This environment allows me to do things like:

Host my own reliable Git Repo (Gitea)
Effortlessly work with large media files using git (LFS)
Access my Git Repo remotely with SSL (Traefik)
Work with docker containers
Build, test and deploy with a git push (Drone)
Monitor all parts of the system (Portainer)
Deploy builds to groups of devices locally and on the internet (Ansible)

I am still inexperienced with this tech and am more big ideas than skill, but I’ve come a long way since I first began and I want to share my journey and process. I do this to learn.

The code referenced in these articles can be found here.

Home Repo

To follow the rest of these blog posts, you must have a basic knowledge of Docker and Git, so let’s take a brief detour…

Docker

My introduction to DevOps was after I built a service which was going to be deployed on an AWS ECS (Elastic Container Service) cluster. This service needed to be “dockerised”. This means it needed to be created within a reliably recreatable environment known as a container.

Docker is the containerisation software which enables us to do the above by building environments with a YAML template known as a Dockerfile. We can build our environments on top of base images such as Ubuntu e.g.

**FROM** ubuntu:18.04
**RUN** apk update

and you can run it like this:

`docker build -t myFirstDockerContainer . && docker run myFirstDockerContainer`

The Docker CLI enables you to control containers and push and pull images from registries such as DockerHub. However, you can use it to control a lot more, including networks, volumes, and even collections of containers working together as part of a swarm.

In my opinion, Docker is great. The main reason for this is that I spend less time debugging… I hate debugging. Software engineering to me is all about creating and I shouldn’t spend my time being a software mechanic.

Go ahead and explore DockerHub. If there isn’t an image for your favourite software, why not make it and publish it (preferably using alpine — the smallest base image).

Docker-Compose

After you’ve explored Docker, you’ll soon want to connect your separate services. For example, if you want to run a web app with a database. To set this up in a single file there exists docker-compose.

version: "3"

networks:
  someNetwork:

volumes:
  someVolume:

services:
  webapp:
    image: someWebAppImage

database:
    image: someDatabaseImage

Docker-compose is elegant, simple and a big upgrade to my developing practice.

Photo by chuttersnap on Unsplash*

Git

For me, git is a must, even when you are working on your own projects at home. It’s just so damn useful. I can’t tell you how many times I have gone through git log to find that magical piece of code that actually works. Hammer! Hammer! Hammer! (one day i will get good enough at TDD).

Photo by Sean Stratton on Unsplash*

The way you fell in love with git is probably by putting your code online onto a website such as Github and collaborating with others on it. You probably then went on to think that what you made was so amazing that you thought someone else might come along and steal it and you’d never get to be the next Facebook. So you decided to make it a private repository.

Gitlab is an open source alternative to Github. It’s also pretty great and offers loads of awesome features for free. An alternative to a managed service is to host your own git repository server, such as the one Gitlab provide (the software requires 4GB of RAM for 1000 concurrent users), which the Raspberry Pi 4 contains, but I’d rather run smaller and separate services. Instead, I opted for Gitea.

Right. Slight detour over, now onto the proper stuff — check out the second article in this series here.

Contents: (1), 2, 3, 4

Thanks for reading

I hope you have enjoyed this article. If you like the style, check out T3chFlicks.org for more tech-focused educational content (YouTube, Instagram, Facebook, Twitter).

☁️ 🚀 Cheaper than API Gateway — ALB with Lambda using CloudFormation

t3chflicks — Mon, 15 Mar 2021 15:07:03 +0000

An alternative to API gateway is Application Load Balancer. ALB can be connected with Lambda to produce a highly performant, cost effective API. In this article, I demonstrate how to create an API using CloudFormation and Python.

API Gateway vs. Application Load Balancer

API Gateway and ALB are two different AWS services, however they can both be used to achieve the same thing: send network requests for a service to the service.

API Gateway is pay per request whereas ALBs have an hourly rate, therefore deciding which to use depends on traffic volume. For an in-depth price and feature comparison, I recommend this article, which shares this shocking statistic:

I expect to pay around $166 per month for ALB, whereas I’m paying $4,163 per month for the exact same service from API Gateway.

This is a massive saving! That said, the cost difference isn’t unmerited and you do get extra features from API gateway. Dougal Ballantyne, the Head of Product for Amazon API Gateway tweeted:

If you are building an API and want to leverage AuthN/Z, request validation, rate limiting, SDK generation, direct AWS service backend, use #APIGateway. If you want to add Lambda to an existing web app behind ALB you can now just add it to the needed route

Well, that’s exactly what we’re gonna do!

API with ALB and Lambda

I am going to build the following the system:

Architecture Diagram*

The complete CloudFormation templates can be found here, split into two templates:

vpc.yml — Configures the VPC. For simplicity, the VPC only contains two public subnets. You can read about a more complex VPC in our previous article.
service.yml— Configures both the ALB and Lambda function.

Lambda

To create a Lambda using CloudFormation, it is necessary to define a Lambda Function, a Role to run the Lambda and a Permission for the ALB to invoke the Lambda:

Lambda:
    Type: AWS::Lambda::Function
    Properties:
    Code:
        ZipFile: |
        def handler(event, context):
        response = {
            'isBase64Encoded': False,
            'statusCode': 200,
            'body': 'HELLO WORLD!'
        }
        return response
    Handler: lambda_function.handler
    Role: !GetAtt LambdaRole.Arn
    Runtime: python3.7

LambdaRole:
    Type: AWS::IAM::Role
    Properties:
    AssumeRolePolicyDocument:
        Statement:
        - Action: ['sts:AssumeRole']
            Effect: Allow
            Principal:
            Service: ['lambda.amazonaws.com']
    Path: /
    ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

LambdaFunctionPermission:
    Type: AWS::Lambda::Permission
    Properties:
    FunctionName: !GetAtt Lambda.Arn
    Action: 'lambda:InvokeFunction'
    Principal: elasticloadbalancing.amazonaws.com

Application Load Balancer

The Load Balancer is placed across public subnets as it needs to be accessible from the internet. The ALB is configured to listen to HTTP traffic on port 80 and forward it to the Lambda:

LoadBalancerSecGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
    GroupDescription: Load balance allow port 80 traffic
    VpcId: !ImportValue VPCID
    SecurityGroupIngress:
        CidrIp: 0.0.0.0/0
        FromPort: 80
        IpProtocol: TCP
        ToPort: 80

LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
    SecurityGroups:
        - !Ref LoadBalancerSecGroup
    Subnets:
        - !ImportValue PublicSubnetA
        - !ImportValue PublicSubnetB

LoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
    Protocol: HTTP
    LoadBalancerArn: !Ref LoadBalancer
    DefaultActions:
        - Type: forward
        TargetGroupArn: !Ref LoadBalancerTargetGroup
    Port: 80

LoadBalancerTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
    TargetType: lambda
    Targets:
        - AvailabilityZone: all
        Id: !GetAtt Lambda.Arn

The complete code can be accessed here and can be deployed using the AWS CLI:

aws cloudformation create-stack --stack-name service --template-body file://template.yml --capabilities CAPABILITY_NAMED_IAM

After a successful deployment, the DNS name of the ALB can be found in the EC2 section of the AWS console. It should look something like:

loadb-LoadB-R7RVQD09YC9O-1401336014.eu-west-1.elb.amazonaws.com

Now it is possible to make a request to this URL and get a response:

$ curl loadb-LoadB-R7RVQD09YC9O-1401336014.eu-west-1.elb.amazonaws.com'

HELLO WORLD!

Photo by Nghia Le on Unsplash*

Cross Origin Response Sharing

In order to access this service from a browser webpage on a different domain, CORS must be enabled. This is done by setting headers:

Lambda:
    Type: AWS::Lambda::Function
    Properties:
    Code:
        ZipFile: |
        def handler(event, context):
        response = {
            'isBase64Encoded': False,
            'statusCode': 200,
            'body': 'HELLO WORLD!',
            'headers': {
                'access-control-allow-methods': 'GET',
                'access-control-allow-origin': '*',
                'access-control-allow-headers': 'Content-Type, Access-Control-Allow-Headers'
            }
        }
        return response
    Handler: lambda_function.handler
    Role: !GetAtt LambdaRole.Arn
    Runtime: python3.7

Use a Domain Name

AWS provides an ugly Load Balancer address such as:

loadb-LoadB-R7RVQD09YC9O-1401336014.eu-west-1.elb.amazonaws.com

But it’s quite simple to use a custom domain using AWS. Firstly, transfer your DNS management to Route 53 and then create a new record set aliased to the load balancer.

Thanks For Reading

I hope you have enjoyed this article. If you like the style, check out T3chFlicks.org for more tech focused educational content (YouTube, Instagram, Facebook, Twitter).

Resources:

🚫 Users Only: Quickstart for creating a SaaS pt. 1 — User Management

t3chflicks — Mon, 15 Mar 2021 15:06:36 +0000

Managing users and API keys is a necessary task for creating a Software as a Service (SaaS). In this article, I demonstrate how to create a simple, cost effective serverless SaaS user management application. The frontend is created using Vue JS and the Amplify plugin. The backend uses Cognito for authentication of the user management API, and a key system created with DynamoDB which authorises users to access a test API created with Application Load Balancer. Check out the live demo 💽.

Not everything is intended for everyone. In a scenario where restricting access is necessary, user management must become a component of the architecture. The AWS solution to user management is the Cognito service. This integrates simply with API Gateway — but as described in the previous article, API gateway can get pretty expensive, and for high load the more cost effective alternative is Application Load Balancer.
Cheaper than API Gateway — ALB with Lambda using CloudFormation
*An alternative to API gateway is Application Load Balancer. ALB can be connected with Lambda to produce a highly…*medium.com

The architecture for this user management application (see below) makes use of both API gateway and ALB for their respective benefits. API gateway is chosen for the User API for two reasons:

the requests are likely to be low in volume meaning costs will be low
the requests are direct from the frontend with authentication using the Amplify package with the Vue JS framework

ALB is chosen for the example service Test API as it is expected to experience a high volume of requests which might rack up a hefty bill on API Gateway. Also, the requests are likely to be from other servers meaning API keys are preferable. Users are generated an API key on sign up which is used to authenticate the Test API. Usage of the API is monitored by incrementing a count on the User table each time a request is made to the Test API.

User Manager Quickstart Architecture Diagram*

This architecture is cost effective for high-load APIs. A quote from this article might help you decide if your service falls into that category:

$15 (ALB’s cost) worth of API Gateway calls will net you around 4.3 million API calls in the month (well, 5.3 million if you count the free tier). So, if your API is small enough to fit under that number of calls, stick with API Gateway — it’s simpler to use. But, 5.3 million API calls is only around two requests per second. So, if your API is used very much at all — or even if you have DNS health checks enabled — you could easily end up paying more for API Gateway than you would for Application Load Balancer.

I presume this article will not bring in more than two reads per second, meaning an ALB would not be suitable for running the demo of the service. Instead, the test API seen in the diagram has been replaced with another API Gateway. You can see a demonstration of the app in use below.

The full code for this project can be found here ☁️

The live demo can be found here 💽

Video of Use Manager App*

Let’s Build! 🔩

Backend

The infrastructure for this system is written as code using the CloudFormation framework.

VPC

ALBs must be placed within a Virtual Private Cloud on AWS. The service uses a stripped down VPC consisting of only two public subnets and an Internet Gateway. A more complete VPC is described in a previous article:
Virtual Private Cloud on AWS — Quickstart with CloudFormation
*A Virtual Private Cloud is the foundation from which to build a new system. In this article, I demonstrate how to…*medium.com

User Management with Cognito

The AWS Cognito service is used to manage users. Users are stored in user pools, and Clients (mobile or web apps) can interact with them using an API. I configured the Cognito user pool to send an email with a verification code on sign up.

CognitoUserPoolClient:
    Type: AWS::Cognito::UserPoolClient
    Properties:
    ClientName: CognitoUserPoolClient
    UserPoolId: !Ref CognitoUserPool
    AllowedOAuthFlows:
    - implicit
    AllowedOAuthFlowsUserPoolClient: true
    AllowedOAuthScopes:
    - email
    - openid
    LogoutURLs:
    - !Sub 
        - https://um-app.${ Domain }
        - Domain: !ImportValue UserManagerApp-RootDomain
    CallbackURLs:
    - !Sub 
        - https://um-app.${ Domain }
        - Domain: !ImportValue UserManagerApp-RootDomain
    SupportedIdentityProviders:
    - COGNITO

You must also define a client; this allows for OAuth on our frontend client.

NB: Setting the URLs to localhost allows for local development.

CognitoUserPoolClient:
    Type: AWS::Cognito::UserPoolClient
    Properties:
    ClientName: CognitoUserPoolClient
    UserPoolId: !Ref CognitoUserPool
    AllowedOAuthFlows:
    - implicit
    AllowedOAuthFlowsUserPoolClient: true
    AllowedOAuthScopes:
    - email
    - openid
    LogoutURLs:
    - !Sub 
        - https://um-app.${ Domain }
        - Domain: !ImportValue UserManagerApp-RootDomain
    CallbackURLs:
    - !Sub 
        - https://um-app.${ Domain }
        - Domain: !ImportValue UserManagerApp-RootDomain
    SupportedIdentityProviders:
    - COGNITO

Cognito user pools offer useful event tiggers such as sign up confirmation. I have used the post-confirmation event to trigger a Lambda function which writes the user’s username, a newly generated API key, and a zero initialised counter to a DynamoDB table. The table has a Global Secondary Index (GSI) on the Key attribute which allows for a user lookup with just the Key.

UserTable: 
    Type: AWS::DynamoDB::Table
    Properties: 
    AttributeDefinitions: 
        - AttributeName: "Id"
        AttributeType: "S"
        - AttributeName: "Key"
        AttributeType: "S"
    KeySchema: 
        - AttributeName: "Id"
        KeyType: "HASH"
    ProvisionedThroughput: 
        ReadCapacityUnits: 1
        WriteCapacityUnits: 1
    GlobalSecondaryIndexes: 
        - IndexName: "KeyLookup"
        KeySchema: 
            - AttributeName: "Key"
            KeyType: "HASH"
        Projection: 
            ProjectionType: "ALL"
        ProvisionedThroughput: 
            ReadCapacityUnits: 1
            WriteCapacityUnits: 1

The PostConfirmation Lambda function template:

PostConfirmationLambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
    CodeUri: ./PostConfirmation
    Handler: main.handler
    MemorySize: 128
    Runtime: python3.8
    Timeout: 60
    Role: !GetAtt PostConfirmationLambdaRole.Arn
    Environment:
        Variables:
        TABLE_NAME: !Ref UserTable

The PostConfirmation Lambda is a Python function which creates an API key and stores it in the DynamoDB table:

import boto3
import os

dynamodb = boto3.resource('dynamodb')
TABLE_NAME = os.environ['TABLE_NAME']
table = dynamodb.Table(TABLE_NAME)

def handler(event, context):
try:
    username = event["userName"]
    table.put_item(
        Item={
                'Id': username,
                'Key': os.urandom(64).hex(),
                'Count': 0
        }
        )
except Exception as err:
    print("ERR: ", err)
return event

User API

Endpoints created for users to access their profile and generate an API key are authenticated for the logged in user. Authentication is easily added to API gateway with Cognito (CORS are set open for development purposes).

UserApi:
    Type: 'AWS::ApiGateway::RestApi'
    Properties:
    Body:
        info:
        version: '1.0'
        title: !Ref 'AWS::StackName'
        paths:
        /user:
            options:
            x-amazon-apigateway-integration:
                type: mock
                requestTemplates:
                application/json: |
                    {
                    "statusCode" : 200
                    }
                responses:
                default:
                    statusCode: '200'
                    responseTemplates:
                    application/json: |
                        {}
                    responseParameters:
                    method.response.header.Access-Control-Allow-Origin: '''*'''
                    method.response.header.Access-Control-Allow-Methods: '''*'''
                    method.response.header.Access-Control-Allow-Headers: '''*'''
            consumes:
                - application/json
            summary: CORS support
            responses:
                '200':
                headers:
                    Access-Control-Allow-Origin:
                    type: string
                    Access-Control-Allow-Headers:
                    type: string
                    Access-Control-Allow-Methods:
                    type: string
                description: Default response for CORS method
            produces:
                - application/json
            get:
            x-amazon-apigateway-integration:
                httpMethod: GET
                type: aws_proxy
                uri: !Sub >-
                arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${GetUserFunction.Arn}/invocations
            security:
                - CognitoUserPoolAuthorizer: []
            responses: {}
        swagger: '2.0'
        securityDefinitions:
        CognitoUserPoolAuthorizer:
            in: header
            type: apiKey
            name: Authorization
            x-amazon-apigateway-authorizer:
            providerARNs:
                - !ImportValue UserManagerApp-CognitoUserPoolArn
            type: cognito_user_pools
            x-amazon-apigateway-authtype: cognito_user_pools

The code for getting and generating keys is very similar to the post-confirmation Lambda code.

Service API — Test

The API I’ve created with ALB is for the high traffic volume endpoints which are authenticated using an API key. This API key is sent in a POST request to the service.

LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
    SecurityGroups:
        - !Ref LoadBalancerSecGroup
    Subnets:
        - !ImportValue UserManagerApp-PublicSubnetA
        - !ImportValue UserManagerApp-PublicSubnetB

LoadBalancerTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
    TargetType: lambda
    Targets:
        - AvailabilityZone: all
        Id: !GetAtt Lambda.Arn

LoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    DependsOn:
    - LambdaFunctionPermission
    Properties:
    LoadBalancerArn: !Ref LoadBalancer
    DefaultActions:
        - Type: forward
        TargetGroupArn: !Ref LoadBalancerTargetGroup
    Port: 443
    Certificates:
        - CertificateArn: !ImportValue UserManagerApp-RegionalCertArn
    Protocol: HTTPS

Lambda:
    Type: AWS::Lambda::Function
    Properties:
    Code:
        S3Bucket: !ImportValue UserManagerApp-CodeBucketName
        S3Key: TestKey.zip
    Description: Test Service function
    Handler: main.handler
    MemorySize: 256
    Role: !GetAtt LambdaRole.Arn
    Runtime: python3.8
    Timeout: 60
    Environment:
        Variables:
        TABLE_NAME: !ImportValue  UserManagerApp-UserTableName

The service uses the user’s API key for a query on a GSI of the user table to get the username and Count, then increment the Count and return the updated Count. Again, I have added open CORS whilst developing. For security, these should been locked down.

import boto3
from boto3.dynamodb.conditions import Key
import os
import time
import json
from decimal import Decimal


dynamodb = boto3.resource('dynamodb')
TABLE_NAME = os.environ['TABLE_NAME']
table = dynamodb.Table(TABLE_NAME)

def handler(event, context):
response = {
    "isBase64Encoded": False,
    "headers": {
        'access-control-allow-methods': 'POST, OPTIONS',
        'access-control-allow-origin': '*',
        'access-control-allow-headers': 'Content-Type, Access-Control-Allow-Headers'
    }
}

try:
    if event['httpMethod'] == "POST":
    count = "UNKNOWN"
    key = json.loads(event["body"]).get("key")
    keySearchResp = table.query(
        IndexName='KeyLookup',
        KeyConditionExpression=Key('Key').eq(key)
    )
    item = keySearchResp["Items"][0]
    id = item["Id"]

    keyUpdateResp = table.update_item(
        Key={
            'Id': id,
        },
        UpdateExpression="set #c = #c +:num",
        ExpressionAttributeNames={
            "#c": "Count"
        },
        ExpressionAttributeValues={
            ':num': Decimal(1),
        },
        ReturnValues="UPDATED_NEW"
    )

    count =  keyUpdateResp["Attributes"]["Count"]
    response["body"] = json.dumps({ "COUNT": int(count)})
    response["statusCode"] = 200

    elif event['httpMethod'] == "OPTIONS":
    response["statusCode"] = 200

    else:
    raise Exception('Method not accepted')

except Exception as err:
    print("ERR: ", err)
    response["statusCode"] = 500

return response

Frontend

Vue JS

My choice of JavaScript framework is Vue JS and I make quite a lot of use of Vuetify, a Material Design styling framework. After installing, it’s as simple as:

vue create frontend
---step through configurations and use defaults
vue add vuetify

Amplify

AWS Amplify is a JS plugin which consists of some useful functions for serverless Auth and API, as well as some frontend components for Vue JS, enabling user authentication flows.

import Vue from 'vue'
import App from './App.vue'
import '@aws-amplify/ui-vue';
import Amplify from 'aws-amplify';
import  { Auth } from 'aws-amplify';

import store from './store'
import router from './router'
import vuetify from './plugins/vuetify';

const ROOT_DOMAIN = 't3chflicks.org';

Amplify.configure({
    Auth: {
        region: '',
        userPoolId: '',
        userPoolWebClientId: '',
        mandatorySignIn: false,
        oauth: {
            scope: [ 'email', 'openid'],
            redirectSignIn: `https://um-app.${ROOT_DOMAIN}/`,
            redirectSignOut: `https://um-app.${ROOT_DOMAIN}/`,
            responseType: 'code'
        }
    },
    API: {
        endpoints: [
            {
                name: "UserAPI",
                endpoint: `https://um-user.${ROOT_DOMAIN}`,
                custom_header: async () => { 
                    return { Authorization: `Bearer ${(await Auth.currentSession()).getIdToken().getJwtToken()}` }                  
                }
            },
            {
                name: "TestAPIKey",
                endpoint: `https://um-test.${ROOT_DOMAIN}`,
            }
        ]
    }
});

Vue.prototype.$Amplify = Amplify;
new Vue({
    store,
    router,
    vuetify,
    render: h => h(App)
}).$mount('#app')

In the Vue App, I use Amplify’s Authenticator UI components for user authentication flows.

<template>
<v-container fluid>
    <v-row align="center">
        <v-spacer/>
        <v-col cols="10">
            <amplify-authenticator username-alias="email" initial-auth-state="signup" v-if="!signedIn">
            <amplify-sign-up username-alias="email"  :form-fields.prop="formFields" slot="sign-up"></amplify-sign-up>
            </amplify-authenticator>
            <div v-if="signedIn ">
            <user-info class="my-2" />
            <amplify-sign-out />
            </div>
        </v-col>
        <v-spacer/>
    </v-row>
</v-container>
</template>

This is what an authenticated user sees:

Unauthenticated User Page*

With the API configured in main.js, calling an API inside a Vue component is pretty simple.

methods: {
  async testKey() {
    if(this.testAPIKey !== ""){
      const myInit = { 
          body: {
            key: this.testAPIKey
          },
      };
      const response = await API.post('TestAPIKey', '/', myInit);
      this.response = JSON.stringify(response);
    }
    else {
      alert("enter your key")
    }
  }

The authorisation header is set for the current logged in user. The code for accessing the User API is just as clean.

methods: {
    async getUserKey(){
    const resp = await API.get('UserAPI', '/user');
    this.APIKey = resp.KEY;
    },
}

full code ☁️ . . . . . . . . . . . . . . . . . . . . . . . . . . .live demo 💽

Deploy

Using CloudFormation templates mean it is simple to deploy this infrastructure using the AWS CLI.

aws cloudformation deploy --template-name ./00-infra.yml

Vue JS builds a static site which is uploaded to S3 and deployed using CloudFront.

npm run build;
aws s3 cp ./dist s3://<< your bucket name >> --recursive

Video of Use Manager App*

After Thoughts

This Architecture is the basis for a SaaS, the next step is to link to a payment service such as Stripe to handle subscriptions, which is done in part 2:
**💸 Pay Me: Quickstart for creating a SaaS pt.2 — Stripe Payments
**C*reating a SaaS solution is fun, but creating a payment system can be a minefield. In this article, I demonstrate how…m*edium.com

Some popular APIs from Rapid API Top 100:

URL Shortener Service — Learn More
Investors Exchange (IEX) Trading — Learn More
Crime Data — Learn More
Youtube To Mp3 Download — Learn More
Web Search — Learn More
JokeAPI — Learn More
Genius — Learn More
Crypto Asset Tickers — Learn More

Thanks for reading

I hope you have enjoyed this article. If you like the style, check out T3chFlicks.org for more tech-focused educational content (YouTube, Instagram, Facebook, Twitter).

The full code for this project can be found here ☁️

The live demo can be found here 💽

Resources:

☁️ 🚀 Virtual Private Cloud on AWS — Quickstart with CloudFormation

t3chflicks — Mon, 15 Mar 2021 15:04:08 +0000

A Virtual Private Cloud is the foundation from which to build a new system. In this article, I demonstrate how to create one using CloudFormation.

Photo by Pero Kalimero on Unsplash*

Virtual Private Clouds (VPCs) are networks in the cloud. A VPC contains a range of IP addresses that can be subdivided into smaller ranges of IPs known as subnets. Communication across these subnets can be controlled using route tables, security groups and access control lists. Communication to and from the internet is enabled by two types of gateway:

NAT Gateways which enable access to the internet.
Internet Gateways which enable access to and from the internet.

A subnet is considered private if it uses a NAT instead of an Internet Gateway. Route tables determine where network traffic from your subnet or gateway is directed. A route is required from the subnet to the gateway to enable internet access.

VPCs are important because they are environments which can be configured to reduce security risks and improve networks speed between components.

Architecture

For this example, I am going to create a place in Amazon’s Cloud. The VPC comprises three public and three private (hybrid) subnets.

Deployment

To deploy this system, I am using the AWS CLI with CloudFormation. It is as simple as:

`aws cloudformation create-stack --stack-name service --template-body file://template.yml --capabilities CAPABILITY_NAMED_IAM`

The full template can be found here.

VPC

To begin, the simplest VPC:

VPC:
    Type: AWS::EC2::VPC
    Properties:
    CidrBlock: "10.0.0.0/16"

The only property we are required to provide is the CidrBlock, a range of IP addresses that you declare available for use in the network.

The next step is to add subnets to the VPC. These occupy sections of the range of IPs in the VPC. You are required to configure CidrBlock which in this case is 10.0.1.0/24 meaning that we can use the addresses in the range 10.0.1.0 — 10.0.1.255 .

Subnet:
    Type: AWS::EC2::Subnet
    Properties:
    VpcId: !Ref VPC
    CidrBlock: "10.0.1.0/24"

The component required to enable access from a subnet to the internet is an Internet Gateway. This sits at the edge of the VPC and orchestrates any traffic routed from or to them with the internet.

The following template configures a public subnet:

PublicSubA:
    Type: AWS::EC2::Subnet
    Properties:
    VpcId: !Ref VPC
    CidrBlock: "10.0.1.0/24"
    AvailabilityZone: !Select
        - 0
        - Fn::GetAZs: !Ref AWS::Region

InternetGateway:
    Type: AWS::EC2::InternetGateway

GatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
    VpcId: !Ref VPC
    InternetGatewayId: !Ref InternetGateway

RouteTablePublic:
    Type: AWS::EC2::RouteTable
    Properties:
    VpcId: !Ref VPC

RoutePublic:
    Type: AWS::EC2::Route
    Properties:
    DestinationCidrBlock: "0.0.0.0/0"
    GatewayId: !Ref InternetGateway
    RouteTableId: !Ref RouteTablePublic

RouteTableAssocSubnetPublicA:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
    RouteTableId: !Ref RouteTablePublic
    SubnetId: !Ref PublicSubA

A private subnet is not directly accessible from the internet. However, it is possible to access the internet from a private subnet using a NAT Gateway. An Elastic IP is required because the NatGateway is not automatically assigned a public IP.

PrivateSubA:
    Type: AWS::EC2::Subnet
    Properties:
    VpcId: !Ref VPC
    CidrBlock: "10.0.4.0/24"
    AvailabilityZone: !Select
        - 0
        - Fn::GetAZs: !Ref AWS::Region

ElasticIPNatGatewayPrivateA:
    Type: AWS::EC2::EIP

NatGatewayPrivateA:
    Type: AWS::EC2::NatGateway
    Properties:
    SubnetId: !Ref PrivateSubA
    AllocationId: !Ref ElasticIPNatGatewayPrivateA

RouteTablePrivateA:
    Type: AWS::EC2::RouteTable
    Properties:
    VpcId: !Ref VPC

RoutePrivate:
    Type: AWS::EC2::Route
    Properties:
    DestinationCidrBlock: "0.0.0.0/0"
    InstanceId: !Ref NatGatewayPrivateA
    RouteTableId: !Ref RouteTablePrivateA

RouteTableAssocSubnetPrivateA:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
    RouteTableId: !Ref RouteTablePrivateA
    SubnetId: !Ref PrivateSubA

Availability Zones (AZs) are groups of data centres inside a Region that are isolated from the failure of another AZ. Each subnet created in a VPC must be placed in a single AZ and a system can be protected from an AZ failure by utilising multiple AZ architecture — you’re recommended to have three public and three private subnets spanning different AZ zones. A VPC template for this can be found here.

A VPC is pretty useless without any services running inside it. For that, check out the following articles:
Cheaper than API Gateway — ALB with Lambda using CloudFormation
*An alternative to API gateway is Application Load Balancer. ALB can be connected with Lambda to produce a highly…*medium.com
AWS Auto Scaling Spot Fleet Cluster — Quickstart with CloudFormation
*Running applications on separate machines quickly becomes an inefficient use of compute and therefore money. In this…*medium.com

Thanks For Reading

I hope you have enjoyed this article. If you like the style, check out T3chFlicks.org for more tech focused educational content (YouTube, Instagram, Facebook, Twitter).

Resources:

✨ Motion Sensing Under Bed Lighting

t3chflicks — Mon, 15 Mar 2021 15:03:17 +0000

Ever tried to get out of bed quietly at night only to trip over something and wake up the whole house?

Motion sensing night lights installed discreetly under your bed provide low-level light bright enough to guide you around obstacles, but dim enough so you’re not woken up fully. As well as sensing motion, it’s also possible to programme the lights to a colour of your choice for a fixed (or indefinite) length of time, adding a cool glow and ambiance to any bedroom.

With some pretty basic kit, you can install these lights with relative ease in a couple of hours.

Supplies

Power supply (5V 6A) Amazon
Addressable LED strip Amazon
Arduino Nano Amazon
Wire clips Amazon
Motion Sensors Amazon
Rocker switch Amazon
AC Plug
Wire

🔗 Get The Motion Sensing Under Bed Lighting Code On Github 📔

Tutorial

Build 🛠️

Measure the Bed

Turn the bed onto its side so the base is readily accessible. Find an appropriate location for the control box - we chose the slightly higher area near the head of the bed (see diagram). Measure the perimeter of your bed and its length and width (see diagram). Note down your measurements.

Determine a location for the three sensors. You want one facing each of the three sides of the bed which are not against the wall. We chose locations which were close to the edge of the bed, but not visible. Measure the distance from the sensor location to the control box.

Cut the Wires and LED Strip

Cut the LED strip to the length of the bed perimeter.

Next, cut the wires: you will need three for each sensor and three for the LED strip, each leading back to the control box, so 12 in total. Taking three lengths of differently coloured wire, cut to size. We used yellow, green and orange — the accepted convention is red for power, black for ground and another (bold) colour for signal. It doesn’t really matter which colours you use as long as you know which is which.

Solder Cables to Motion Sensor

We housed our motion sensors in 3D-printed cases (you can find the file link below). It’s not completely necessary to have these, but they make the sensors tidier and easier to put on the bottom of your bed.

If you are using the 3D-printed case, start by threading the three differently coloured wires through the lid. The motion sensors have three different pins: ground (GND), power (VCC), and signal (S) (shown below). When holding the sensor as shown in the diagram above (i.e. with the pins on the bottom edge of the module), attach the three different coloured wires their respective pins and solder them into place. Then, cover the wires using heat shrink. Repeat this for the wires cut for each of the three sensors.

Push the dome of the motion sensor through the hole in the main case. It should click into place. Close the case, leaving the three coloured wires trailing through the rear hole.

Wire the LED Strip

The LED strip has the three same connections: power, signal and ground — except the signal pin is an input.

These LEDs take instructions from the Arduino, each one of them addressable. We can change colour (RGB) and brightness. Solder three coloured wires to the LED strip, these will be used to connect to the Arduino later.

Power Switch

If you’re using the 3D-printed control box, you will need to install the power switch and connect it up to the wires.

Firstly, ensure there’s nothing on the end of your plug, if there is, cut it off. Thread the wire through the hole in the front of the box and out again through the hole for the switch immediately next to it. Strip the outer covering of the AC wire so that 10cm of the three inner wires (Live, Neutral and Earth) can be seen.

Then, cut and remove 8cm of the live (red) and neutral (blue) wires and set aside for later. Using the remaining 2cm of the end of the AC plug wire, solder the live (red) and neutral (blue) wires to the switch on the bottom two prongs (as shown in the diagram).

Next, take the 8cm pieces of live (red) and neutral (blue) wires you cut earlier and solder them onto the two prongs on the top of the switch (as shown in the diagram) — these wires will attach to the power box within the control box. Pulling the wires through first, push the switch into its hole in the box.

Connect Power Supply

Put the power supply in the box so the wire connection points face the switch.

Connect the live (red) and neutral (blue) wires from the switch to the live and neutral connection point (marked l and n respectively) on the power supply. The connection points on the power supply are screws, ensure these are done up tightly once the wires are in place.

Connect the Arduino

The power supply has output connections for 5V and ground (see diagram). Take the Arduino and cut a power wire (conventionally red, but whatever colour you are using) approximately 8cm long.

Connect the Arduino to the power supply by screwing one end of the power wire into the ‘5V’ connection point and soldering the other end to the ‘VIn’ on the Arduino.

Repeat the process with a ground (black, or whatever colour you have chosen) wire, connecting ‘GND’ on the power supply and Arduino.

Connect LED Strip to Power Supply and Arduino

Thread the LED strip’s wires through the remaining empty hole in the box.

Strip the LED strip’s power and ground wires. Connect the power (red) wire to the power supply’s ‘5V’ connection point (the Arduino is already attached to this) and the ground (black) wire to the power supply ‘GND’ connection point (the Arduino is already attached to this, too).

Solder the LED strip’s signal wire to the digital pin 9 of the Arduino.

Connect Motion Sensors to Arduino

Thread the motion sensors’ wires (9 in total) through the hole where the LED strip’s wires are.

Solder the three power wires to the +5V of the Arduino, the ground wires to the gnd of the Arduino and solder the individual signal wires to Arduino pins 10, 11 and 12.

Programme the Arduino

Download the code below called ‘motion_sensing_lights.ino’. Then, using the Arduino software downloadable form here, upload the code to your Arduino module.

If you are unsure about how to do this, find out more here. You will also need to download the FastLED library from here.

The code is pretty simple: it continuously checks if the motion sensors have outputted a signal and if so, starts a timer and turns on the LED strip to glow up, stay on a minute and then glow down.

Put on Bed

Shut the control box — the only things outside it should be the LED strip and the AC plug.

Stick the box to the bottom of the bed in your chosen location — we did this using strong double sided tape.

Then, attach the motion sensors to the bottom of the bed using double sided tape. The motion sensors should face outwards along the three sides of the bed which are not along the wall. Next, mount the LED strip around the perimeter of the bed.

Although the LED strip has a sticky back, this is not strong enough to hold its weight. Therefore, we held it in place using plastic wire clips which we hammered into the bottom of the bed. Plug in and turn on the control box and turn the bed the right way up.

Adjust, Test and Admire

Test out your motion-sensing under bed lighting. You can adjust the motion sensor sensitivity by putting a screwdriver through the top hole of the case and twisting the sensitivity resistor.

Taking It Further

Using an ESP8266 module (Amazon) instead of the Arduino, it is possible to control the LED strip with your phone or with Alexa by linking it to the open source home automation platform Home Assistant. There’s already been a great tutorial on how to do this, and you can find it here.

🔗 Get The Motion Sensing Under Bed Lights Code On Github 📔

Thanks for Reading

I hope you have enjoyed this article. If you like the style, check out T3chFlicks.org for more tech focused educational content (YouTube, Instagram, Facebook, Twitter).