In London there is a bicycle hire system that I frequently use. There are about 700 stations throughout the network, with so many of them I have always been intrigued to know how they flow across the city.
It turns out that the authority in charge of the system (TFL) has an API that allows anyone to find out how many bicycles there are per station at the time of querying it; And it turns out that this is just the information needed to create a visualisation that would allow me to observe the flow of bikes across the city throughout the day.
So with that in mind I decided to create a bot that queries the network status and presents the results, this is what the end result looks like, which we will get to when this series of posts is finished.
Yes, it is a tweet, and what we are going to do is an application that tweets the status of the cycle network at intervals of time. In this series I am going to talk about: Python with pandas, geopandas, twython, AWS Lambda, Docker, GitHub...
Secrets
Before proceeding, it is necessary to perform a few administrative tasks. To remember that what we are going to do is:
- Deploy the necessary infrastructure in AWS programmatically – which means that we will need access keys
- Tweet programmatically – which means we'll need access keys
AWS
😅 I am not sure about the costs for this exercise, I am around 80% sure everything falls under the AWS Free Tier, but I strongly encourage you to check for yourself.
🚨 Update: If you follow this tutorial to the letter, it will cost you exactly ~2 USD a month, the cost comes from the secret storage. As such, I've decided to turn off the lambda. The code works, I just don't want to pay for it anymore 😅
Lambdas run on AWS, so you'll need an AWS account. There are plenty of tutorials on the web on how to create an account so I'm not going to cover it.
The next step is to create a user and give it specific permissions for the tasks that we are going to perform.
Within your console, find the IAM service:
From there navigate to Access Management > Users and click in the Add users button:
Select a name for this user and, something very important, select the checkbox that says Access key - Programmatic access since we are going to access programmatically with this user – then click on the Next: Permissions button to assign the required permissions:
On the permissions screen, first select the Attach existing policies directly option, which will let you select pre-set permissions, from them make sure you select the following: SecretsManagerReadWrite, IAMFullAccess, AmazonEC2ContainerRegistryFullAccess, CloudWatchEventsFullAccess, AWSLambda_FullAccess y AmazonS3FullAccess, as shown in the image below:
To finish, and once you have chosen all the permissions that I outlined, click on the Next: Tags button, then on the Next: Review button and finish by clicking on Create user. You will then be presented with a success screen, where there are a few values that are better kept secret, very, very secret. I redacted them in the image below:
I reiterate, keep those secrets safe but close at hand, we will use them later; and while we're at it, we're going to be referring to these as follows: Access key ID → AWS_ACCESS_KEY_ID y Secret access key → AWS_SECRET_ACCESS_KEY.
😅 I recommend that you create a dedicated Twitter account for this task, unless you want tweets to come out of your “personal” account – any account you use must be verified via your cell phone.
Since we are going to be tweeting, it is necessary to enable our Twitter developer account at developer.twitter.com, if you have not already done so, in the top right corner you'll see a Sign Up button:
You will have to fill a few forms, when you are filling in this information make sure you select Making a bot in the What's your use case? question. This is how I filled out mine:
After accepting the terms and conditions and verifying your email account your developer account should be active, now you can return to developer.twitter.com to finally create your app and get a few other secrets.
In the developers portal, navitate to Projects & Apps and scroll until you see an + Add App button that you must click.
The next screen will ask us for a name for our app and as soon as we choose one, it will give us a part of the credentials we need to start tweeting:
Of the keys shown on your screen, store the API Key and API Key Secret, keep them safe and secure, we
De las llaves presentadas, guarda muy bien la de API Key y API Key Secret, we are going to use them to tweet from the Lambda. Oh, and from now on we'll refer to API Key as TWITTER_API_KEY and API Key Secret as TWITTER_API_SECRET. To finish, click on the App Settings button.
Inside App Settings scroll until you find a section named User Authentication Settings, and click Set Up.
In this new screen select only OAuth 1.0a, and in App Permissions choose *Read and Write:*
Whatever you put in the rest of the fields is not so important, for the web addresses that are required you can put the address of your blog or simply https://www.google.com, this value does not matter.
We will now return to the main screen of our app, ready to get the rest of the secrets we need. For this, navigate to the Keys and tokens section, towards the bottom of the page you will find Authentication tokens – if you notice, it indicates that the token was created with “read only” permissions, to change this, choose Regenerate and it will give you new tokens that, yes, you will have to protect and have at hand:
From now on, we will refer De ahora en adelante vamos a conocer a Access Token como TWITTER_ACCESS_TOKEN y a Access Token Secret como TWITTER_ACCESS_TOKEN_SECRET.
And if you notice, the token now says that it was created with read and write permissions:
So far you should already have 6 secrets, 2 from AWS:* AWS_ACCESS_KEY_ID y AWS_SECRET_ACCESS_KEY; from 4 from Twitter: TWITTER_API_KEY, TWITTER_API_SECRET, TWITTER_ACCESS_TOKEN y TWITTER_ACCESS_TOKEN_SECRET. We're almost done, just one more task to go.
Back to AWS
Secretos
Since we don't want to go around posting our secrets publicly, we're going to store them in AWS so that we can later access them during our lambda deployment without worrying about manually typing them.
From the console look for the Secrets Manager service:
You should find a button to add a new secre, this button says Store a new secret, click on it, which will take you to a screen where you must add the Twitter secrets, the first thing is to select Other type of secret, then in Key/value pairs add the secrets, just the Twitter ones, remember that there are 4, use + Add row, to add more fields – leave the rest of the options with their default value and click Next:.
The next screen will ask you about the name of your secret, choose something descriptive, you can use / to separate different namespaces in order to make the name more descriptive:
If you want, add a description but leave the remaining fields blank (or with their default values) and click Next until this button says Store, this one you should click too.
Once this is done, you will return to the Secrets Manager home screen and that is it, the secret has been stored.
A bucket for Terraform
Later we will use a tool called Terraform to manage the infrastructure needed to deploy our lambda on AWS. As part of its operation, this tool generates a file where the state of the infrastructure is stored.
If you work in a team, or are going to use a CI/CD tool (like us!), it is recommended that this file is available so that whoever is going to add or modify the infrastructure can access it. One of the suggested (and safe) ways to share it is through an S3 bucket in AWS –or similar if you are working with another provider–.
We are going to create this bucket since we are in the AWS console, search for S3 within the available services:
Upon clicking it you will see a Create bucket button:
You will be presented with a screen asking for the name of the bucket and the region in which it should be created, all my infrastructure will exist in eu-west-1, yours may be different, just make sure you use the same one consistently throughout this tutorial.
Leave all other options at their default values and finish creating the bucket.
GitHub
We are doing all this because we are going to automate the deployment of our app from GitHub, that is why you should create a new repository in which we will be putting our code – and as you will see, also our secrets.
I won't explain how to create a new repo, but I will explain the secrets part. First go to Settings in the newly created repository, then on the left bar select Secrets and then Actions. Finally click on New repository secret.
This one is a simple task, you must choose a name for the secret and assign a value to it. Remember, we need to save both AWS secrets: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, one at a time. Make sure you don't introduce unnecessary spaces in the values!
When finished, you should have a screen very similar to this one below:
And that is it, the "administrative" tasks are done.
Conclusion
There was no code in this entry, rather a lot of configuration and administrative tasks required to allow us to run continuous deployment and to be able to tweet from an AWS Lambda.
Remember that you can find me on Twitter at @feregri_no to ask me about this post – if something is not so clear or you found a typo. The final code for this series is on GitHub and the account tweeting the status of the bike network is @CyclesLondon.
Top comments (1)
Great post about using the Twitter API - thanks for sharing this! Look forward to reading more.