DEV Community: Chloe McAteer

Resizing EC2 instances without downtime

Chloe McAteer — Mon, 12 Jun 2023 21:05:27 +0000

As your project grows and user base expands, scaling up your compute resources becomes essential. In this blog I will explore the process of resizing AWS EC2 instances without incurring downtime and discuss some considerations to ensure a seamless transition.

Real world walkthrough

When scaling your infrastructure, it is important to regularly evaluate whether your current instance sizes are meeting the demands of your project. Consider factors such as CPU utilisation and user growth to determine when resizing becomes necessary.

I have been working on a project that has recently went live and over the past few months, its user base has been continually growing. I have auto scaling policies in place to scale up when the CPU utilisation gets to a certain level - however, there is a point when you realise you need to scale up your instances instead of continuously scaling out.

I have used Terraform for all my AWS infrastructure as code - so this is where I have defined my EC2 configuration. I have the config set up so that there should always be at least one healthy instance.



resource "aws_autoscaling_group" "example_ec2_asg" {
  name                      = "example-asg"
  vpc_zone_identifier       = [aws_subnet.private_subnet_1.id, aws_subnet.private_subnet_2.id]
  launch_configuration      = aws_launch_configuration.ecs_launch_config.name
  force_delete              = true
  health_check_grace_period = 10

  desired_capacity = 2
  min_size         = 1
  max_size         = var.max_size

  lifecycle {
    create_before_destroy = true
  }
}

# EC2 launch configuration
resource "aws_launch_configuration" "example_ecs_launch_config" {
  name_prefix          = "ecs_launch_config-"
  image_id             = data.aws_ami.example_ami.id
  iam_instance_profile = aws_iam_instance_profile.ecs_agent.name
  security_groups      = [aws_security_group.ecs_tasks_sg.id]
  instance_type               = var.ec2_instance_type // passing in as var
  associate_public_ip_address = false
  user_data                   = <<EOF
    #!/bin/bash
    echo ECS_CLUSTER=${aws_ecs_cluster.main.name} >> /etc/ecs/ecs.config
    EOF

  depends_on = [aws_ecs_cluster.main]
  lifecycle {
    create_before_destroy = true
  }
}

In the code snippet above you can see the launch configuration for my EC2 instances. You will notice that I pass the instance type in as a variable - this is because I use different instance sizes depending in deployment environment (e.g. QA, test, production).

You can see I have a lifecycle event set up, which specifies to always create a new instance before one is destroyed. I also have a desired capacity set of 2 instances running at any time.

The Goal

My EC2 instances are currently t2.small and I want to update them to be t2.medium

The Worry

For code deployments I have it set up to do rolling deployments, but my worry was that this change is bigger than just a code update, it is updating the actual compute the code is running on.
I was afraid if I updated my terraform configuration and ran it, it might try to update all instances at once and cause downtime.
I decided to test this update in a test AWS environment, before trying in QA or Production so I wouldn't cause any issues for the development team or end customers, while trying to understand this process.

The Plan

I updated the environment variable for terraform to now be a t2.medium and ran a terraform plan to view the pending changes:

In the screenshot, you can see the terraform plan is showing that there will be a force replacement of the EC2 instance to update the type.

To test and see if it would create downtime, I confirmed the deployment in my test environment. Once the terraform plan had finished, I went into the EC2 console and I could see that my instances were both still size t2.small.

I thought when the deployment had completed, it would have triggered each instance to start updating to the new size. However this is not the case - because I have updated the EC2 size in the launch configuration, which only gets triggered when you launch a new instance.

The Test

I have two instances running, so I thought I would try and stop one instance to test my theory and see what would happen. Within the AWS Console (again in my test environment), I selected one of my instances and selected to Stop it.

When I clicked this, I began hitting my APIs health endpoint, to verify traffic was still being routed to the running instance and when I checked back in the console I could a new medium sized instance was automatically being created.
I hit a health endpoint a few more times to verify everything was still running as expected while the new instance was being created and it was!

Once the new medium instance was in a ready state and I could see in ECS that the service was up and running, I then stopped the second small instance. Again once it was stopped, I could see a new medium instance being created in its place and being registered in ECS.

I now have confidence that I can do this upgrade on my QA and Production environments without any end user downtime.

Considerations

Utilise an infrastructure-as-code tool to help keep track of configuration updates.
Implementing rolling deployments to ensure one deployment is successful before you start another one.
Ensure you have a desired instance count higher than one instance, to enable a rolling update strategy and also help in the event one of your instances become unhealthy.
Ensure you have a health endpoint, set up to allow you to easily test the service during the infrastructure update to make sure traffic is being routed as expected.
Have a roll back strategy in place in event that something fails during your update

Conclusion

Resizing EC2 instances without incurring downtime is an essential aspect of managing a growing project. By following best practices such as the ones above you can seamlessly resize your instances to meet the changing demands of your application and confidently scale your infrastructure while providing a smooth user experience.

Dynamo DB Cross Region Migration

Chloe McAteer — Sat, 24 Oct 2020 10:05:35 +0000

When setting up a database such as DynamoDB it is important to choose a region that is closest to where your users are located in order to reduce latency for sending/retrieving data. However, if you have realised that your database is not in the optimal region for your users, it is possible to do a cross region restore with DynamoDB. In this blog, I am going to take you through restoring a table and updating it with Cloud Formation templates!

Overview of steps we will take:

Create a backup from the Point-in-Time Recovery
Restore the backup to a new region
Import the newly restored table into Cloud Formation
Update the Cloud Formation stack for the new table with the template of the original table

Creating the backup

When you create an on-demand backup, a timestamp of the request is catalogued. The backup is created asynchronously by applying all of the changes until the time of the request to the last full table snapshot.

To create a backup of a table, click on the Backups tab on the left hand side panel in the DynamoDB console and click on the blue Create backup button at the top.

This will then prompt you for the name of the table you wish to backup and also for a name for the backup itself:

Once the backup has been created you will see it listed on the console and you are able to select it:

With the backup selected you can then press the Restore backup button at the top!

💡 For more information on backups check out the docs here!

Cross Region Restore

To restore the table you’ll need to provide a name for the new table, in this case I kept it the same as the original as it will be moving regions:

To restore this table in to a different region all you have to do is select Cross Region and specify the region you want to restore it to — in this case I want to move it to EU West:

You are presented with an overview of the restore and then you can simply hit Restore at the bottom!

And now if you go to the region you selected, you will be able to see that the table is being restored:

Once it is restored, the tables status will be set to Active:

However, the restored table that is created is not actually identical to the original table. It is missing the following items:

Point-In-Time-Recovery
Auto scaling policies
AWS IAM policies
CloudWatch metrics & alarms
Tags
Stream settings
Time to Live

To set the above items to the same as what the original table had we will want to use the Cloud Formation template of the original table!

💡 For more information on restores check out the docs here!

Importing to Cloud Formation

As the table has already been created through the restore, it doesn’t have a Cloud Formation template associated with it yet. However, we can import this resource into a Cloud Formation stack through the following steps:

In the Cloud Formation console in the selected region (in this case Ireland), click Create Stack and in the drop down menu select With existing resources

Once selected you will need to specify a template — for this you can select Upload a template file. When importing resources into a stack, no changes are allowed to the existing resources, so we can’t add any of the tags or extra permissions at this point, we need to add a template that reflects the restored table at this given time! I have created an example Cloud Formation template that matches the exact specifications of the table that is created during a restore and the role associated with it, which can be viewed here, so you can use it and select next!

Now you will need to provide an identifier to map the logical IDs in the template with the existing resources, in this case I used the name of the DynamoDB table and the name of the role:

Then you just have to give the stack a name:

And your presented an overview of the changes that will occur — so in this case the imports required:

Once the import is complete, we can now move onto updating the stack!

💡 For more information on importing existing resources to cloud formation check out the docs here!

Updating the Cloud Formation Template

Now the resources are imported into a stack in Cloud Formation, we can update them to use the same template that the original DynamoDB and attached role had — to do this select the new stack and select Update at the top right hand side:

For this, we just want to Replace current template and give it the template we used to create the original table to make sure everything is kept consistent. The template for my original table can be found here.

You’ll then see the resources that are going to be modified and you can go ahead and select Update Stack at the bottom:

And that’s it! The table has now been migrated from one region to another, with identical settings 🎉

You can check the tags and permissions of the table in the new region and you will see that they have been updated!

Analysing Chocolate with Athena

Chloe McAteer — Fri, 14 Aug 2020 18:06:06 +0000

Recently, I have been wanting to up my game when it comes to analysing data — so I decided to use this as an opportunity to take AWS Athena for a whirl and see what it’s capable of.

Throughout this blog I am going to try and understand Athena and the features it has while working with a chocolate dataset.

So what is Athena? 🤔

AWS Athena is an interactive query service that analyses data using standard SQL. Athena is able to work with both structured and unstructured data and can work directly with data stored in s3!

What we’ll be using 👩‍💻

In this blog we are going to be integrating with a number of different AWS services including:

S3
IAM
Glue
Athena

We are going to be storing our data in an S3 bucket and then using a Glue crawler to create the table schema required by Athena — don’t worry if this sounds a bit scary now, we will be going through and explaining this step by step.

Show me the data 🍫

I am going to be using a chocolate dataset from Kaggle, which is a CSV file containing over 1700 ratings for chocolate bars and includes information regarding the type of bean being used, the regional origin and the percentage of cocoa they contain. The rating is a score between 1–5 (5 being great and 1 being unpleasant.)

Storing the data 🗄

Before we dive straight into working with Athena we need to put our data in AWS Simple Storage Service (S3).

You will need to create a bucket within s3 that has two folders inside it, one for the chocolate dataset and one for the results of the queries.

If you have not worked with S3 before, check out my previous post that will guide you though creating your bucket, uploading data and creating folders!

Athena Time ⏰

So in this tutorial we want to use Athena to run the following queries against our data:

Get all the countries listed in the dataset
Sort countries by rating
Discover the relationship between cocoa solids percentage and rating

Lets dive straight in 🏊‍♂️

In the AWS console we will navigate to Athena:

Once Athena opens we can go ahead and click on get started:

Connecting the Data 🧩

First thing we will need to do is connect to the data that we have stored in S3. In Athena you will see at the top left hand side of the screen there is an option to Connect Data Source, we will want to select this.

Once selected, you will see this:

This will allow us to choose our data source and connection method. We will choose S3 for our datasource as this is where our data lives and for the metatdata catalog we will go for the default which is AWS Glue.

What is Glue I hear you asking? 🤔

AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics

In Athena, tables are formed from the metadata definitions of the data’s schema. However, since S3 is just the data, we need to use a Glue Data Catalog to store this metadata about what lives within our selected S3 location (e.g. location, structure, column names, data types etc.) It is this metadata that will allow Athena to query our dataset!

One thing you might be thinking at this point is, how are we going to get this metadata to store in our Glue Data Catalog? Well, this is were Glue Crawlers come into play!

We can use a Glue Crawler to automatically extract our metadata, and create our table definitions!

In the previous step when we choose our data source and connection method, this screen is then displayed and it is here where we want to select to set up our Glue Crawler:

When we go ahead and click connect AWS Glue, it will open up the Glue console for us:

Here we select Get Started then select Add table using a Crawler :

This prompts us to give our crawler a name:

With our crawler named, we now also need to select Data Stores as our crawler source type:

Now we need to actually point it to our S3 bucket, typically a connection is not required for Amazon Simple Storage(S3) sources/targets, so we can leave this part blank:

Creating an IAM role

An IAM role is an Identity and Access Management entity that defines a set of permissions for making AWS service requests.

Our next step involves creating an IAM role to allow the crawler to have permission to access the data that we have put in S3.

When working with data that is constantly changing, you might have it set up that new data is being added to S3 hourly, daily or monthly — for this you can schedule the crawler to make sure it is always working with your most up to date data, by creating Crawler schedules expressed in cron format. However for this tutorial we are just going to select the Run on Demand setting as we only have the one dataset and we want to trigger it:

Now we have nearly got our crawler set up, we just need to add a database for the data to be store in:

Once the database has been created, you are presented with an overall summary, if everything looks good — click Finish

With our crawler set up, we can go ahead and kick it off, by selecting Run Now and we will then be notified once the crawler has ran:

The crawler has now went through our data, and inspected portions of it to determine the schema. Once we click into view it, we can see that it has been able to pick out each of the columns names and the data type for each column:

Now if we flip back to Athena, we can see that our database and table have now been populated with what we have just created:

One last thing we will need to set up before we query our data, is the results location — to do this, you can click on the link at the top of the page in the blue notification box:

Athena needs to know where the results from each query should be stored. For this we want to direct it to the results folder we created in S3:

Let the fun begin 🎬

Now we have all our set up done, we can dive in and start querying the data!

To query the data we can use standard SQL commands such as SELECT, FROM, WHERE, GROUP BY, ORDER BY, etc. I will go over some of this below, but to actually run the queries we need to enter them into the Query Panel on Athena, which is shown in the screen shot below:

To start off I am just going to try and select all the data, just to make sure everything is set up correct and to make sure we are getting data back. To do this I am going to run the following query in the query panel:

SELECT * FROM athena_chocolate_analyser;
~~~~~

and we can see, everything has been set up correctly and we are receiving results back:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/kflk8i4c4spwayyifob6.png)

Now let’s try out some of our other queries! First up, getting a list of all the countries contained in the dataset:

~~~~
SELECT DISTINCT companyLocation
FROM athena_chocolate_analyser;
~~~~

Here we have also used the `DISTINCT` statement, to make sure that we aren’t getting back duplicates! This gives us back the following list:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/uhd50vr28mb1pa0blbwl.png)

Next we want to sort countries by their ratings to see which ones produce the highest rated bars, for this I used the following query:

~~~~
SELECT companylocation,
         AVG(rating) AS averageRating
FROM athena_chocolate_analyser
WHERE
  companylocation IS NOT NULL AND rating IS NOT NULL
GROUP BY  companylocation
ORDER BY averageRating DESC
~~~~

Here I have thrown in a couple more SQL statements for example `AVG` to find the average rating, `AS` to create a alias temporary name for a column, `NOT NULL` to make sure we aren’t getting any null or empty values back and then also the `GROUP BY` and `ORDER BY` statements to group and sort the data returned!

Which then brings back a list sorted by the average rating for that country, so we can see that the highest rated chocolate comes from Chile!

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/tm6kgw4q9pnrvpy5djs8.png)

Our final query, is to try and see the relationship between the percentage of cocoa and the average rating. To do this I used the following query to find the average rating and the cocoa percentage and to group the results by the cocoa percentage:

~~~~
SELECT cocoapercent,
         AVG(rating) AS averageRating
FROM athena_chocolate_analyser
WHERE
  cocoapercent IS NOT NULL AND rating IS NOT NULL
GROUP BY cocoapercent
ORDER BY averageRating DESC
~~~~

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/tvmkieip9fts23tz6wft.png)

I find this all really impressive, as it’s super easy and fast to query the data to get these results!

You can view the history of the query’s ran against the data here in the history tab, which is useful to look back on:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/n38havz75a72ibvwbyn4.png)

And if you want to save any of your query results, you can click the `Save As` button at the bottom of the query panel and this will save your results into the results folder you have set up in s3 — you will notice in the screen shot about that each query has a unique identifier called a `Query ID`. This makes it easier work with/find query result files.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/pqq6eb0slssfw8xifn3r.png)

##Conclusion

I have really enjoyed my first attempt working with Athena, it seems super fast and powerful. With its ability to query data sitting in S3 and export results, I can already see so many real world use cases for example; querying billing/usage reports, to gather insights on spending.

I plan on taking a more detailed look into it, now that I have scratched the surface, so watch this space for more blogs to come!

DynamoDB Scan Vs Query

Chloe McAteer — Thu, 14 May 2020 21:52:51 +0000

DynamoDB is Amazon's managed NoSQL database service. This blog will be focusing on data retrieval and how it is critical to think about what your data will look like, to make an informed decision about your database design.

When working with DynamoDB there is really two ways of retrieving information - one being scanning and filtering and the other being querying the database! So what is the difference and what should I use?

Before we get started, something we will be talking about a lot is partition keys, so let's start with a short definition of what this is:

Partition Key - Is a primary key that DynamoDB uses to partition the data and determine storage.

First things first, what is scanning?

Scanning involves reading each and every item in the database. It allows you to add filters if you are looking for something in particular, so that only items matching your requirements are returned. However, every single record still needs to be read, as the filter is only applied after the scan has taken place!

Example:

If we had the following data and say we set the employeeID as the partition key once we set up the database:

const item = {
    employeeID: { S: "4beb73f0-2fc0-41b2-a8e9" }, //Set as partition key on DB creation
    startDate: { S: "2020-05-09T17:53:00+00:00" }
    name: { S: "example-name" },
    title: { S: "example-title" },
}

We could scan the database using the following as our scan params:

const params = {
        TableName: "employees",
        ProjectionExpression: "employeeID, name",
        FilterExpression: "title = :title",
        ExpressionAttributeValues: {
         ":title": {S: "example-title"}
        }
    };

The above code snippet would scan each item and would then filter for items that have a title the same as the one specified! The filter expression here could filter for any column/attributes in this database (e.g. employeeID, startDate, name, title).

But what is querying and why is it different?

Querying allows you to retrieve data in a quick and efficient fashion, as it involves accessing the physical locations where the data is stored. However, the main difference here is that you would need to specify an equality condition for the partition key, in order to query!

Example:

If we take the same example again:

item = {
    employeeID: { S: "4beb73f0-2fc0-41b2-a8e9" }, //Set as partition key on DB creation
    startDate: { S: "2020-05-09T17:53:00+00:00" },
    name: { S: "example-name" },
    title: { S: "example-title" },
}

Since we want to query the table this time, we can make use of employeeID as the partition key and we would be able to write query params like this, where our KeyConditionExpression is looking for a particular ID:

queryParams = {
    tableName = "employees",
    KeyConditionExpression: "employeeID = :id",
    ExpressionAttributeValues: {
            ":id": "4beb73f0-2fc0-41b2-a8e9",
    },
}

With using the partition key the query would be more efficient as it doesn't need to read each item in the database, because DynamoDB stores and retrieves each item based on this partition key value!

But what if we want to Query for something that is not the partition key?

If I want to query another value that is not the partition key e.g. what if we only have the employees name and want to get all their details by that name?

At the minute with our current set up, we would not be able to write a query for this because as I mentioned before - queries need to use the partition key in the equality condition! However, there is still a way we could query for this without having to do a scan.

Secondary Indexes

Using secondary indexes allows us to create a subset of attributes from a table, with an alternative key to create a different access point for query operations.

You can create multiple secondary indexes on a db, which would give your applications access to a lot more query patterns.

We can create a secondary index on DyanmoDB by specifying the partition key for it and naming the index:

Now with our secondary index set up, we can go ahead and query using it:

queryParams  = {
        "TableName": "employees",
        "IndexName": "name-index", 
        "ProjectionExpression": "employeeID, startDate, name, title",
        "KeyConditionExpression": "name= :name",
        "ExpressionAttributeValues": { 
            ":name": {S: "example-name"}
        } 
    };

Notice that we are using the new secondary index within our query. We can now find the employee details by using the employees name!

Setting up secondary indexes do have associated costs, but when working with large amounts of data, it can really increase the performance and efficiency of data retrieval. It can get items based on storage location without having to read every item in the whole database.

To improve efficiency further, you could also look into adding composites keys or indexes which can be made up of a partition key and a sort key.

Scan or Query?

So coming back to our main question, when do we use scan and when does it make sense to use query?

And honestly, it all depends on the size and amount of data you are working with!

If you are working with a small amount of data, you could totally go for scanning and filtering the database and not have to worry about adding all these extra keys. If the data is already small, the scan time won't take long anyway, so adding in things like secondary keys to partition into even smaller sets, isn't likely to increase your performance by a significant amount and therefore might not be worth the additional overhead of implementing these.

However, if you are working with large amounts of data, that is likely to keep growing - it is really worth spending time and making sure you choose the right secondary indexes.

When creating a database with indexes, it is really beneficial to spend time considering what queries are you likely to be doing. Understanding what data you will need to retrieve will help you choose your partition keys. Taking the initial time to think this through will make sure your database is set up the right way for you to retrieve data from it in the quickest, most efficient manner! Failure to think about this up front may limit you data access points down the line.

Transforming Teaching with Teachingo - Update #4

Chloe McAteer — Mon, 20 Apr 2020 20:00:36 +0000

This is an update on our #TwilioHackathon project progress - you can see the original post here:

Transforming Teaching with Teachingo - #TwilioHackathon Submission

Chloe McAteer ・ Apr 6 '20 ・ 4 min read

#twiliohackathon #javascript #node #react

Authentication

Because the software is going to be used by schools it needs to be secure - for this we wanted to ensure that not just anyone can access it and join any lesson! We wanted to set up accounts for students and teachers to ensure that only the students that belong to that particular class can access it.

Keeping passwords secure

Of course we didn't want to store the users password directly in the database, because if the passwords where stored in plain text, it would mean that if anyone, either an attacker or a developer carrying out maintenance on the database would be able to see exactly what people have set as their passwords and so the security of the system would be breached.

To overcome this we wanted some way of salting and hashing them. For this we decided to use Bcrypt - due to some previous experience using it.

As soon as a user creates an account, I use Bcrypt to salt & hash the password and then store the hashed version of the password in the database. Then once a user tries to log in we can use the Bcrypt .compare() function to compare the password the user entered with the hashed version from the database to authenticate them.

Handling User Sessions

As an extra layer of security, to ensure that users have been authenticated to use the applications services, the project creates user session tokens when the user logs in. To facilitate this, we decided to utilise JSON Web Tokens (JWT). Doing so ensures that no one can bypass login and access the services pages by changing the URL or hit the backend API directly.

Once a user successfully logs in, a session token is created for them and this token is sent with each request the user sends. Once the request is being handled, we do a check for two things - one, is the token valid and two, has the token expired. If these checks pass the request is carried out, however if it fails, a 401 error is thrown as the user is not authorised!

Transforming Teaching with Teachingo - Update #2

Chloe McAteer — Sat, 18 Apr 2020 16:17:19 +0000

This is an update on our #TwilioHackathon project progress - you can see the original post here:

Transforming Teaching with Teachingo - #TwilioHackathon Submission

Chloe McAteer ・ Apr 6 '20

#twiliohackathon #javascript #node #react

Data is the new oil 🛢

When thinking about the different user flows for an application like this, it is clear to see that there are a lot of different data points being generated, which meant we had to think about adding a persistence layer to our application.

So we initially spent some time thinking about what the best way to store this data was. Both of us have previous experience working with NoSQL, so we considered using MongoDB since it is quick & easy to get up and running.

However, we took the time to understand what all data we would be working with, we realised we would need to store all of the following:

Users email address, password, name, mobile and if they where a student or teacher.
Class names, teacher that teaches it, students that attend it
Lesson time/date, number of questions asked in lesson, lesson feedback etc.

From seeing this, it was clear that it made more sense for us to opt for a more structured database approach, the relationships between the different data points were more complex than we initially thought. We had a quick brainstorm about the database platform, ultimately settling on PostgreSQL.

Integrations.js ✨

Having decided on the platform, we needed to understand how our Node.js service could interact with a Postgres instance. I came across Knex.js which is an SQL query builder that can be used with Postgres!

It allowed me to be able to define schemas for each table within the code and create functions for getting, adding and removing data from the db - I was amazed by how powerful it was and how much of the heavy lifting it could do out of the box.

It also meant that once someone else pulled down the repository, they could run the database migrations, to get all the tables set up the correct way!

Knex also allowed me to define and generate seed data for the application, which meant I could put large amounts of dummy data into the database.

We now have our database up and working, but we did face some problems along the way, when it came to actually modelling it. For example: duplication of data and over complicated tables.

Transforming Teaching with Teachingo - #TwilioHackathon Submission

Chloe McAteer — Mon, 06 Apr 2020 21:48:34 +0000

The Team

Peter and I are two software engineers from Belfast, Northern Ireland who graduated from university last summer! Whenever we discovered the Twilio/DEV hackathon, we thought it was a great opportunity to jump into something that we can really get stuck in to.

With the hackathon being the majority of the month of April, this allowed us to really take our time with the idea and build something that could have a real impact in the world right now.

Given the current state of affairs in the world, everyone is flocking to the internet and various software/resources for communicating. Everyone.

The Problem

Most governments around the world have temporarily closed educational institutions in an attempt to contain the spread of the COVID-19 pandemic.
These global closures are impacting over 89% of the world's student population (source - https://en.unesco.org/covid19/educationresponse).

With these closures, schools across the globe are attempting to transition to online, remote learning. With some of our personal connections working in the education space, they had voiced their frustration with the lack of general tools in the wild that they could use to fit their needs - specifically, when it comes to video conferencing software.

Sure, there are a bunch of services that provide that facility to video call one another but they are mostly aimed at corporate businesses rather than education.

Our Proposed Solution

We are creating an E-Learning platform that is specific to teachers in order to fulfil their different needs when teaching a remote lesson. Some of these features include:

Video conferencing including: screen share, the ability to mute users and facilitate live chat.
Automated attendance checker.
Automated message sent to students that did not attend lesson.
Reported lesson statistics - who/how many asked questions in live chat, what percentage of the class attended (list of who did and who didn't).
Request a transcript or recording of the lesson (could be emailed to the students who attended or to the students that missed the class).
Feedback request - students can show they understand the topic being taught via red, amber, green feedback mid-lesson.
General student feedback about topics post-lesson.

What we built

Category Submission: COVID-19 Communications/Interesting Integrations

Choosing the right technology 📚

We wanted to make our solution platform agnostic, thus we opted to create a web application that both the teachers and students could use.

We decided to play to our strengths, with both of us having some experience with React and a lot of experience with JavaScript, we decided to build all the things with JS. For more details on our tech choice, check out our first progress update blog 👇

Transforming Teaching with Teachingo - Update #1

Peter McAree ・ Apr 14 '20 ・ 2 min read

#twiliohackathon #javascript #node #react

Let's talk data 🔢

With our tech stack chosen, it was time to really think about the data that would be passing through our systems. How would it be structured? Where would it be stored? And most importantly, how would it be secured? Check out more about our data decisions in our second progress update blog:

Transforming Teaching with Teachingo - Update #2

Chloe McAteer ・ Apr 18 '20 ・ 2 min read

#twiliohackathon #postgres #node #javascript

Prepare to Lanuch 🚀

With just a simple spike created of our front end application, server and database - we wanted to hit the ground running and set up a CI/CD pipeline to automate deployment. Check out how we set it up below 👇

Transforming Teaching with Teachingo - Update #3

Peter McAree ・ Apr 19 '20 ・ 3 min read

#twiliohackathon #node #react #javascript

Security, Security, Security 🕵️‍♀️

As the application is going to be used as an educational tool, it is essential that it is secure - to see some of the security measures we have taken, have a read of our blog number 4:

Transforming Teaching with Teachingo - Update #4

Chloe McAteer ・ Apr 20 '20 ・ 2 min read

#twiliohackathon #javascript

Twilio Time ⏰

Next step was to check out the Twilio services, SDKs and more to see what they could offer us. At a first glance we couldn't believe what Twilio could do out of the box and jumped straight into working with it. Take a look at how we got started and some of our code snippets in progress update number 5!

Transforming Teaching with Teachingo - Update #5

Peter McAree ・ Apr 28 '20 ・ 3 min read

#twiliohackathon #node #javascript #twilio

Demo Link

We recorded a short demo of the main features in Teachingo.

You can check out the deployed application here

Link to Code

pmc-a / teachingo-client

Web application that powers the Teachingo platform 🧠

pmc-a / teachingo-api

Node.js service that powers the Teachingo Client 🚀

GitHub Profiles:

Completed Features

Once we started this hackathon, we had an endless list of possible features we wanted to add to this application, but unfortunately due to time constraints it wasn't possible to add everything we wanted, but below is a full feature list of what we achieved:

Ability to securely log in as a teacher or student
Ability to view upcoming lessons
Ability to start/join a video call
Ability to mute/unmute mic, turn on/off camera and share screen
Ability to live chat with everyone in the lesson to ask questions
Abiltity for the teacher to view summary lesson statistcs at the end of the lesson.
Ability for the teacher to send an SMS to students who missed the lesson.

Some additional features that we wanted to add and didn't have time to complete:

Ability for students to show red, amber or green to highlight their understanding of the topic
Ability to request a transcript/recording of the lesson
Ability to obtain general student feedback at the end of the lesson

Additional Resources/Info

We hope you like our submission! We have been tweeting our progress throughout the hackathon so if you want to see our journey check us out at @chloeMcAteer3 & @PMc_A!