DEV Community: Harry@StartQuick Tech

AWS Copilot Saves Developers

Harry@StartQuick Tech — Mon, 17 Apr 2023 05:41:04 +0000

If you’re a developer building containerized applications on AWS, it can be a complex and time-consuming process to manage all the infrastructure and deployment tasks. From setting up clusters to configuring load balancer and scaling, there are many things to consider.

That’s why you need AWS Copilot.

If you like to follow the hands-on with me, you can watch the video below. I am happy if you can subscribe my channel if you want to learn more .

Using AWS Copilot, you need to understand below concepts.

Application — an Application is a collection of services and environments.
Environment — a Environment is specific configuration of the Applications, such as staging or production. Each Environment has its own set of resources, and Copilot manages them separately from other Environments.
Service — a Service in Copilot represents a single component of your App, such as a web server. Once you deploy a service in the environment, Copilot will build the image, push it to Amazon ECR and set up the required infrastructure for running the containers.

Let us use AWS copilot to create a django website running on AWS fargate cluster and CICD pipeline using AWS Code Pipeline.

Prerequisites

AWS CLI v2 and AWS Copilot are both installed.
AWS Credential is configured correctly.

Prepare the initial code

Go to my Github repository django-start branch and clone the branch to you local machine.

git clone -b django-start https://github.com/harryzhou1987/startquick-aws-copilot-django.git

Replace Django Secret Key in settings.py file which is in django-project/mysite/mysite.

SECRET_KEY = '[You Own Django Secret]'

Run Docker Compose to set up a local development environment

cd startquick-aws-copilot-django
docker compose up -d --build

Go to http://localhost:8080 in your web browser and check if the local environment is up.

Set up and Deploy the containerized service on AWS Fargate using AWS Copilot

Initialize an App.

copilot app init

Create Environment named test.

copilot env init

Check the manifest.yaml for the environment and update it with VPC & subnet information. Add certificate arn for the domain which you are going to use. You can also use your own existing VPC and subnets. Then deploy the test environment using below command

copilot env deploy --name test

Once it is done, a new VPC is created. We need to create an RDS instance for this project. You can either create the database using AWS console or command line below. You can also use your own database if you have but make sure you set up the correct environmental variables.

# Create a subnet group for RDS
aws rds create-db-subnet-group \
    --db-subnet-group-name [Subnet Group Name] \
    --db-subnet-group-description "DB subnet group for private subnets" \
    --subnet-ids [Private Subnet ID1] [Private Subnet ID2] ...

# Create security group for RDS
aws ec2 create-security-group \
    --group-name [Security Group Name] \
    --description "Security group for database instance in the private subnets" \
    --vpc-id [VPC ID]

# Here you can record the security group ID or check the security group ID via AWS console.

# Create ingress rule for the security group
aws ec2 authorize-security-group-ingress \
    --group-id [DB Security Group - Output of above command] \
    --protocol tcp \
    --port 3306 \
    --source-group [Service Security Group - Copilot created already]

# Create RDS instance
aws rds create-db-instance \
    --db-instance-identifier [DB Instance Name] \
    --db-instance-class db.t2.micro \
    --engine mysql \
    --engine-version 8.0 \
    --allocated-storage 20 \
    --master-username dbuser \
    --master-user-password SecretPassword \
    --db-subnet-group-name [Subnet Group Name] \
    --vpc-security-group-ids [DB Security Group] \
    --db-name djangodb \
    --no-multi-az

When the DB instance is ready, start the service in the test environment

copilot init

It should fail due to the missing alias for the load balancer. You need to add below in the environment part of the manifest yaml file.

environments:
  test:
    http:
      alias: # The "test" environment imported a certificate.
        - name: "[Domain Name for the Service]"
          hosted_zone: [Hosted Zone for your Domain]

Add variables for the service. Refer to docker compose yaml file. DB Host needs to tbe the endpoint of your RDS instance.

DB_HOST: [Endpoint - RDS instance]
      DB_NAME: djangodb
      DB_USER: dbuser
      DB_PASSWORD: SecretPassword
      ALLOWEDSOURCE: 0.0.0.0
      DEBUG: true

Add Allowed_Host in your django project settings.py file. I use below for the test environment.

ALLOWED_HOSTS = [
    ip_address,
    'localhost',
    'www.cloudcracker.click' # Replace this with your own
]

Run below command to deploy the service again.

copilot svc deploy --env test

Go to Site URL in your web browser and check if the site is up.

Create Deployment Pipeline

Check out to a new Git branch if needed.

git checkout -b "new-branch"

Run below to build the pipeline. The pipeline is using AWS Code Build and Code Pipeline.

copilot pipeline init
git add copilot/ && git commit -m "Adding pipeline artifacts" && git push
copilot pipeline deploy

During the deployment, you need to set up the authorization of your code repository for AWS.

Final Test

Assuming your docker compose environment is still up.

Do the dev work locally and confirm your dev work via http://localhost:8080
Push the code to your remote repository and wait until the automatic deployment is done.

Easy as! You don’t need to manually create the infrastructure for your containerized service and AWS did everything for you. You can check CloudFormation to see what services AWS helps you create.

Feedback

I might miss something. Please leave your comments for any questions. Thank you.

Harry@NZ

Software Upgrade Can Cause Service Outage

Harry@StartQuick Tech — Tue, 04 Apr 2023 01:19:58 +0000

Software upgrade or migration can be a headache for operation team. You need to think about all the potential issues which might cause the system down. This will decide how long is your maintenance window going to be.

I have got an interesting topic for you to think about.

Imagine you have lots of VMs or containers holding the application and a Database instance which can be either MySQL or PostgreSQL. As a web app, you also have load balancer to serve the traffic.

Now you have a requirement to upgrade to a new version of the software on your servers or containers and also at the same time, the database is also upgraded. Once this happens, the database is not compatible with the servers with the old containers.

What questions should you ask?

Does the software upgrade require the database upgrade?
Do we expect an outage?
What deployment strategy should we use?

With regards to upgrading the software, we need to run a upgrade script on the container or servers. This script not only upgrade the software on the server, it also need to upgrade the database such as changing the add new tables, changing schemas…etc. Once the database is upgraded to the new version, the containers or servers with previous software versions cannot query correctly.

This can cause service outage, right? You can probably pause here and think about your solutions.

Our real case is having multiple containers running on AWS Fargate and the database we use is Amazon RDS. You may be not familiar with AWS Fargate, so I will try explain my thoughts in a simple way.

I have two options.

Option 1 — Add new container to run upgrade script first

Create a new container with upgraded software in the cluster and let it run upgrade scripts for the database. If the upgrade is done, the script does nothing.
When the upgrade script is done, you can imagine the old containers cannot take requests as they are not compatible with the upgraded database. With the health check configured on ALB, the old containers will be marked as unhealthy and removed and rebuilt with the new version of software, but it takes some time which causes service outage.
You can manually remove the old containers on your ALB when the new container is healthy, but it still takes time though it might be shorter than letting ALB automatically do it.
In our case, another interesting thing we noticed during testing phase is we cannot create multiple new containers for the upgrading process as the upgrade script will run at the same time which caused error as well. So we have to add one first and rebuild the rest.

Option 2 — Add new containers to the new database

In this method, you need to duplicate the database first.
When you create the new containers, you need to let the containers connect to the new database, which means old containers talk to old and new containers upgrade the new database and talk to it.
But one thing needs to be very careful, you need to stop all the write behaviour to the database. Otherwise, database will happen.

Above methods both introduce outage if we follow the common deployment strategy. However, the second one just require you to stop write actions which might not introduc outage for some types of sites or systems.

Do you have any good ideas? Feel free to leave your comments.

Harry@NZ

How Docker Compose helps you setup multi-container application

Harry@StartQuick Tech — Thu, 30 Mar 2023 10:08:24 +0000

Lots of companies are using container technology to deploy their applications but their developers are still using the traditional way for the local dev environment set up such as installing framework, dependancies on their local machine.

Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

If you like to learn it with hands-on practice, you can checkout my video which shows you how to use docker compose step by step. Hope you can also leave you comments with your suggestions or anything you wanna learn.

Regarding learning Docker Compose, there are some key points you should understand.

YAML File for Docker Compose:

Below is the example we are using in the video for a local wordpress development environment setup. This is really helpful if you production environment is using container technology such as AWS Fargate, Kubernetes.

version: '3.1'

services:
  wordpress:
    image: wordpress
    build:
      context: .
      dockerfile: ./Dockerfile
    restart: always
    depends_on:
      db:
        condition: service_healthy
    ports:
      - 8080:80
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
    volumes:
      - ./varwww:/var/www/html
    networks:
      - wp-network-default  

  db:
    image: mysql:8.0
    restart: always
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_RANDOM_ROOT_PASSWORD: '1'
    healthcheck:
      test: ['CMD', 'mysqladmin', 'ping', '-h', 'localhost']
      interval: 5s
      timeout: 5s
      retries: 5
    volumes:
      - db:/var/lib/mysql
    networks:
      - wp-network-default  

volumes:
  db:

networks:
  wp-network-default:

Version: This specifies the version of Docker Compose to use.
Services: This section defines the different containers that make up the application. Each container is defined as a service, and multiple services can be defined in a single YAML file.
Image: This specifies the Docker image to use for each service. If the image is not found locally, Docker Compose will pull it from a registry.
Ports: This maps the ports used by the container to the host machine, allowing the container to be accessed from outside the container.
Environment: This section defines environment variables that will be passed to the container. These variables can be used to configure the container at runtime.
Volumes: This maps files or directories on the host machine to files or directories inside the container, allowing data to persist even if the container is deleted.
Restart: This defines the restart policy for the container. It specifies whether the container should be automatically restarted if it crashes or is stopped.

Commands for start, stop, remove the environment. Below are the most basic commands:

Start containers

docker compose up
docker compose up -d # quite mode
docker compose up -d --build # rebuilt the image

Stop containers

docker compose stop

Stop and Remove containers

docker compose down

Volumes and Networking (Very important for advanced usage)

Volumes are a way to persist data in Docker containers. They allow you to store and share data between containers or between a container and the host machine. When you create a volume in Docker, it creates a separate storage location outside of the container’s file system that is managed by Docker.

There are two types of volumes in Docker: named volumes and host volumes.

Named volumes are created and managed by Docker, and are ideal for sharing data between containers.
Host volumes, on the other hand, allow you to mount a directory from the host machine into the container, and are useful for persisting data on the host machine.

Volumes are useful for a number of reasons, such as ensuring that data is not lost if a container is deleted, and allowing multiple containers to access the same data. They can be used for a variety of applications, such as storing database data, configuration files, or user data.

Docker networks are used to connect multiple containers together, allowing them to communicate with each other. When you create a network in Docker, it creates a virtual network that allows containers to communicate with each other as if they were on the same physical network.

There are several types of networks in Docker, including bridge networks, host networks, and overlay networks.

Bridge networks are the default network type in Docker, and are used to isolate containers from each other.
Host networks allow containers to use the host machine’s network stack, and are useful for applications that require low-latency communication.
Overlay networks are used to connect containers across multiple Docker hosts.

Networks are useful for a number of reasons, such as enabling containers to communicate with each other, allowing multiple instances of the same container to communicate with each other, and isolating containers from each other for security purposes.

I do use Docker Compose quite a lot in my work. Please feel free to leave your questions and I am happy to chat with you and learn from each other.

Thank you.

Harry@NZ

How to Become a Cloud DevOps Engineer #Skills You Need

Harry@StartQuick Tech — Wed, 01 Feb 2023 07:02:28 +0000

People say DevOps will be replaced by long-term platform engineering but don’t worry as the skills you learn will not expire.

Today, we are going to talk about what key skills you need to master to be a DevOps engineer.

If you are new to my Channel, you can go have a look at the videos and subscribe. You can watch the video if you like.

Below is the script.

I broke the technical skill sets base for a DevOps engineer down into 8 domains.

1 Operating System

No matter you are a developer, tester or administrator, Linux is always a must-learn operation system.

You don’t have to understand Linux or Unix in depth such as Kernel Development but something you should be very familiar with. For example, you should understand what linux basic file structure looks like such as what /etc or /var or /bin folder are used for. You should know how to manage packages using tools such as rpm, yum, apt. Also you should know the basic command lines such as ls, mv, cp, rm, mkdir for file system, chmod, chown for permissions; top, ps for monitoring, etc… And it is better that you know how to write a bash scripts for automation tasks.

For above, I think 2 weeks ( about 20 hours) learning and hands-on exercise should be enough for beginners.

2 Code Management

Git is the most popular version control system.

For git repository tool, I have used Github, Bitbucket and AWS Commit but for learning purpose, Github is what I recommended.

If you haven’t used Git before, I would suggest you to start from the theory and basic commands for creating local repository, committing code changes, pushing commits to remote repository. Also, you need to understand how to use branch, how to create pull request to merge code. If you have got some experience on Git, just check if you understand how to control the user access and permission as an administrator, how to manage the code such as complex code workflow and different types of merging strategies. I believe most of you have used Git already but if you haven’t, you may just need a week to learn and practice in your daily work.

3 Container and container orchestration.

Containers is getting more and more popular nowadays. It allows you to build your application and ship it in a container with all its runtime and dependencies, which could hugely simplify the deployment process.

The most popular one I believe you must have heard is Docker.

For beginners, start from the theory and basic docker commands such as docker build, docker run, docker ps, docker images. Then you should learn how to build docker image via Dockerfile and run a container based on the image you created. Also, you should understand how docker network works such as port mapping. Once you get this, you can try using docker compose for building multi-container applications which is type of an orchestration locally. For orchestration, I would highly suggest you to learn AWS ECS Fargate which could be an easy start and cheap for its serverless mode. If you have experience, try Kubernetes as it’s getting popular and almost in a dominate position. You can build a local cluster using miniKube or spin up VMs locally.

4 Cloud

You can choose AWS or Azure or GCP. I use AWS, Azure and even Tencent Cloud. I could not say which one is better but I prefer AWS a little bit more. You can choose one of them based on your working scope but if you are an absolutely beginner, AWS is never a wrong choice.

I will keep making AWS learning video in my YouTube channel. For learning AWS, maybe start from a certificate such as Cloud Practitioner or Solutions Architect Associate. Getting the certificate does not mean you are qualified for the role but the learning guideline can point you to the correct learning path. This is at least my real experience. You can check the video in my channel for my learning path.

5 CI/CD

CI/CD stands for Continuous Integration and Continue Deployment or Delivery. This is one of the core parts of DevOps concept. What is it? I actually like it call it auto-deployment but in order to make it accurate, let’s see the chart.

Continuous Delivery includes from coding, building, deploying to stage as well as acceptance tests. Continuous delivery or deployment include another step which is to deploy into production. But the difference is delivery is manual and deployment is automatic. In the real world project, there are quite a lot of things to consider about the CICD pipeline such as how many environments you have, the balance between quality and speed, what technical tools you gonna use, etc. From learning perspective, I would suggest you to start with AWS code build, Code Deploy and Code Pipeline if you use AWS, or you can choose Azure DevOps as another starting point. Other than this, I use Github Action quite a lot and I also find it quite easy to learn. There are some other tools such as Jenkins, Circle CI and Team City which are all very popular but I would not suggest you start from here as they are either need complex setup or not free.

6 Infrastructure as Code

With IaC, you can easily manage and provision your infrastructure through code instead of manual process. Especially when you have multiple environments for your app, IaC hugely improve the efficiency. I had been using AWS CloudFormation for multiple years but when I started using Terraform, I believe this is what I really want.

I use Terraform to build infrastructure on AWS, Azure as well as Tencent Cloud. There are quite a lot of modules you can use but for real world project, I usually build modules myself. For learning, I would suggest you can use Terraform to build a small project such as three-tier application or static website with CDN service. I am also thinking about make another video about real world Terraform project. You can subscribe my channel when the video is ready. There is a sub domain which is called configuration management. The tool that I recommend is Ansible. It is an easy-to-learn tool so you can take this as lower priority. I actually don’t use it quite often as we are now in containerisation-heavy mode.

7 Coding and Automation

The seventh one is programming which is actually the key part that you implement automation. There is not much to say how to learn coding as there are so many tutorials on the internet. If you have no coding experience, I would suggest you take this as the top priority and learn python as your first the programming language as it is the most popular one for automation and very easy to understand.

Well, size of the community is very important for your learning. However, if you have already know how to code, just pick a language you are most familiar with.

8 Monitoring

This is a very broad topic. The key words are metrics and logs.

In order to understand what to monitor, you need to understand software or cloud architecture. You should know how to monitor the stats or metric such as CPU, memory, Storage, IOPs as well as active connections and errors…etc. You also need to know how to collect logs, if you need to install agent? What types or formats of data you should saved as and what the data lifecycle should be like? Other than these, you need to learn at least one tool to build dashboards. There are many tools in the market, I have used Splunk, DataDog, ElasticSearch, Grafana, PowerBI and Tableau integrated with relational database. Grafana might be your first choice for learning as the free version has very strong capabilities.

If you build your monitoring solution on the Cloud, you can start with CloudWatch for AWS or Azure Monitor for Microsoft Azure. I would recommend CloudWatch for beginners. You can learn how to use it during your journey of learning AWS.

Others

Other than above 8 domains, you’d better have understanding of networking , security and database management. As a DevOps engineer, you should always look at the system or solution from top to button instead of focusing on a specific part. Also you should remember, you will never have learned everything. Just keep yourself with a growing mind and learn the skills with practical exercise at the same time.

I hope you can achieve your career goal in 2023! And make sure you subscrible my youtube channel if you want more valueble content.

Thanks.

Harry in New Zealand

How I passed AWS Certified Solutions Architect Professional

Harry@StartQuick Tech — Sat, 31 Dec 2022 04:33:41 +0000

It was an unforgettable Christmas as I finally cleared the AWS SAP exam at almost the last minute of Year 2022.

It was also a fantastic year for me. I achieved all my goals which I planned at the beginning of the year.

I may just stop here for now and next year, I will spend more time on my Youtube Channel and this Medium Blog on sharing my experience and knowledge on Cloud and IT. I hope I can help more people to start their journey to the Cloud World.

Let’s get back to the exam preparation.

Firstly, I would like to say this is not a short article. It covers all the aspects for the exam at least based on my preparation. Secondly, you cannot pass the exam via just reading this article. Instead, you need to spend lots of time to do document reading and hands-on practise.

Learning Materials

I know you might be interested in the materials that I use. I did two training sessions on Udemy but honestly, I haven’t finished all of them.

If you are beginner who is preparing Solution Architect Associate Exam, you can also use this course from DolfinED for a broader and deeper knowledge. But unfortunately, it is not allowed to enrol on Udemy. I just found this website which has the same content. This course has more than 40 hours videos and it covers everything. But for the experienced AWS engineer, I would not suggest this as it takes too long time to go through the videos and lots of concepts which are a bit boring.

Another course I use is from Neal Davis SAP-02 course on Udemy. It has only 20-hour video and the knowledge is explained quite clearly. Compare with the exam guide, the knowledge which is covered in this course is not 100% but you can definitely pass the exam with high score. Also, I would like to emphasise that this exam is not just testing if you remember the knowledge, instead, it test if you know how to use the knowledge to build a proper solution. Remember, there is no best solution in the world. A better one is always the one you should select.

Other than the training course, you should do some practice exam. I used Jon Bonso’s practice exam on Udemy which cost me just $9.99 when it is in discount. I did three of them but only passed one. But don’t worry, this will not impact your final result in the real exam as the more mistakes you make, the more knowledge you know you misses. The explanations are very in details and I would suggest you to go through the explanation for the correct and incorrect answers no matter you make the right choice or not. If you still don’t understand it, go to AWS official document and do some hands-on as well.

Most importantly, for my last 5-year learning and working on AWS, I read lots of AWS architecture blogs and AWS re:Invent videos on Youtube. These help a lot and I found some of the questions are coming from them which are real customer use case. Also, AWS white-papers are highly recommended! Below two are must-read which I recommended many times in my previous blogs:

Learning Plan

Not like DevOps Engineer Pro exam, I just spend less than a month to prepare this course as I have already got quite a lot of experience on the scope of the exam requirements.

For the first two weeks, I spend 2–3 hours a day to watch the Neal’s training video and do some hands-on at the same time. I ignored some sessions such as S3, EC2, RDS… which I believe I have mastered so it saved me quite a lot of time. But for the knowledge that I was not so familiar with such as Direct Service, Direct Connect, I watched them for multiple times.

For the last 2 weeks, I started to do practice exams. Most of the time, I cannot have a full-3-hour time to do the exam but I tried to do it within a day. And after that, I will spend 2–3 days to go through all the questions and answers. Then go back to the knowledge and do hands-on if I am still not so sure. I use Microsoft OneNote as my notebook to record all the knowledge I know and I missed based on different services.

And before the exam, I will just go through my notes multiple times to remember them. Because in the real exam, you will make the wrong choice just because you don’t know or you forget an unimportant knowledge point. If this happens multiple times in the exam, you might lose your patience and focus.

As I mentioned in below video, you should first know the key concepts of Cloud Compute and the corresponding services on AWS.

Before I walk you through the knowledge and key points, I would suggest you to learn multi-tier/3-tier architecture.

There are four domains for SAP-02 exam based on the exam guide.

Design for organisational complexity
Design for new solutions
Continuously improve existing solutions
Accelerate workload migration and modernisation

But I will not follow this. I will show you the knowledge structure based on my own priorization.

I broke the services into three levels based on the importance and popularities and also point out the key info related to each service. Don’t take this as you knowledge checklist as it does not cover all. But you can go through them and see if you have covered them.

Level 1 (Most Important — Need deep understanding and familiar with hands-on operations)

AWS Organization — For SA Pro, this is a the most critical service which can be connected to different aspects of the Cloud services. More than 1/3 of the questions are related to AWS Organization. You need to know how it works with SCP, SSO, Billing, CloudTrail
Control Tower — on top of AWS organization but I did not see questions related to it in my exam.
Config — Whenever you want to assess, audit or evaluate the configuration of AWS resources.
EC2 — You should know everything about it!!!
Auto Scaling — Well-architected Framework will help you understand more deeply.
ELB — ALB/NLB…what are difference and what are their limitations.
ECS — Know how container works and understand the benefit that Fargate has. Understand how network mode awsvpc works.
Lambda— You should know everything about it!!!
RDS — You should know everything about it!!!
Aurora — You should know everything about it!!!
DynamoDB — You should know everything about it!!!
ElasticCache — Memcached or Redis? Cluster mode enabled or disable? How to scale?
CloudFormation — What is template, stack, stackset, change set? when should you use nested stack? What is the deletion policy for different types of resources?
CloudWatch — Logs, Metrics, Events(EventBridge)…How it is integrated with other services for your monitoring system?
CloudTrail — API events… How to include global services such as CloudFront, Route53?
Systems Manager — Multiple Components included (Automation, Run Command, Inventory, Patch Manager, Session Manager, Parameter Store…). Understand how Maintenance windows works; Understand difference between parameter store and secrets manager; How to enable EC2 access with session manager.
Secrets Manager — Key Rotation!!!
CloudFront — Global Service!!! How to use Lambda@Edge to customize the content that CF delivers? Difference between signed URL and signed cookies? How to set origin and origin group? How geo restriction works?
API Gateway — Most of the time, it works with Lambda functions…
Route53 — Public or Private Hosted Zone? Health Check? Routing Policies? Route53 Resolver…difference between inbound and outbound Endpoint?
VPC — Everything related to Cloud Networks!!!
ACM — Public or Private? Used on EC2? Any limitation for using it on CloudFront distribution?
KMS — How it is integrated with other AWS services such as S3
Cognito — User Pool vs Identity Pool? Very confusing name but totally different concepts.
IAM — You should know everything about it!!!
S3 — You should know everything about it!!!
EBS — How it is different from S3 and EFS
EFS — How it is different from S3 and EBS
Storage Gateway — One diagram help you understand everything.

Kinesis — Data stream, Data analytics, Firehose… Video Stream?
STS — AssumeRole Process.
SNS — Fanout pattern
SQS — How to Decouple you system. Standard vs FIFO? Long vs Short Polling?

Level 2 (Important — Need deep understanding with less hands-on experience)

Cost Explorer — Understand how to get cost information.
Budget
Batch — How batch job works? Where it can be run on? What types of instances are most suitable?
Elastic Beanstalk — Deployment modes
Redshift — Only use it when you need data warehouse.
Developer Tools (CodeCommit/CodeBuild/CodeDeployCodePipeline)
IoT — Understand how AWS IoT Manage the Core, devices, edgeCompute, Analytics as well as security
Service Catalog — Understand the different types of the constraints such as launch constraints, notification constraint, template constraint? How to manage the permissions via IAM?
Global Accelerator — what is the benefit? Compare with CloudFront? Compare with S3 Accelerator?
Transit gateway — Full Mesh
VPN — Differnce vs Direct Connect? Customer gateway vs Virtual private Gateway?
Direct Connect — Private vs Public VIF? Multi-Region Case? Site-2-Site VPN for redundant connection?
Directory Service — How it works with Microsoft AD or Azure AD? What is AD connector?
WAF/Shield — Difference? Shield is for DDoS only but very expensive. Think about alternative and cheaper ways?
SES — Send customized email
Resource Access Manager — Share resources…what resources can be shared?
Migration Services (Application Discovery Service, Application Migration service, Database Migration Service, Server Migration Service, DataSync, Migration Hub) — 7 Rs of Migration: Refactor/Replatform/Repurchase/Rehost/Relocate/Retain/Retire
SCT — Used with database migration process
Snow — Difference among Snowball, Snowball Edge and Snowmobile?
Glue — ETL
Athena — Serverless SQL
EMR — managed cluster for big data framework (Apache Hadoop and Spark). What are the node types…master, core, task nodes? For one-off run task, how to choose instance purchasing options?

Level 3 (Less Important — Just need to understand the concept and use cases)

EKS
Data Pipeline
Data Exchange
Lake Formation
OpenSearch
QuickSight
AppFlow
AppSync
Step Function
SWF
Lightsail
Outpost
PrivateLink
CloudHSM
DocumentDB
Timestream
X-Ray
AppStream 2.0
Workspaces
Amplify
Device Farm
Machine Learning Services (Rekognition, Transcribe, Translate, SageMaker, Comprehend, Lex, Polly, Textract, Kendra)
Health Dashboard
Trusted Advisor
Well-Architected Tool
Network Firewall
GuardDuty
Inspector
Macie

Finally

Hope this is helpful for your preparation and wish you good luck on your exam. Please follow me if you want to get yourself notified.

Feel free to comment and your questions are always welcomed.

Also, my new Youtube Channel — StartQuick Tech is in preparation. This channel is used to share all my experience on tech and I may start from Cloud Knowledge. I would appreciate it if you can subscribe. I believe you can get lots of value from it.

Thanks.

Harry Z @ NZ

How to copy S3 objects across different AWS Accounts

Harry@StartQuick Tech — Sun, 18 Dec 2022 10:32:46 +0000

Copying S3 files across different AWS accounts happens quite often if you manage multiple AWS accounts no matter for different teams, products or environments. It is not a complex setup but it shows how AWS perfectly design and operate cross-account permissions.

I created below video for the hands-on process. If you find it useful, please help to like and subscribe my channel for future videos

There are two main steps regarding the permission setup:

S3 Bucket policy: All root access from the destination account;

Below is the bucket policy. Destination account ID and source bucket name need to be replaced.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowCopy",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::[destination_account_id]:root"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::[source_bucket_name]/*",
                "arn:aws:s3:::[source_bucket_name]"
            ]
        }
    ]
}

IAM User/Role: Entity with required permissions to execute the copy/sync action.

You need to create a policy with below permissions and attach it to the IAM role or user which will execute the copy/sync action.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::[source_bucket_name]",
                "arn:aws:s3:::[source_bucket_name]/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::[destination_bucket_name]",
                "arn:aws:s3:::[destination_bucket_name]/*"
            ]
        }
    ]
}

Source bucket name and destination bucket name need to be replaced. From above policy we can see, we need to allow GetObject from the source bucket and PutObject to the destination bucket which is for the copying behaviour.

In this article, I will use AWS CLI to execute the copy/sync action.

Below are examples and the region parameter is actually optional.

# copy file from source to destination
aws s3 cp s3://source_bucket_name/filename s3://destination_bucket_name/filename --source-region [source_bucket_region]
# sync source foloder to destination folder
aws s3 sync s3://source_bucket_name s3://destination_bucket_name --source-region [source_bucket_region]

Now you should know how to do the cross-account s3 objects copying.

Thanks.

Harry

How to Access the Containers on AWS Fargate Cluster

Harry@StartQuick Tech — Mon, 14 Nov 2022 11:06:24 +0000

We have lots of containerised applications running on AWS Fargate Cluster and recently, more and more requirements from our dev team jumped out and they would like to access the containers directly for trouble shooting purpose.

AWS Fargate is a serverless compute engine for containers and you can just deploy your containerised application without managing the servers. However, without servers, you cannot just easily run docker exec -it to access the container. You need AWS Session Manager’s help to build a channel for you. See below diagram.

Workflow Diagram

From configuration perspective, you need to have AWS CLI v2 and session-manager-plugin installed on your machine. You also need to give the ecs:ExecuteCommand permission to your IAM role (if you use EC2 as a bastion host) or IAM user (if you configure AWS credential). Lastly, you need to give SSM permission to the Task Role.

Let’s go through the procedure step by step:

I assume you have got existing service/tasks running on Fargate.

Install AWS Cli v2, please follow the AWS document.
Install the session-manager-plugin, please follow this.
Update the task role with below permissions. Please read this to understand what these actions are

{
   "Version": "2012-10-17",
   "Statement": [
       {
       "Effect": "Allow",
       "Action": [
            "ssmmessages:CreateControlChannel",
            "ssmmessages:CreateDataChannel",
            "ssmmessages:OpenControlChannel",
            "ssmmessages:OpenDataChannel"
       ],
      "Resource": "*"
      }
   ]
}

Update the IAM role or IAM user for your connecting machine. If you have Administrator with Allow *, you can skip this step. See below permission.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ecs:ExecuteCommand",
            "Resource": "*"
        }
    ]
}

The policy above only gives you ExecuteCommand permission which you can associate it to your dev team user or group. But for updating the service with — enable-execute-command as an admin, I would suggest you can use below policy for an admin user to manage the ECS/Fargate Cluster.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ecs:*",
            "Resource": "*"
        }
    ]
}

(Optional) Check the task if executeCommand is enabled or not.

aws ecs describe-tasks --cluster <cluster-name> --task <task-id>

Enable ECS executeCommand

aws ecs update-service --cluster <cluster-name> --task-definition <task-definition-name> --service-name <service-name> --enable-execute-command

If you have forceNewDeployment enabled, you just need to wait the new task is up. If you want it urgently and the container is not a critical one, you can stop the task manually and the new task will be created.

Verify if executeCommand is enabled. The task-id will be changed.

aws ecs describe-tasks --cluster <cluster-name> --task <task-id>

aws ecs execute-command --cluster <cluster-name> --task <task-id> --container <container-name> --interactive --command "/bin/sh"

Best Practice

Honestly, I would not recommend to access the Fargate container especially for production environment. Our team is also build a very practical way for developers to trouble shoot on their local environment.

We have been using Docker Compose to build the local dev environment for quite a long time and it is running perfectly for trouble shooting.

Hope this is helpful! If you have any questions about this, feel free to leave your comments.

Harry in NZ

How Do I Pass Microsoft Azure AZ-305 (Solution Architect Expert)

Harry@StartQuick Tech — Fri, 04 Nov 2022 08:42:49 +0000

I have cleared AZ-305 exam a couple of weeks ago and finally got time to write my learnings. Hope it can help you to achieve this certificate if this is your goal.

I started to use Azure a few years ago but in a very light mode. For my current role, as we have a client whose platform is built on Azure, I started to immerse myself in it. I have learned quite a lot from the project but for clearing the exam, it is definitely not enough.

I am a DevOps/SRE engineer and have almost 5-year experience on Cloud and System Architecture. So for the well-architected framework, I have not spent too much time so I mainly focus on functions/features of each Azure services and how it can provide users with well-architected solutions. If you have not done any Cloud Architecture project, I would suggest you to read and learn it. The resource can be used as below:

Regarding the materials I used, I would highly recommend the official Microsoft online training material for this exam. You don’t have to go through every section if you already have experience on them but the material could definitely help you fill the gaps of your knowledge base.

I would not copy-paste the exam instruction to waste your time on this article. Below I listed the key knowledge I think the most important, which I spent lots of time to learn and do hands-on. I would like to emphasise there is no better way to learn Azure other than hands-on exercise.

Azure AD

It covers quite a lot of knowledge. You need to apply the 30-day free trial on the premium license to learn the advanced features such as PIM(Privileged Identity Management), access review…

Also, you need to understand the difference between AD conditional access policy and Azure Policy. These names can make people very confused.

Another key point for AD is to understand how the domain service works, especially for integration with on-premise environment.

For IAM, you need to know RBAC, managed identities (system-assigned and user-assigned), AD application… If you have some experience on AWS like me, you may need to spend more time to understand the difference…

Governance and Monitoring

For governance, the key thing is to understand the Azure management hierarchy. See below diagram, you need to understand what these levels mean and what use cases they fit into differently.

Other than this, as a solution architect, you need to know how to design Blueprints and Landing zones. The goal of Blueprints is to maintain consistency and compliance for your organisation during deploying cloud resources. It includes not only resources but also policies, roles…(Different from ARM template). For Blueprints and Landing zones, do some exercise if you have enough time.

For monitoring, the keywords are logs and metrics. You need to know where these two things are generated and stored and presented. Make sure you know application insights, diagnostic settings, log analytics, Azure monitor and log agent. There are quite a lot of details for each.

Multi-Tier Application

This is not a key word in AZ-305 but you do need to make sure you fully understand this concept. Understanding how to design a multi-tier application is the foundation for a solution architect. It requires you to have comprehensive knowledge such as compute, network, database, storage, system resiliency, scalability, disaster recovery, security… Any of these include exam knowledge.

For Compute, you need to know how to create a proper VM solution, or use Function App(Serverless) or Batch service for automation. Also, if containerised system would give you more flexibility or performance, etc…

For network, you need to know what are virtual network, subnet, peering, how to connect on-premise network with Cloud (VPN, express route), how to securely provision your resources within the private network, how to make the connection resilient…

For Database and Storage, you need to know different types of database such as relation database(SQL, Elastic Pool), NoSQL(DynamoDB, Table Storage), Blob(Object) and File Storage, Data Lake. Always think about the availability, scalability and COST (I did not take this as a section but it is very important in the exam)… Also, there are some data integration services you need to know: Data Factory, , DataBricks, Import/Export…

For Security, this is a very big topic. What I suggest for quickly checking what you know is to go through these parts:

Identity: AD…AD…AD…Very Important.
Encryption: think about data in rest and in transit
Network: NSG, ASG, WAF, NAT, Private subnet…

Some other key services I would to emphasise here which include Load balancer, Front Door, API management… If you still don’t know what these services are and what their features are, you might not be ready for the exam.

Migration

Migration solution is also very important for a solution architect as more and more on-premise applications are migrated to Cloud these days. I would suggest you can go through this.

Design Migrations

Learning Materials

Other than the official training and Microsoft Azure documents, I would like to recommend some other materials from my own perspective.

For the practice exam, most of the answers are correct but please don’t always trust the answers. Think about them and try find answers from Microsoft and you can ask questions to the course owner as well and they are always fast responded.

Conclusion

Honestly, compared with AWS DevOps Professional, I don’t think this exam is too difficult. However, hands-on exercise is very important if you don’t have experience on all or some of the services.

Hope this could point you to the right direction for your exam preparation. If you think it helpful, please follow me and you will see my next following tech blogs.

Thanks.

Harry in NZ

How I cleared AWS Certified DevOps Engineer — Professional

Harry@StartQuick Tech — Sun, 24 Apr 2022 05:54:56 +0000

How I passed AWS Certified DevOps Engineer — Professional

It was an exciting day for me as I passed the AWS Certified DOP exam. It was the toughest exam that I have experienced and I had been preparing it for more than two months.

I have to say thank you to the contributors of the learning resources from AWS, YouTubers and Blog Writers. Special thanks for the course instructed by Stephan Maarek on Udemy with lots of hands-on videos.

I have 4+ years experience on AWS but during preparation for this exam, I still learned a lot. The key thing (difference from associate level) for this exam is that it requires lots of hands-on experience. There are quite a lot of tools and services on AWS I had never played with but the best way to understand it is to USE it.

I spent a lot of time on Elastic Beanstalk to deploy web applications and integrated it with AWS developer tools such as CodeDeploy/Code Pipeline for automating and testing the deployment strategies. Also, during the learning, I read the article for Golden AMI pipeline. Instead of just reading it through, I did build it myself using SSM and Inspector. I believe even if you work in the Cloud domain for very long time, you are still possibly not be able to know all the services that the exam requires. Do hands-on as much as possible, and you will finally find it easy and get it well-prepared.

There are six domains for this exam and it covers quite a lot of services as below. Make sure you know all the services included and understand how these services work together to solve practical problems.

AWS-Certified-DevOps-Engineer-Professional_Exam-Guide

SDLC Automation

AWS Developer Tools (CI/CD)— CodeCommit, CodeBuild, CodeDeploy and CodePipeline (Very Important)

CodeBuild can be used as Jenkins slave(worker) node

Configuration Management and Infrastructure as Code

CloudFormation (Very Important. Difference between nested stack and cross-stack reference)

Elastic Beanstalk (Very Important. Deployment methods: pros and cons)

OpsWorks (I have done too much hands-on on it)

Lambda and API Gateway (Understand how serverless works and benefit and limitation)

Elastic Container Service (Very big topic and lots of concepts)

IAM (Very Important)

Monitoring and Logging

CloudWatch (Very Important. Metrics/Logs/Events/Alarms…)

CloudTrail (Very Important.)

Kinesis

OpenSearch (successor of Amazon ES. Just use this when searching required)

Tagging

X-Ray (Understand what it is used for)

Policies and Standards Automation

Systems Manager (Very Important.)

Config (Compliance and Non-Compliance)

Service Catalog

Inspector

Health

Trust Advisor

GuardDuty

Macie

Secret Manager (Comparison with SSM Parameter Store)

Incident and Event Response

CloudWatch (AGAIN!!!! Very Important! Work with a lot of services)

High Availability, Fault Tolerance and Disaster Recovery

Auto-Scaling Group (Understand Multi-tier Architecture)

Elastic Load Balancer

DynamoDB

RDS (Aurora is special!!! Please don’t ignore it.)

AWS Organization

Learning Materials

The course I use is on Udemy which is instructed by Stephan Maarek. There are lots of hands-on demos in this course that gave me a very straight-forward view on how to use the services.

I read a lot of AWS documents and mainly focus on the white papers and service FAQs. And Jayendra’s blog is getting much more popular!

AWS provides free courses on AWS Skill Builder for this exam. Please do take it as an add-on or a knowledge check. You will get nothing to lose.

Practice test is also very necessary from my own perspective. I have done two tests on AWS Certified DevOps Engineer Professional Practice Exams from Jon Bonso and one test on Practice Exam | AWS Certified DevOps Engineer Professional from Stephan Maarek. I failed all the three tests for the first try and could not have done it within 3 hours. However, I took these questions as my checklists and deep dive into all of them and understand why the answers are correct and the rest are incorrect.

There are always multiple ways to solve one problem, but as a professional DevOps engineer, you need to find out the easiest, the most cost-optimised, the most secured or most efficient one, etc.

Finally

Hope this is helpful for your preparation and wish you good luck on your exam.

I will prepare another blog about the notes I made during my preparation soon. Please follow me if you want to get yourself notified.

Feel free to comment and your questions are always welcomed.

Harry Z @ NZ

Is LoRa Bridge a Good Idea to Improve Coverage

Harry@StartQuick Tech — Thu, 24 Jun 2021 01:27:40 +0000

In the IoT(Internet of Things) World, everyone should be familiar with LoRa Technology which is first invented in 2009 by Nicolas and Oliver in France and the company Cycleo they established was acquired by Semtech in 2012.

LoRa is a spread spectrum modulation technique derived from chirp spread spectrum (CSS) technology which can meet most of the requirements for IoT with Long Range and Low Power. It is quite flexible for rural and indoor use cases and has been utilised in smart cities, smart homes and smart agriculture, etc.

LoRa Fills a Technology Gap

This article is not to describe what LoRa is and what role LoRa plays in IoT. You could almost get everything about LoRa from Semtech website.

Instead, I would like to say LoRa has its own limitations. Similar to Cellular base stations, LoRa Gateway also needs to have LAN, Cellular, WIFI or Satellite as connection to the network server for LoRaWAN solution.

This brings in a big problem — Coverage.

As far as I know, the theoretic maximum range of existing LoRa Gateway can reach to 15–20 km. Wow, sounds brilliant! But thinking about more use cases, can LoRa be used in the further sea to manage the ship fleet or in the Amazon rainforest to monitor the ecosystem? The answer might be NO as 20km is not large enough even if we don’t think about the other factors that will limit the range of LoRa coverage.

LoRa Bridge

Simply speaking, LoRa Bridge is to provide connection to LoRa Gateway with other Gateways.

See below, there are multiple sensors on each ship but these ships are too far away from the LoRa Gateway. Due to the connection and some other factors, the gateway can only be located at the edge of the cellular coverage.

Sensors on the ships are out of LoRa gateway coverage

How are these further sensors talking to the gateway?

If the Gateway has capability of transmitting data via RF between Gateways, the problem will be solved easily. This is why we need LoRa Bridge feature/function on the LoRa gateways.

See below example, we have a LoRa Gateway installed at a fixed place in the sea (or it can be installed on the ship). All the sensors are covered under it and the data can be transferred via LoRa Bridge to the Gateway which has LAN/Cellular connection. Problem Solved!

Conclusion

LoRa Bridge is not a physical device but a feature/function on the gateway. There are still a lot of questions need to be discussed such as tagging the gateway with type of connection or assigning different roles on the gateways for inter-gateway communication.

With LoRa Bridge, the bigger coverage is built by LoRa Gateway itself instead of expanding network with huge amount of cost.

Author:

My name is Harry Zhou who has been working in Telco Industry for more than 10 years. I am an enthusiast on new technology especially IoT. Welcome to connect me via LinkedIn and let us make the world better via IoT!