DEV Community: Troy

AWS Certified Security: Specialty recap

Troy — Fri, 11 Dec 2020 02:18:14 +0000

AWS Certified Security: Specialty

Overview

The AWS Certified Security: Specialty is geared towards individuals who focus primarily on security within AWS. I found that as a solutions architect, it is extremely important to have security at the forefront of my mind while building. Building an environment secure initially reduces the risk of failing compliance and potential exploitation at production launch.

Exam components

I found that an exam-taker should focus heavily on the following resources:

Exam components (AWS Documentation)

AWS KMS

AWS Key Management Service

- FAQ

AWS IAM

AWS Identity and Access Management (IAM)

- FAQ

Amazon EC2

Amazon Elastic Compute Cloud

- FAQ

AWS Cloudtrail

AWS Cloudtrail

- FAQ

Amazon CloudWatch Events

Amazon CloudWatch Events

Security documentation

- Quotas (Limits)

Amazon Inspector

Amazon Inspector

- FAQ

AWS WAF

AWS Web Application Firewall

- FAQ

AWS Shield

AWS Shield

- FAQ

AWS Config

AWS Config

- FAQ

AWS Organizations

AWS Organizations

- FAQ

Training Resources

The following training resources helped prepare for the above components:

Future Reading

Please come join me over at my personal site, in which I cover varying AWS related topics and more!

Using aws-auto-cleanup to keep an AWS test account neat & tidy (and minimal cost!)

Troy — Tue, 22 Sep 2020 01:56:36 +0000

aws-auto-cleanup

Using aws-auto-cleanup to keep an AWS test account neat & tidy (and minimal cost!)

Functional Requirements

Reduce operational run-time of resources used for a testing\development, or for temporary spin-up of resources
Reduce cost\operational expenses
Ability to whitelist AWS resources that need to be retained

Operating Cost

< $2.00/mo for the following:

AWS::Events::Rule
AWS::Lambda::Function
AWS::Lambda::Permission
AWS::Logs::LogGroup
AWS::IAM::Role
AWS::S3::Bucket
AWS::S3::Bucket
AWS::DynamoDB::Table
AWS::DynamoDB::Table

Node	Type	Unit	Usage	Total
auto-cleanup-settings-prod (DynamoDB-Table)	provisioned read	$0.00013 per hour for units of read capacity	1 units per month	$0.10
auto-cleanup-settings-prod (DynamoDB-Table)	provisioned write	$0.00065 per hour for units of write capacity	1 units per month	$0.48
auto-cleanup-settings-prod (DynamoDB-Table)	storage	$0.25 per GB-month	1 GB	$0.25
auto-cleanup-whitelist-prod (DynamoDB-Table)	provisioned read	$0.00013 per hour for units of read capacity	1 units per month	$0.10
auto-cleanup-whitelist-prod (DynamoDB-Table)	provisioned write	$0.00065 per hour for units of write capacity	1 units per month	$0.48
auto-cleanup-whitelist-prod (DynamoDB-Table)	storage	$0.25 per GB-month	1 GB	$0.25
auto-cleanup-prod (Lambda-Function)	1000	128MB * AvgTime * Invocations per month (Usage)	10000 invocations	$0.02
ServerlessDeploymentBucket (S3-Bucket)	Standard	$0.023 per GB - first 50 TB / month of storage used	10 GB	$0.23
auto-cleanup-prod-resourcetreebucket-troydieter (S3-Bucket)	Standard	$0.023 per GB - first 50 TB / month of storage used	10 GB	$0.23
Dynamo Storage	Discount	First 25GB Free (-$0.25 per GB)	-2	($0.50)
Dynamo Backup	Point In Time Recovery	$0.20 per GB-month (Continuous backups)	0	$0.00
Dynamo Backup	OnDemand	$0.10 per GB-month (On Demand)	0	$0.00
Dynamo Network Outbound Traffic	Transfer	$0.09 per GB (Data Transfer Out)	1	$0.09
Lambda Service	Number of invocations	Invocation call for a Lambda function	10000	$0.00

Diagram

Stack

Serverless Stack Framework Core: 2.1.1 Plugin: 4.0.4 SDK: 2.3.2 Components: 3.1.3
AWS CLI
NPM

Preferred configuration

After you clone the aws-auto-cleanup repository in the next step, you will have the opportunity to change the parameters of the serverless-stack deployment. A few notable changes:

Change the serverless.yml file line 4, to your company name or project name.
Within the auto_cleanup/data/auto-cleanup-settings.json file, you will find the default parameters for the function. I have (obviously) found that the TTL (time-to-live) of 7 days to be too low. I have raised that to 120 days. Example:
```
    "ttl": {
      "N": "120"
    }
```

Deploy

Following the directions listed on the README, over at the aws-auto-cleanup Github page!

Useful commands

serverless deploy --region region-example --aws-profile profile-example

serverless invoke --function AutoCleanup --region region-example --aws-profile profile-example --type Event

Outputs

Under CloudWatch, you'll see under Logs > Log Groups the following log group (if you kept the default Lambda function name):

/aws/lambda/auto-cleanup-prod

With a log stream of the latest Serverless invocation (or scheduled Lambda execution):

| 1600638849212 | [INFO] S3 Bucket
'outbound-email-send-dev-serverlessdeploymentbucke-46346' was created
234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849212 | [INFO] S3 Bucket
'outbound-email-send-dev-serverlessdeploymentbucke-46346' was created
234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849212 | [INFO] S3 Bucket
'outbound-email-send-hand-serverlessdeploymentbuck-46346' was created
234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849212 | [INFO] S3 Bucket
'outbound-email-send-hand-serverlessdeploymentbuck-46346' was created
234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849212 | [INFO] S3 Bucket
'46346-ai-chat-bot-dev-serverlessdeploymentbucket-46346' was created
230 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849268 | [INFO] S3 Bucket '46346-artifacts' was created 505
days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)

| | 1600638849269 | [INFO] S3 Bucket '46346-cpds-infra' was created
494 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849269 | [INFO] S3 Bucket '46346-sree52-testbkt' was
created 430 days ago and has been deleted. (s3_cleanup.py, buckets(),
line 168)

| | 1600638849269 | [INFO] S3 Bucket 'python-myservice-bucket' was
created 158 days ago and has been deleted. (s3_cleanup.py, buckets(),
line 168)

| | 1600638849269 | [INFO] S3 Bucket 'python-poc-deployment-bucket'
was created 166 days ago and has been deleted. (s3_cleanup.py,
buckets(), line 168)

| | 1600638849269 | [INFO] S3 Bucket 's3-to-sns' was created 264 days
ago and has been deleted. (s3_cleanup.py, buckets(), line 168)

| | 1600638849269 | [INFO] S3 Bucket
'serverless-telegram-bot-serverlessdeploymentbuck-46346' was created
236 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849269 | [INFO] S3 Bucket 'swagger-bucket-1' was created
199 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849269 | [INFO] S3 Bucket 'terraform-bucket-lokesh' was
created 243 days ago and has been deleted. (s3_cleanup.py, buckets(),
line 168)

| | 1600638849270 | [INFO] S3 Bucket
'twilio-voice-test-dev-serverlessdeploymentbucket-46346' was created
230 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

Performing clean-up

Change the dry_run item in the DynamoDB table (auto-cleanup-settings-prod - if you kept the default settings name) to false to perform clean-up:

{ "dry_run" : { "BOOL" : false }}
1. Invoke using:

serverless invoke --function AutoCleanup --region region-example --aws-profile profile-example --type Event

Recap

aws-auto-cleanup helps to keep multi-region AWS accounts clean. Resources are either white-listed, or deleted after specified amount of days. This reduces monthly expenses and conflicting resources within the same region!

sharedsecret - an AWS-native secret sharing service

Troy — Tue, 04 Aug 2020 05:16:32 +0000

Goal
Create an end-to-end fully encrypted, publicly accessible secret storage tool

Components

AWS Code Commit (Ruby v2.7) - a fork of OneTimeSecret with additional features, more developed UX & mobile capabilities.
AWS Elastic Beanstalk (which includes EC2 instances deployed with boot-strap configurations defined in the beanstalk configuration)
AWS S3 (Terraform Remote State, misc. components)
AWS Elasticache (v3.2.10)
AWS ELB (Application Load Balancer)
AWS ACM (Amazon Certificate Manager)
AWS Route 53 (DNS Management)

Deployment Model
The goal of the deployment is to deploy all initial platform components through Terraform. Due to the complexity/work-exerted model for Elastic Beanstalk, those components may be deployed manually.

Deployment Overview

Feedback
Looking for general feedback - not looking to monetize.

AWS Certified Big Data: Specialty study blueprint

Troy — Tue, 31 Dec 2019 15:43:05 +0000

AWS Certified Big Data: Specialty study outline

In another installment of study blueprints for AWS certification exams; I am happy to provide my suggested outline for what I used to pass the AWS Certified Big Data Specialty certification in December 2019.

Deep-dive re:Invent videos:

White Papers:

Big Data Analytics Options on AWS (January 2016) PDF
Streaming Data Solutions on AWS with Amazon Kinesis (July 2017) PDF
Data Warehousing on AWS (March 2016) PDF
Best Practices for Amazon EMR (August 2013) PDF

Study Guide:

EMR

EMR security :

Data at rest

Data residing on Amazon S3—S3 client-side encryption with EMR
Data residing on disk—the Amazon EC2 instance store volumes (except boot volumes) and the attached Amazon EBS volumes of cluster instances are encrypted using Linux Unified Key System (LUKS)

Data in transit

Data in transit from E,MR to S3, or vice versa—S3 client-side encryption with EMR
Data in transit between nodes in a cluster—in-transit encryption via Secure Sockets Layer (SSL) for MapReduce and Simple Authentication and Security Layer (SASL) for Spark shuffle encryption
Data being spilled to disk or cached during a shuffle phase—Spark shuffle encryption or LUKS encryption

https://aws.amazon.com/blogs/big-data/secure-amazon-emr-with-encryption/

EMR Consistent View: EMRFS consistent view is an optional feature available when using Amazon EMR release version 3.2.1 or later. Consistent view allows EMR clusters to check for list and read-after-write consistency for Amazon S3 objects written by or synced with EMRFS. When you create a cluster with consistent view enabled, Amazon EMR uses an Amazon DynamoDB database to store object metadata and track consistency with Amazon S3. If consistent view determines that Amazon S3 is inconsistent during a file system operation, it retries that operation according to rules that you can define. By default, the DynamoDB database has 500 read capacity and 100 write capacity. You can configure read/write capacity settings depending on the number of objects that EMRFS tracks and the number of nodes concurrently using the metadata.
Use Apache Zeppelin as a notebook for interactive data exploration.Apache Zeppelin is an open source GUI which creates interactive and collaborative notebooks for data exploration using Spark. You can use Scala, Python, SQL (using Spark SQL), or HiveQL to manipulate data and quickly visualize results.
The Ganglia open source project is a scalable, distributed system designed to monitor clusters and grids while minimizing the impact on their performance. When you enable Ganglia on your cluster, you can generate reports and view the performance of the cluster as a whole, as well as inspect the performance of individual node instances. Ganglia is also configured to ingest and visualize Hadoop and Spark metrics.
Apache Cassandra and Apache HBase are columnar Database. Compare to Dynamo DB, Apache HBase is much more flexible in terms of what you can store (size and data type wise). Apache HBase gives you the option to have very flexible row key data types, whereas DynamoDB only allows scalar types for the primary key attributes. DynamoDB, on the other hand, provides very easy creation and maintenance of secondary indexes, something that you have to do manually in Apache HBase.
In EMR, The default input format for a cluster is text files with each line separated by a newline (\n) character, which is the input format most commonly used. If your input data is in a format other than the default text files, you can use the Hadoop interface toInputFormat specify other input types.
If you are using Hive, you can use a serializer/deserializer (SerDe) to read data in from a given format into HDFS.
Hue (Hadoop User Experience) is an open-source, web-based, graphical user interface for use with Amazon EMR and Apache Hadoop. Hue groups together several different Hadoop ecosystem projects into a configurable interface. Hue acts as a front-end for applications that run on your cluster, allowing you to interact with applications using an interface that may be more familiar or user-friendly. The applications in Hue, such as the Hive and Pig editors, replace the need to log in to the cluster to run scripts interactively using each application's respective shell. After a cluster launches, you might interact entirely with applications using Hue or a similar interface.
Amazon EMR supports Apache Mahout, a machine learning framework for Apache Hadoop. Mahout is a machine learning library with tools for clustering, classification, and several types of recommenders, including tools to calculate most-similar items or build item recommendations for users. Mahout employs the Hadoop framework to distribute calculations across a cluster, and now includes additional work distribution methods, including Spark.
You can run your EMR cluster as a transient process: one that launches the cluster, loads the input data, processes the data, stores the output results, and then automatically shuts down. This is the standard model for a cluster that performs a periodic processing task. Shutting down the cluster automatically ensures that you are only billed for the time required to process your data.
Spark has micro-batching but can guarantee only-once-delivery if configured. Spark's four modules are MLlib, SparkSQL, Spark Streaming, and GraphX.
When your cluster runs, Hadoop creates a number of map and reduce tasks. These determine the number of tasks that can run simultaneously during your cluster. Run too few tasks and you have nodes sitting idle; run too many and there is significant framework overhead.
If you want that data to be encrypted in-transit between nodes, then Hadoop encrypted shuffle has to be setup. Encrypted Shuffle capability allows encryption of the MapReduce shuffle using HTTPS. When you select the in-transit encryption checkbox in the EMR security configuration, Hadoop Encrypted Shuffle is automatically setup for you upon cluster launch
Do not use Spark for batch processing. With Spark, there is minimal disk I/O, and the data being queried from the multiple data stores needs to fit into memory. Queries that require a lot of memory can fail. For large batch jobs, consider Hive. Also avoid using Spark for multi-user reporting with many concurrent requests.

RedShift

When you create a table, you designate one of three distribution styles; EVEN, KEY, or ALL.
Even distribution

The leader node distributes the rows across the slices in a round-robin fashion , regardless of the values in any particular column. EVEN distribution is appropriate when a table does not participate in joins ** or when there is not a clear choice between KEY distribution and ALL distribution. **EVEN distribution is the default distribution style.

Key distribution : The rows are distributed according to the values in one column. The leader node will attempt to place matching values on the same node slice. If you distribute a pair of tables on the joining keys, the leader node collocates the rows on the slices according to the values in the joining columns so that matching values from the common columns are physically stored together.
*ALL distribution : A copy of the entire table is distributed to every node. * Where EVEN distribution or KEY distribution place only a portion of a table's rows on each node, ALL distribution ensures that every row is collocated for every join that the table participates in.
ALL distribution multiplies the storage required by the number of nodes in the cluster, and so it takes much longer to load, update, or insert data into multiple tables. ALL distribution is appropriate only for relatively slow-moving tables; that is, tables that are not updated frequently or extensively. Small dimension tables do not benefit significantly from ALL distribution, because the cost of redistribution is low.
RedShift UNLOAD: Unloads the result of a query to one or more files on S3, using Amazon S3 server-side encryption (SSE-S3). You can also specify server-side encryption with an AWS Key Management Service key (SSE-KMS) or client-side encryption with a customer-managed key (CSE-CMK).You can manage the size of files on Amazon S3, and, by extension, the number of files, by setting the MAXFILESIZE parameter.
Number of the slice in Redshift Node

o ds2.xlarge , dc1.large, dc2.large = 2

o ds2.8xlarge, dc2.8xlarge = 16

o dc1.8xlarge = 32

For Redshift COPY command, Split your load data files so that the files are about equal size, between 1 MB and 1 GB after compression. The number of files should be a multiple of the number of slices in your cluster.
To validate the data in the Amazon S3 input files or Amazon DynamoDB table before you actually load the data in Redshift, use the NOLOAD option with the COPY command. Use NOLOAD with the same COPY commands you use to actually load the data. NOLOAD checks the integrity of all of the data without loading it into the database. The NOLOAD option displays any errors ins same manner as that would occur if you had attempted to load the data.
There are two types of snapshots: automated and manual. Amazon Redshift stores these snapshots internally in Amazon S3 by using an encrypted Secure Sockets Layer (SSL) connection. If you need to restore from a snapshot, Amazon Redshift creates a new cluster and imports data from the snapshot that you specify. Read more - https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-snapshots.html
You can use a manifest to ensure that the COPY command loads all of the required files, and only the required files, for a data load. Instead of supplying an object path for the COPY command, you supply the name of a JSON-formatted text file that explicitly lists the files to be loaded. The URL in the manifest must specify the bucket name and full object path for the file, not just a prefix. You can use a manifest to load files from different buckets or files that do not share the same prefix.
RedshiftCopyActivity : Copies data from DynamoDB or Amazon S3 to Amazon Redshift. You can load data into a new table, or easily merge data into an existing table. In addition, RedshiftCopyActivity let's you work with an S3DataNode, since it supports a manifest file.
In Redshift, You can efficiently add new data to an existing table by using a combination of updates and inserts from a staging table. While Amazon Redshift does not support a single merge, or upsert, command to update a table from a single data source, you can perform a merge operation by creating a staging table
Redshift cluster resizing - https://docs.aws.amazon.com/redshift/latest/mgmt/rs-resize-tutorial.html
EXPLAIN command Displays the execution plan for a query statement without running the query in Redshift
By default, Amazon Redshift configures one queue with a concurrency level of five, which enables up to five queries to run concurrently, plus one predefined Superuser queue, with a concurrency level of one. You can define up to eight queues. Each queue can be configured with a maximum concurrency level of 50. The maximum total concurrency level for all user-defined queues (not including the Superuser queue) is 50.
In Redshift , If you enable SQA, you can reduce or eliminate workload management (WLM) queues that are dedicated to running short queries. In addition, long-running queries don't need to contend with short queries for slots in a queue, so you can configure your WLM queues to use fewer query slots. When you use lower concurrency, query throughput is increased and overall system performance is improved for most workloads.
In RedShift , Short query acceleration (SQA) prioritizes selected short-running queries ahead of longer-running queries. SQA executes short-running queries in a dedicated space, so that SQA queries aren't forced to wait in queues behind longer queries. With SQA, short-running queries begin running more quickly and users see results sooner.
In Redshift The CASE expression is a conditional expression, similar to if/then/else statements found in other languages. CASE is used to specify a result when there are multiple conditions.
In Redshift , You enable encryption when you launch a cluster. To migrate from an unencrypted cluster to an encrypted cluster, you first unload your data from the existing, source cluster. Then you reload the data in a new, target cluster with the chosen encryption setting. During the migration process, your source cluster is available for read-only queries until the last step. The last step is to rename the target and source clusters, which switches endpoints so all traffic is routed to the new, target cluster
Amazon Redshift logs information about connections and user activities in your database. These logs help you to monitor the database for security and troubleshooting purposes, for database auditing. The logs are stored in the S3 buckets for convenient access with data security features for users who are responsible for monitoring activities in the database.
S3, DynamoDB, and EMR/EC2 instances directly integrate with Redshift using the COPY command.
The UNLOAD ENCRYPTED command automatically stores the data encrypted using client side encryption and uses HTTPS to encrypt the data during the transfer to S3.If you want to ensure files are automatically encrypted on S3 with server-side encryption, no special action is needed. The unload command automatically creates files using Amazon S3 server-side encryption with AWS-managed encryption keys (SSE-S3).
For RedShift data , bzip2 compression algorithm has the highest compression ratio. ** **
To manually migrate an Amazon Redshift cluster to another AWS account , follow these steps:
Create a manual snapshot of the cluster you want to migrate.
Share the cluster snapshot with another AWS account to view and restore the snapshot.
Before you copy a snapshot to another region, first enable cross-region snapshots.
In the destination AWS account, restore the shared cluster snapshot.

Kinesis

The unit of data stored by Kinesis Data Streams is a data record. A stream represents a group of data records. The data records in a stream are distributed into shards.
A shard has a sequence of data records in a stream. When you create a stream, you specify the number of shards for the stream. Each shard can support up to 5 transactions per second for reads, up to a maximum total data read rate of 2 MB per second. Shards also support up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second (including partition keys). The total capacity of a stream is the sum of the capacities of its shards. You can increase or decrease the number of shards in a stream as needed. However, you are charged on a per-shard basis.
If you have sensitive data, you can enable server-side data encryption when you use Amazon Kinesis Data Firehose. However, this is only possible if you use a Kinesis stream as your data source. When you configure a Kinesis stream as the data source of a Kinesis Data Firehose delivery stream, Kinesis Data Firehose no longer stores the data at rest. Instead, the data is stored in the Kinesis stream.
When you send data from your data producers to your Kinesis stream, the Kinesis Data Streams service encrypts your data using an AWS KMS key before storing it at rest. When your Kinesis Data Firehose delivery stream reads the data from your Kinesis stream, the Kinesis Data Streams service first decrypts the data and then sends it to Kinesis Data Firehose. Kinesis Data Firehose buffers the data in memory based on the buffering hints that you specify and then delivers it to your destinations without storing the unencrypted data at rest.
In Kinesis , To prevent skipped records, handle all exceptions within processRecords appropriately.
For each Amazon Kinesis Data Streams application, the KCL uses a unique Amazon DynamoDB table to keep track of the application's state. Because the KCL uses the name of the Amazon Kinesis Data Streams application to create the name of the table, each application name must be unique.
If your Amazon Kinesis Data Streams application receives provisioned-throughput exceptions, you should increase the provisioned throughput for the DynamoDB table. The KCL creates the table with a provisioned throughput of 10 reads per second and 10 writes per second, but this might not be sufficient for your application. For example, if your Amazon Kinesis Data Streams application does frequent checkpointing or operates on a stream that is composed of many shards, you might need more throughput.
** ** PutRecord returns the shard ID of where the data record was placed and the sequence number that was assigned to the data record. Sequence numbers increase over time and are specific to a shard within a stream, not across all shards within a stream. To guarantee strictly increasing ordering, write serially to a shard and use the SequenceNumberForOrdering parameter.
Details regarding Kinesis records :
- https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-concepts.html
- https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-scaling.html
For live streaming Kinesis gets ruled out if record size greater than 1 MB , in that case Kafka can support bigger records.
You can trigger One lambda per shard. If you want to use Lambda with Kinesis Streams, you need to create Lambda functions to automatically read batches of records off your Amazon Kinesis stream and process them if records are detected on the stream. AWS Lambda then polls the stream periodically (once per second) for new records.
In Kinesis stream ,the PutRecordBatch() operation can take up to 500 records per call or 4 MB per call, whichever is smaller. Buffer size ranges from 1 MB to 128 MB.
In circumstances where data delivery to the destination is falling behind data ingestion into the delivery stream, Amazon Kinesis Firehose raises the buffer size automatically to catch up and make sure that all data is delivered to the destination.
If data delivery to Redshift fail from Kinesis Firehose , Amazon Kinesis Firehose retries data delivery every 5 minutes for up to a maximum period of 60 minutes. After 60 minutes, Amazon Kinesis Firehose skips the current batch of S3 objects that are ready for COPY and moves on to the next batch. The information about the skipped objects is delivered to your S3 bucket as a manifest file in the errors folder, which you can use for manual backfill. For information about how to COPY data manually with manifest files, see Using a Manifest to Specify Data Files.
If data delivery to your Amazon S3 bucket fails , Amazon Kinesis Firehose retries to deliver data every 5 seconds for up to a maximum period of 24 hours. If the issue continues beyond the 24-hour maximum retention period, it discards the data.
Aggregation refers to the storage of multiple records in a Streams record. Aggregation allows customers to increase the number of records sent per API call, which effectively increases producer throughput._ Aggregation Storing multiple records within a single Kinesis Data Streams record while Collection_ using the API operation PutRecords to send multiple Kinesis Data Streams records to one or more shards in your Kinesis data stream.You can first aggregate stream record and then send them to stream using collection putrecords() in multiple shard.
** ** Spark Streaming uses the Kinesis Client Library (KCL) to consume data from a Kinesis stream. KCL handles complex tasks like load balancing, failure recovery, and check-pointing
The Amazon Kinesis Connector Library helps Java developers integrate Amazon Kinesis with other AWS and non-AWS services. The current version of the library provides connectors for Amazon DynamoDB, Amazon Redshift, Amazon S3, Elasticsearch.

Amazon ML

Amazon ML Key component:

o Datasources contain metadata associated with data inputs to Amazon ML

o ML Models generate predictions using the patterns extracted from the input data

o Evaluations measure the quality of ML models

o Batch Predictions asynchronously generate predictions for multiple input data observations

o Real-time Predictions synchronously generate predictions for individual data observations

Amazon ML supports three types of ML models: binary classification, multiclass (Categorial) classification, and regression. - https://docs.aws.amazon.com/machine-learning/latest/dg/types-of-ml-models.html
Details on Amazon ML - https://docs.aws.amazon.com/machine-learning/latest/dg/amazon-machine-learning-key-concepts.html

AUC	Area Under the Curve (AUC) measures the ability of a binary ML model to predict a higher score for positive examples as compared to negative examples.
Macro-averaged F1-score	The macro-averaged F1-score is used to evaluate the predictive performance of multiclass ML models.
RMSE	The Root Mean Square Error (RMSE) is a metric used to evaluate the predictive performance of regression ML models.

A lower Area Under Curve (AUC) reduces accuracy of the prediction; AUC values well below 0.5 may indicate a problem with the data. The F1 score's range is 0 to 1. A larger value indicates better predictive accuracy.
Amazon ML is limited to 100 'categorical' recommendations . Amazon ML can support up to 100 GB of data.
You must upload your input data to S3 because Amazon ML reads data from Amazon S3 locations. You can upload your data directly to Amazon S3, or Amazon ML can copy data that you've stored in Redshift or RDS into a .csv file and upload it to S3.

ElasticSearch

ElasticSearch is suitable to analyze large set of streaming data from Kinesis.
Elastic Search Snapshots are backups of a cluster's data and state. They provide a convenient way to migrate data across Amazon Elasticsearch Service domains and recover from failure. The service supports restoring from snapshots taken on both Amazon ES domains and self-managed Elasticsearch clusters.
Amazon ES takes daily automated snapshots of the primary index shards in a domain, as described in Configuring Automatic Snapshots. The service stores up to 14 of these snapshots for no more than 30 days in a preconfigured Amazon S3 bucket at no additional charge to you. You can use these snapshots to restore the domain.

IoT and Others

For AWS IoT - Devices connect using your choice of identity (X.509 certificates, IAM users and groups, Amazon Cognito identities, or custom authentication tokens) over a secure connection according to the AWS IoT connection model. Note that KMS is not in list.
AWS IoT rule actions specify what to do when a rule is triggered. You can create actions for the following services: CloudWatch, DynamoDB, Elasticsearch, Kinesis Firehose, Kinesis Streams, Kinesis Firehose, Lambda, S3, SNS, and SQS. Not for any relational store like Aurora and Redshift.
The Rules Engine transforms messages using a SQL-based syntax. The Device Gateway (Message Broker) uses topics to route messages from publishing clients to subscribing clients. Then a message is published to the topic, the SQL statement is evaluated and rule action is triggered, sending the message to another AWS service.
In AWS Data Pipeline, a precondition is a pipeline component containing conditional statements that must be true before an activity can run. For example, a precondition can check whether source data is present before a pipeline activity attempts to copy it. AWS Data Pipeline provides several pre-packaged preconditions that accommodate common scenarios, such as whether a database table exists, whether an Amazon S3 key is present, and so on. However, preconditions are extensible and allow you to run your own custom scripts to support endless combinations.
AWS Data Pipeline supports A JDBC database , A RDS Database and Redshift . computational resource that perform the work on EC2 and EMR cluster.
Data Pipeline integrate with on-premise servers. AWS provides you with a Task Runner package that you install on your on-premise hosts. Once installed, the package polls Data Pipeline for work to perform. If it detects that an activity needs to run on your on-premise host (based on the schedule in Data Pipeline), the Task Runner will issue the appropriate command to run the activity, which can be running a stored procedure or a database dump or another database activity.
A QuickSight Dashboard is a read-only snapshot of an analysis. You can share it with other Quicksight users for reporting. The data in the dashboard reflects the data set that is used by the analysis. If you share a dashboard with users, they can then view and filter the dashboard data, but they cannot save any filters applied to the dashboard.

DynamoDB

Number of Dynamo DB partition = readCapacityUnit/3000 + WriteCapacityUnit/1000. 10 GB data per partition after that it create new partition.
Details on GSI Vs LSI in Dynamo DB - https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SecondaryIndexes.html
COPY command used for Loading Data from DynamoDB Into Amazon Redshift.
DAX (DynamoDB Accelerator)is a DynamoDB-compatible caching service that enables you to benefit from fast in-memory performance for demanding application.
A VPC endpoint for DynamoDB and S3 enables Amazon EC2 instances in your VPC to use their private IP addresses to access DynamoDB/S3 with no exposure to the public Internet.
DynamoDB Streams supports the following stream record views:
- KEYS_ONLY—Only the key attributes of the modified item
- NEW_IMAGE—The entire item, as it appears after it was modified
- OLD_IMAGE—The entire item, as it appears before it was modified
- NEW_AND_OLD_IMAGES—Both the new and the old images of the item
DynamoDB supports many different data types for attributes within a table. They can be categorized as follows:

o Scalar Types – A scalar type can represent exactly one value. The scalar types are number, string, binary, Boolean, and null.

o Document Types – A document type can represent a complex structure with nested attributes—such as you would find in a JSON document. The document types are list and map.

o Set Types – A set type can represent multiple scalar values. The set types are string set, number set, and binary set.

AWS Certified Solutions Architect: Professional study guide

Troy — Mon, 07 Oct 2019 22:25:42 +0000

As a follow up to my previous post regarding AWS Certified Solutions Architect: Associate study guide, I figured I'd provide an update.

I passed the Certified Solutions Architect: Professional! Here are my key-takeaways for those who want a TL;DR:

It was long, like very long. It's definitely an exercise in quick comprehension and elimination.
It was difficult, easily one of the hardest exams I've taken to date.
Most of the exams posted online only replicate about 20% of what will actually be on the exam.
Hands-on experience at this level is a MUST. Unlike the Architect: Associate, you must live & breath Amazon Web Services.

As a reference, here is the certification learning path for the AWS Certified Solutions Architect:

Here are the resources I used to study:

Jon Bonso's Solutions Architect Professional practice exams on uDemy
All of my material from the previous post (see above for link)
The great folks over at ExitCertified for a full day boot camp (virtual). This covered exam tactics, question dissemination and was a great live classroom setting.
re:Invent playlist videos regarding the topics on the exam.

Also, there are things that I subscribe to on a daily basis that you don't really think about. This helps me keep fresh on service updates, releases, etc:

Follow the Amazon Web Services Facebook & LinkedIn pages
Follow the AWS Twitter handle along with finding those who work for AWS and tweet daily about their service.

Otherwise that sums it up, thanks for reading!

HashiCorp Vault+AWS Secrets+Rotation tool?

Troy — Thu, 05 Sep 2019 20:06:54 +0000

In the process of implementing HashiCorp Vault and utilizing the AWS secrets engine. We have LDAP set up as an auth method and users are able to curl the vault endpoint to pull credentials:

curl --request POST --data @payload.json https://vault-api.test.com/v1/auth/ldap/login/me | jq '.'

curl --header "X-Vault-Token: mytoken" --request GET https://vault-api.test.com/v1/aws/creds/grp-aws-r-usersrole | jq

Access keys are granted - does anyone know of a script/tool that's already in place that:

Securely creates the payload.json with your LDAP password
CURL's the Vault endpoint with the payload
Deletes the payload file
Retrieves the token
CURL's the Vault endpoint with the token to retrieve the ephemeral access keys
Loads the access keys in the .aws/credentials store

I've used https://github.com/Fullscreen/aws-rotate-key before we moved to Vault but it's no longer applicable.

Should I write something and share?

AWS Certified Solutions Architect: Associate study guide

Troy — Thu, 15 Aug 2019 15:06:37 +0000

With scheduling my AWS Certified Solutions Architect: Professional for late September 2019, I figured i'd finally compile all of the notes and gathered content for the AWS Certified Solutions Architect: Associate.

As a reference, here is the certification learning path for the AWS Certified Solutions Architect:

General preliminary reminders:

Ensure you have hands-on experience with AWS prior to the exam
The content below will likely change as AWS releases new services. This was up-to-date when I took the exam in December 2018.
Some of the content is mixed up, still working to get it all properly organized.
Portions of this has been piece-mealed from various sources. I bring it here to you, to help! :)

The Basics (101)

AWS Global Infrastructure
You will never be tested on numbers (e.g. number of regions/availability zones)

Region
-> Geographical (Brazil, Europe, Asia, etc)). Each region consists of two or more availability zones (AZ). It's not in the same phisical space (if a flooding occurs, other data center can still answer)
Availability zone -> Data center

North America Regions:

US East (Northern Virginia)
US East (Ohio) Region
US West (Oregon) Region
US West (Northern California) Region
AWS GovCloud (US-West Region)
Canada (Central)

Edge Locations -> CDN (content delivery network). They add new ones all the times, over 100 so far. There are many more Edge Locations than Regions.
Main services provided by Amazon. Based on late 2017/early 2018.

Route53: DNS service

EC2: Virtual machines and/or compute

ECS: Virtual machines + docker

Elastic Beanstalk: Deploy apps and don't worry about infrastructure. Good for starting users.

Lambda: Serverless. Upload code, no need to configure any server/virtual machines. Used by Amazon Echo

Lightsail: out of the box cloud. Virtual servers with fixed configs

STORAGE

S3: Virtual disk in the cloud. Object based storage

Glacier: Archive - low cost, but access is not inmediate

EFS: Elastic file service, mount disks without a specific size, automatically (elastically) grows

Storage Gateway: VM in premise with S3 support.

Cache Gateway: cache information from s3 in premise

DATABASES

RDS: Mysql, MariaDB, Microsoft SQL, Oracle, Aurora, etc.

DynamoDB: No relational database (NoSQL database)

Red Shift: Data warehouse. Copy your own data to create reports

Elasticache: Cache (can use two technologies: redis or memcached)

MIGRATION

Snowball: Portable disk that Amazon sends you, you fill it and send it back, and they import it. Several flavors (depends on how much data you need to move)

DMS: Database Migration Service: migrate database to AWS

SMS: Server migration Service: migrate Virtual Machines to AWS

ANALYTICS

Athena: Run SQL queries on S3. CVS or XML. Turn flat files into a database

Elastik Map Reduce (EMR): Big data processing. Large amount of data. Uses HADOP/ Apache Spark in the back

SEARCH

ElasticSearch: Uses Lucene. Open source

Cloud Search: Fully Managed by AWS. Same tecnology as ES

Kinesis: Stream and analize stream data. Big data

Sentient Analisis: Social media streams

Data Pipeline: move data from place to place. Move S3 to DynamoDB, or viceversa

Quicksight: Analyze data, create dashboards, etc.

SECURITY AND IDENTITY

IAM: Authenticate, permissions, etc.

Inspector: Inspects virtual machines (status, etc)

Certificate Manager: Manage SSL Certificates

Directory Service: Active Directory Management

WAF (Web Application Firewall): App net protection against DDOS, hacking. On top of the Network firewall

Artifacts: Compliance documents

MANAGEMENT TOOLS

Cloudwatch: Get information on VM, CPU, memory, etc. Stores different kind of logs

Cloud Formation: Turn servers into code. Templates to build entire networks/servers

Cloud Trail: Audit AWS Resources

OpWorks: configuration management service, provides instances of Chef and Puppet. Automated deploys

Service Catalog: Manage images (vm) and authorized servers in the org

Trusted Advisor: Automated tips and performance optimizations. Automated scan enviroment for problems/security issues

APPLICATION SERVICES

Step functions: visualize steps inside app/microservices. Serverless orchestration

SWF: simple workflow service. Facilitate task both automated (jobs) and human (IE, pick something from a storage)

API Gateway: Door to AWS services, backend services or your own code, in AWS. Can call Lambda functions, for example

AppStream: Stream desktop apps to users

Elastic Transcoder: video tools. Change format, resize, etc.

DEVELOPER TOOLS

Code commit: Store code in the cloud. GIT

Code build: Pay by minute, compile code in different environments

Code deploy: Deploy code to EC2. Automatic building, etc.

Code Pipeline: Keep track of code differences between environments, build pipeline (IE, trigger compile when committing code, run unit test, etc)

ARTIFICIAL INTELLIGENCE

Polly: Service that transform text into Mp3

Machine Learning: dataset, analize data, predict

Rekognition: Image recognition and processing

MESSAGING

SNS: Simple notification service - alert system, via SMS, HTTP endpoint, email, etc.

SQS: Simple Queue Service: message queue

SES: Simple Email Service: SMPT

API Gateway

caches responses from endpoint for set period of time (TTL)
cache can be encrypted, it can be flushed and you can define size
you can't regenerate the cache
low cost, efficient
scales effortlessly
throttle requests to prevent attacks
connect to CloudWatch to log requests and troubleshoot
contains default DDOS protection

Cross-Origin Resource Sharing (CORS)

server on other end can relax same-origin policy
mechanism that allows restricted resources on web page to be from another domain

AWS CLI

Command line to configure AWS

You don't use your console user and password, you use the security and access key that was provided when you added the IAM User
If you lost this values, you have to re-generate them, it's only show after the user is added. AWS Config

Detailed view of configuration resources

Evaluate config with desired settings
Get snapshots of current config in AWS account
Retrieve historical configurations
Receive a notification when resources are created, modified or deleted
View relationships between resources Compute EC2 - Elastic Compute Cloud, virtual/dedicate machines in AWS Lightsail - Virtual Private Server, dumb server with fixed-IP with SSH/RDP access not fully utilizing AWS services Elastic Container Service - Run and manage docker containers at scale Lambda - Code uploaded to cloud, you control when it executes (no worrying about machines underneath) Batch - Used for batch computing in the cloud Elastic Beanstalk - Developers can upload apps and AWS auto-configures

Storage

S3 - Simple Storage Service, object-based storage. Files are uploaded into AWS Buckets
EFS - Elastic File System, network-attached storage (NFS)
Glacier - Data archival, used for data that isn't accessed very often and is cheap
Snowball - Way to bring in large amounts of data to AWS, physical snowball is sent to data center and is sent to AWS where they import for you
Storage Gateway - Virtual Appliances, virtual machines you install in your office and data is replicated back to Amazon

Databases

RDS - Relational Database Service (MySQL, MSSQL, PostgreSQL, Aurora, Oracle, etc.)
DynamoDB - Non-relational Databases (Redis, etc.)
Elasticache - Caching commonly-queried things from database server (top 10 products, etc.)
Red Shift - Data warehousing/business intelligence, complex queries (doing P&L analysis, time-intensive queries, etc.)

Migration

AWS Migration Hub - Tracking service to track application as they are migrated to AWS Application Discovery Service (ADS)- Automated tools that detects application type and related dependencies Database
- Migration Service (DMS) - Easy way to migrate database from on-premise to AWS Server Migration Service - Easy way to migrate virtual/physical server into AWS Cloud Snowball - Similar to Storage, this helps to migrate large amounts of data into the cloud

Networking & Content Delivery

VPC (Virtual Private Cloud) - Basically a virtual data center (configure firewalls, AZ's, address ranges, network ACL's, root tables, etc.). NEED TO UNDERSTAND THIS INSIDE AND OUT TO PASS!
- CloudFront - Content Delivery Network by Amazon, delivers media assets to users (video files, audio files, etc) by storing close to users
Route53 - Amazon's DNS Service
- API Gateway - Amazon's way to create API's for services to talk to
- Direct Connect - Amazon's way of running dedicated line from your business into AWS

Developer Tools

CodeStar - Project managing of code for developers
CodeCommit - Place to store code (source control), private git repository
CodeBuild - Compiles, tests code and build packages ready for deployment
CodeDeploy - Deployment services that will deploy applications to EC2, Lambda, on-premise
CodePipeline - Continuous Delivery to Model/Visualize/Automate steps for software release
X-Ray - Used to debug/analyze serverless applications by showing traces
Cloud9 - IDE Environment to develop code inside AWS console

Management Tools

CloudWatch - Monitoring service, need to know for SysOps Admin exam
CloudFormation - Automated way to deploy servers/services in AWS, agnosticizing aspects to make deployments faster everywhere
CloudTrail - Everything that happens in AWS is recorded and logged in CloudTrail and makes for easy tracking of things happening in your environment
Config - Monitors configuration of entire AWS environment and keeps snapshots of entire AWS environment (visualize AWS environment)
OpsWorks - Similar to elastic beanstalk, used to automate configuration of environments (convered in Sysops Admin test)
Service Catalog - Way of managing a catalog of IT-approved services in AWS. Typically used for governance/compliance in big organizations
Systems Manager - Managing AWS resources (EC2 patch maintenance, for example). Resources can be grouped by department or application
Trusted Advisor - Will give advice across many different disciplines. Make sure to know the difference between - Trusted Advisor and Inspector Managed Services - Amazon will take care of EC2, auto-scaling

Media Services

Elastic Transcoder

MediaConvert - File-based video transcoding with broadcast-grade features
MediaLive - Broadcast-grade live video processing services (video streams)
MediaPackage - Prepares and protects videos for delivery over the internet
MediaStore - Place to store media (storage optimized for media)
MediaTailor - Allows to do targeted-advertising into video streams without sacrificing broadcasting quality

Machine Learning

SageMaker - Makes it easy for developers to use deep learning when coding for their environments Comprehend - Sentiment analysis around data
DeepLens - Artificially-aware camera (can understand what it's looking at--localized detection, not running in cloud--physical hardware)
Lex - Powers Amazon Alexa service, artifical intelligence Machine Learning - Different to deep learning, entry-level. AWS will analyze data sets and predict outcomes
Polly - Takes text and turns into speach
Rekognition - Upload a file and it will do file analysis (upload pic of dog on beach, it will tell you "dog", "beach", etc.)
Amazon Translate - Machine translation service, like google translate but from Amazon
Amazon Transcribe - Used for those that are hard of hearing--takes audio and creates text

Analytics

Athena - Run SQL queries against items in S3 buckets (serverless)
EMR - Elastic Map Reduce, used for processing large amounts of data CloudSearch - Search service for AWS ElasticSearch Service - Elastic Search service for AWS
Kinesis - Way of ingesting large amounts of data into AWS (i.e. social media feeds for specific hashtag) Kinesis Video Streams - Allows you to ingest large amounts of data on people streaming your media QuickSight - BI tool, significantly cheaper than competitors
Data Pipeline - Way of moving data between different AWS services
Glue - Used for ETL (extract, transform, load), glue is optimized to achieve this

Security & Identity & Compliance

IAM - Identity Access Management Cognito - Way of doing device authentication (2-factor auth) GuardDuty - Monitors for malicious activity on AWS account
Inspector - Agent installed on EC2 instances, run tests against it to check for vulnerabilities--these can be scheduled Macie - Scan S3 buckets for any personally identifiable information (PII) and alert you
Certificate Manager - SSL certificates for free if registered through Route53
CloudHSM - Cloud Hardware Security Module, Dedicated hardware used to store your private/public keys or other keys, can also use keys to encrypt objects on AWS
Directory Service - Incorporate Microsoft Active Directory with AWS
WAF - Web Application Firewall (7-layer firewall), monitoring application layer
Shield - DDoS Mitigation Artifact - Used for audit and compliance, download audit and compliance reports

Mobile Services

Mobile Hub - Management console for mobile services
Pinpoint - Targeted push notifications for increased mobile engagement
AWS AppSync - Automatically updates data in web/mobile applications and will update offline users when they reconnect
Device Farm - Testing apps on real, live devices
Mobile Analytics - Analytics service for mobile devices

AR/VR

Sumerian - Used for AR/VR 3D app design

Application Integrations

Step Functions - Way to manage lambda functions and steps to go through it
Amazon MQ - Message queues (like RabbitMQ)
SNS - Notification service when triggers are hit
SQS - Way to decouple intrastructure, take messages in, and allow EC2 instances to poll data
SWF - Create a workflow to be modeled after a process that you have

Customer Engagement

Amazon Connect - Contact Center as a Service (CCaaS)
***Simple Email Service - Easy way to send large amounts of emails

Business Productivity

Alexa For Business - use to dial into rooms, inform IT of problem--Alexa in the workplace
Amazon Chime - Video conferencing by Amazon
Work Docs - Like Dropbox for AWS
WorkMail - Like O365/Gmail for Amazon

Desktop & App Streaming

Workspaces - VDI (Virdual Desktop) that can be accessed in the cloud
AppStream 2.0 - Stream application live to device (like Citrix)

Internet of Things

iOT - Can have devices sending back information
iOT Device Management - Used to manage AWS iOT devices
Amazon FreeRTOS - Realtime OS by Amazon
Greengrass - Software to run local compute services in a secure way

Game Development

GameLift - Service to help develop game services in AWSShared Responsability:

AWS is responsible for:

Base hypervision
Zones
Network
Region
Operative system and patches in RDS databases
etc

YOU are responsible for:

Encryption
Operative system in your EC2 instances
Firewalls
Customer data
etc.

AWS Trusted Advisor

Application that learns from existing AWS customers
Inspects AWS environment and makes recomendations for

Saving money
Improve performance
Close security gapsBeanstalk

Deploy, monitor and scale app quickly
Highly abstract focus towards infrastructure
Simplify infra management, uses GUI to configure things. Good for people with few AWS experience
Uses CloudFormation in the background.
Can be used for Workers/Jobs

Pre-configured Instance support:

NodeJS, Python, PHP, Ruby, Tomcat, .NET (Win IIS), JAva, Go, Packet
Docker images
Generic docker

Can have multiple versions of your app
Can be split into tiers (Web / App / DB tier / Front end / Backend, etc)
You can update the configs after created

Updates can be

1 instance at a time
% of instances
Immutable (launches all apps from 0 again)

If Beanstalk creates your RDS, will be deleted if/when you delete the EBS instance

Business Benefits of Cloud

Almost zero upfront infrastructure investment
Just-in-time infrastructure
More efficient resource utilization
Usage-based consting
Reduced time to market

Technical Benefits of Cloud

Automation (Scriptable Infrastructure)
Auto-scaling
Proactive Scaling
More efficient development lifecycle
Improved testability
Disaster recovery and Business Continuity
"Overflow" traffic to the CloudCloudformation

Allows to transform hardware into code
Easy way to create/manage AWS resources
Can apply versioning to AWS infraestructure (like code)

Template --> Diagram

Stack --> Result of the diagram

Format: JSON or YAML

Template Elements:

Required: list of AWS resources
Optional:
- Version, file format
- Template parameters (up to 60)
Output
- Public IP, ELB addresses (up to 60)

Naming

You can assign local names, and they are used partially when creating resources.
Names are not fixed/enforced to avoid conflicts. Some exceptions exists (IE bucket names).

You can install software with a set of bootstrapping scripts.

Includes integrations with Chef and Puppet

Supports tagging , EBS volumes are automatically tagged

Once provisioned, you have control of the resources

Automatic rollback if error is ON by default (everything is deleted if an error occurs). Keep in mind you are charged for errors, but usage of
CloudFormation is free
Stacks can wait for app to be provisioned using WaitCondition Route53 is supported
IAM Role creation is also supported
Can define deletion policies for resources, when you delete the stack, resources are not deleted
200 stacks max, can request more

If you want to hide something from Cloudtrail/Cloudwatch, mark the parameter with NOECHO

Difference with Elastic Beanstalk?

CloudFormation and Elasticbeanstalk compliment eachother Beanstalk deploys and runs app in the cloud, integrated with dev tools, manage life cycle of apps CloudFormation is a mechanism to provision AWS resources, template to build the entire infrastructure, including Beanstalk apps

Content Delivery Network (CDN)

Edge Location - Location where content will be cached; separate to an AWS Region/AZ

Origin - Origin of all the files that the CDN will distribute. This can be an S3 Bucket, EC2 Instance, Elastic Load Balancer, or Route53, or not with AWS
Web Distribution - Typically used for websites
RTMP - Used for media streaming (adobe flash)
Edge Locations are not just read only, you can write to them, too
Objectes are cached for life of TTL -- Default: 24 hours -- Max: 365 days
You can clear objects from the Cloudfront, but you will be charged
Restrict Viewer access -- Signed URLs -- Signed Cookies
Geo restriction

Cloudtrail is used to log all the API calls made internally on AWS, mostly for audit

Since all the settings you change via the console are actually API calls made to the internal AWS API, if you enable Cloudtrail you can get all the information about everything that was done via the console or via specific API calls.

You can turn on a trail across all regions for your AWS account. Cloudtrail will deliver log files from all regions to a S3 bucket and an optional Cloudwatch log group you specify.

Standard Monitoring = 5 minutes
Detailed Monitoring = 1 Minute

You have to pay if you want Detailed Monitoring

In Cloudwatch you can

Create dashboards
Create alarms
Create events (state changes for AWS resources for example) Logs (agregate, monitor and store logs) Aurora
MySQL-compatible
combines speed and availability of high-end commercial databases
has simplicity an dcost-effectiveness of open source databases
five times better performance than MySQL at 1/10th price of commercial databases
storage starts with 10GB, scales in 10GB increments up to 64TB
compute scales up to 32 vCPU's and 244GB memory
2 copies of data in each AZ, minimum of 3 AZ's
designed to transparently handle loss of 2 copies without affecting DB write availabilty
designed to transparently handle loss of 3 copies without affecting read availability
self-healing; data blocks/disks are continuously scanned for errors and repaired automatically

Aurora Replica Features

Aurora Replicas
- 15 MySQL Read Replicas
- 5DynamoDB
NoSQL database for consistent, single-digit milisecond latency at any scale
Stored on SSD storage
Spread across 3 geographically distinct data centers
Eventually Consistent Reads
- Consistency across all copies of data is uaully reached within a second. Repeating read after short time will return updated data (best read performance)
Strongly Consistent Reads
- Returns a result that reflects all writes that received a successful response prior to the read
Autoscalling supported (% target utilization, min/max)

DynamoDB Pricing

Provisioned Throughput Capacity
- Write Throughput $0.0065 per hour for every 10 units
- Read Throughput $0.0065 per hour for every 50 units
Storage Costs
- First 25 GB --> Free
- $0.25GB/month
Free tier: 25 units read / 25 units write

DynamoDB Streams

Capture changes to DynamoDB for 24 hours. Audit trail like (Add, change (before and after), delete). Use LAMBDA if you want to store the data more than 24 hrs

Max size of each item with attributes: 400KB
BatchWriteItem: 25 items, 16MB
BatchGetItem: 100 items, 16MB

Scan:
Eventual or consistency, add parameter ConsistentRead
Iterator: returns 1MB and LastEvaluatedKey (to paginate)

Data types:
Number, string, binary, boolean, NULL
JSON: stored as document, can create keys and filter by attribute, can update a sub-element, can use document SDK as wrapper (JS)

Indexes:
Global Secondary Index:
Can add up to 5 per table
Local Secondary Index
Can add up to 5 per table, AT CREATION (can't add them later)
10GB PER PARTITION

Security

Fine granular access control allows users in IAM to access/deny information (table, items or even attributes)

Reserved capacity can be bought at discounted price. Limited to a single region.

Triggers are supported (uses DynamoDB w/Lambda)

Can specify TTL on tables. Needs to have a timestamp

DAX: In memory cache (in SDK Node.js & Java)ElastiCache

Memcached
- No Multi-AZ support
Redis
- Multi-AZ support

When asked which service to use to alleviate stress/load on database:

Elasticache is good choice if database is read heavy and not prone to frequent changing
Redshift is good if reason database is stressed is because management keeps running OLAP transactions on it Automated Backups
Allow you to recover database to any point in time within retention period (1-35 days)
Take full, daily snapshot
Store transaction logs
Enabled by default
Stored in S3, free equal to size of database
Deleted when RDS instance is deleted

Snapshots

Database snapshots are done manually (stored even after RDS instance is deleted)

Restoring Backups

When using either restore option, restored version of database will be a new RDS instance with new DNS endpoint

Encryption

Done using AWS Key Management Service (KMS)
Encrypting existing RDS is not currently supported

Multi-AZ RDS

Used for Disaster Recover (DR) only
Availability
- SQL Server
- Oracle
- MySQL Server
- PostgreSQL
- MariaDB

Read Replicas

Used for scaling, not DR
Must have automatic backups turned on to deploy a Read Replicas
You hcan have up to 5 Read Replicas of any database
Allow you to have a read-only copy of your database
Achieved using asynchronous replication
Used for performance improvements, read-heavy database workloads
Each read replica will have its own DNS end point
You can have read replicas that have Multi-AZ
You can create read replicas of Multi-AZ source databases
Read Replicas can be promoted to be their own databases (breaks replication)
You can have a read replica in another Region Redshift
Datawarehousing
Column Data. Agregation
Single Node (160GB)
Multi-Node
- Leader Node, manages client connections and receives queries
- Compute Node, store data and perform queries and computations
- Up to 128 Compute Nodes
Columnar Data Storage
Massively Parallel Processing (MPP) - Automatic distribution of data and query loads across all Nodes
Currently only available in one AZ
Can restore snapshots to new AZ in event of outage

Costs

Computer Node Hours
Backup
Data Transfer (within a VPC, not outside of)

Encryption

Encrypted in transit using SSL
Encrypted at rest using AES-256
Redshift takes care of key management

Availability:

SINGLE AZ
Can restore snapshots to other AZ
Enable Cross-Region snapshot for recovery

VPC:
Turn on Enhaced VPC routing for VPC endpoints (So data doesn't leave your own VPC)AWS Database Types
Maximun size: 16 TB. If larger, consider Redshift

RDS - OTLP (Online Transaction Processing)

SQL Server
Oracle
MySQL
PostgreSQL
Amazon Aurora
MariaDB
- DynamoDB
- RedShift OLAP (Online Analytics Processing, Datawarehousing)
- Elasticache

Non-Relational Database Structure

Database
- Collection (table)
  - Document (row)
  - Key/Value Pairs (fields)

Data Warehousing

Used for Business Intelligence (Cognos, Jaspersoft, etc.)
OLTP Vs. OLAP --- OLTP (Online Transaction Processing) --- --- Order number 2120121 --- --- Pulls up a row of data (name, date, address, status) --- OLAP (Online Analytics Processing, used for Datawarehousing) --- --- Pull in large number of records --- --- Uses different type of architecture for database and infrastructure

Elasticache

Web service that makes it easy to deploy, operate, and scale in-memory cache in the cloud
Types
- Memcached
- Redis

Summary
Database Types

RDS (OLTP)
- SQL
- MySQL
- PostgreSQL
- Oracle
- Aurora
- MariaDB
DynamoDB (NoSQL)
RedShift (OLAP)
Elasticache (in-memory)
- Memcached
- Redis

Multi-AZ

Used for DR
Not used for performance gains

Read Replicas

Used for scaling, performance gains
You can have up to 5 Read Replicas
You can have replicas of replicas (higher latency)
Can be in a different region

Aurora scaling

2 copies of data in each AZ, 3 AZ's minimunm (total of 6 copies)
Designed to handle losses transparently
Self-healing storage

Aurora Replicas

Up to 15 Replicas

MySQL Replicas

Up to 5 Replicas

DynamoDB vs RDS

DynamoDB offers "push button" scaling
RDS requires bigger instance size or to add Read Replica

DynamoDB

stored on SSD storage
spread across 3 geographically distinc data centers
Types
- eventually consistent reads (default)
- strongly consistent reads

Redshift Configuration

Single Node (160GB)
Multi-Node
- Leader Node (manages client connections)
- Compute Node (stores data, performs queries, up to 128 nodes)

Elasticache

Memcached
- Multi-AZ NOT available
Redis
- Multi-AZ available

Two types of backup:

Automated -> retention period: between 1 and 35 days
Database snapshots Stored in S3. Not deleted when the RDS instance is deleted

Encryption at rest: KMS. Can't enable encryption on existing DB. Must perform a copy and enable the encryption on the restored copy.
SOA record stores:

name of server that supplied data for the zone
admin of the zone
current version of the data file
number of seconds a secondary name server should wait before checking for updates
number of seconds a secondary name server should wait before retrying a failed zone transfer
maximum number of seconds a secondary name server can use data before it must refresh or expire
default number of seconds for the time-to-live (TTL) file on resource records

NS Records (Name Server Record)

used by top level domain server to direct traffic to the Content DNS server which contains authoritative DNS records

A Records (Address Record)

used to translate domain name to IP address

TTL Record (Time-To-Live Record)

The Length that a record is cached on either the Resolving SErver or the users local PC

CName Record (Canonical Name Record)

can be used to resolve one domain name to another

Alias Record

works like CName record in that you can map one DNS name to another
CName can't be used for naked domain names, can't have CName for violetfamily.com, it must be either A Record or Alias

EBS Vs Instance Store

All AMI's are categorized as either backed by Amazon EBS or backed by instance Store
EBS Volumes: --- The root device for an instance launched from the AMI is an Amazon EBS volume created from an Amazon EBS snapshot
Instance Store Volumes: --- The root device for an instance launched from the AMI is an instance store volume created from a template stored in Amazon S3 --- Sometimes called Ephemeral Storage --- If host fails, you lose your data
Only EBS backed instances can be stopped
Both instances types can be rebooted
By default, ROOT volumes will be deleted on termination, but with EBS volumes you can tell AWS to keep the root device volumeAutoScalling

Launch configuration on Autoscalling group -> Choose AMI
Can't change the AMI ID, it's chosen on creation

Grace period: time that takes an instance to warm up. Will starts the checks after this period.

You can find load logs related to autoscaling in

Cloudwatch (metrics)
Access logs
Request tracing
Cloud trail logs

How to register a LB group?

Instance Id
IP Address of the instance

3 ways to scale the servers

Manual Scaling
Dynamic scaling -- In Target Tracking Scalling, you select a metric and set a target value, and EC2 Autoscalling sets the Cloudwatch Alarms to trigger the scaling based on the metric that you set (or as close as possible) -- Step scaling allows you to "step up" the number of servers (IE, add 2, add another 2, add another 2, etc), depending on the alarm breach -- Simple scaling increases the current capacity of the group based on a single scaling adjustment. If you can, use step scaling even if you have a single metric.
Scheduled scaling -- You can predict the load changes and how long you need it to run (IE, add 2 more servers between 9am and 12pm from Monday to Friday)

Volumes & Snapshots

Volumes exist on EBS --- Virtual Hard Disk
Snapshots exist on S3
Snapshots are point in time copies of Volumes
Snapshots are incremental - only blocks that have changed since your lat snapshot are saved

Snapshots of Root Device Volumes

Can create AMI's from Volumes and Snapshots
Can change volume sizes on the fly, including size and storage type. If you change a Volume on the fly, you have to wait 6 hours to change it again. Can't change volume type of Magnetic Std HD
Volumes MUST BE in the same AZ as the EC2 instance
If you need to restore/move a EBS Volume to another AZ, you need to create a snapshot of the volume, and create a new volume based on that snapshot, in the other AZ

Encryption

To encrypt root volume, you need to create an AMI image of your boot disk first, OR use a third party software to encrypt

Volumes Vs Snapshots - Security

Snaps of encrypted volumes are encrypted automatically
Volumes restored from encrypted snaps are encrypted automatically
You can share snapshots only if they are unencrypted --- Snaps can be shared with other AWS accounts or made public

Default option is to delete volume when instance is terminated. Can be turned off in EC2 settings

EBS Volumes only scale up; can't shrink in size
Elastic File System (EFS) - file storage service for EC2 instances.

Supports NFSv4
only pay for storage used
can scale up to petabytes
supports thousands of concurrent NFS connections
multiple EC2 can point to the same EFS
data is stored across multiple AZ's within a region.
read after write consistency
can restrict permission to file level or directory level
can't mount an EFS in multiples VPC; only one at a time
uses port 2049 (NFS) -- file system and VPC must be in the same REGION
Two types: -- General purpose: low latency -- Max IO: higher latency, but useful for big data Application Load Balancers
best suited for load balancing http and https traffic
operate at layer 7
application-aware

Network Load Balancers

best suited for load balancing TCP traffic
operate at layer 4
can handle millions of requests per second with low latency

Classic Load Balancers

legacy Elastic Load Balancer
load balance HTTP/https
operates at layer 7
can use strict layer 4 load balancing
if application stops responding, ELB responds with 504
X-Forwarded-For header can pass on users public IP address

Is the load balancer not answering?

Internet facing load balancer is attached to PRIVATE SUBNET (should be in the public one)
Security group ACL does not allow traffic Placement Groups:
Only certain types of instances can be launched in placement group (compute, GPU, Memory, Storage)
AWS recommends homogeneous instances
Can't merge placement Groups
Can't move existing instance into a placement group
Can create AMI from instance and launch that into placement group

Clustered Placement Group:

Grouping of instances within a single AZ. Recommended for applications that need low network latency, high network throughput, or both
Can't spread multiple AZ

Spread Placement Group:

Group of instances that are each placed on distinct underlying hardware. Recommended for applications that have small number of critical instances that should be kept separate from each other
Can spread over multiple AZExam Notes
Security Group updates are applied immediately
Security Groups are stateful (adding inbound rule automatically adds outbound rule)
All inbound traffic is blocked by default
All outbound traffic is allowed
Any number of EC2 instances within a security Group
You can have multiple security groups attached to an instances
Security Groups are STATEFUL and Network Access Control Lists are STATELESS
Cannot block specific IP addresses using Security Groups
You can specify allow rules but not deny rulesEC2 - web service that provides resizable compute capacity in the cloud

On Demand

for users that want low cost and flexibility
applications with short term, spiky, unpredictable workloads
initial testing on EC2

Reserved Instances

apps with steady or predictable usage
applications that require reserved capacity
users can make up-front payments to reduce total cost
Standard Reserved Instance --- up to 75% off on-demand cost
Convertible Reserved Instance --- up to 54% off on-demand cost --- capability to change attributes of Reserved Instance as long as exchange results in creation of Reserved Instances of equal or greater value
Scheduled Reserved Instance --- available to launch within time window you reserved --- allows you to match capacity reservation to a predictable, recurring schedule

Spot Instances

flexible start and end times
only feasible at very low compute prices
users with urgent need for large amounts of additional computing capacity

Dedicated Hosts

useful for regulatory requirements that may not support multi-tenant virtualization
useful for licensing that doesn't support cloud deployments
can be purchased on-demand
can be purchased as reservation for up to 70% off on-demand price

Instance Types
F - FPGA (Field Programmable Gate Array)
I - IOPS
G - Graphics
H - High Disk Throughput
T - cheap general purpose (think T3 micro)
D - Density
R - RAM
M - Main choice for general purpose apps
C - Compute
P - Graphics (think Pics)
X - Extreme Memory

EBS - Elastic Block Storage

Attach block storage to EC2 instances
placed in a specific AZ, automatically replicated to protect you from single-component failure
if windows/linux installed on disk, it's called the "root device volume"
Can't mount 1 EBS volume on multiple EC2 instances. Use EFS instead

EBS Volume Types

General Purpose SSD (GP2) --- General purpose, price/performance balance --- 3 IOPS/GB up to 10,000 IOPS and bursts up to 3,000 IOPS for extended periods for volumes 3334GB+ --- Less than 500 MiB/s
Provisioned IOPS SDD (IO1) --- Designed for IO intensive applications or NoSQL databases --- Used when needing more than 10,000 IOPS --- Can provision up to 20,000 IOPS/Volume --- More than 500 MiB/s
Throughput Optimized HDD (ST1) --- Big Data --- Data warehouses --- Log processing --- Cannot be boot volume
Cold HDD (SC1) --- Lowest cost storage, infrequently accessed --- Typically a file server
Magnetic (Standard) --- Lowest cost per gigabyte of all volume times that are bootable --- start dev here and move up when you're ready

Instance Metadata

http://169.254.169.254/latest/meta-data/

Status Checks:

System Status Checks: underlying layer (TCP, etc, to see if the instances recieves network packages)
Instance Status Checks: software and network

Termination protection is OFF by defaultECS (Elastic Container Service)

ECS: Elastic Container Service
ECR: Elastic Container Registry
Task definition: blueprint
Service: Launches and maintains copies of tasks definitions
Cluster: Where tasks runs. Set of containers running ECS Service
Task: instaces of a task definition

ECS Cluster

Container instance \ - Task Service /- Task
Container instance / - Task

10,000-foot Overview

Know EC2 Pricing Models

On Demand --- Pay by the second or hour
Reserved --- Reserve capacity, contracts are from 12-36 months
Spot --- Set a bid price and if spot price meets your bid it will be provisioned --- Instances terminated when spot price goes out of range --- won't be charged if AWS terminates instance, but you will be charged if you terminate it
Dedicated Hosts --- Used when licensing or multi-tenant is an issue

Know EC2 Instance Types

(FIGHT DR MCPX)

Know EBS

Storage Types --- SSD, General Purpose GP2 (up to 10,000 IOPS, less than 500 MiB/sec) --- SSD, Provisioned IOPS IO1 (MOre than 10,000 IOPS, more than 500 MiB/sec) --- HDD, Throughput Optimized ST1 (frequently accessed workloads) --- HDD, Magnetic Standard (cheap, infrequently accessed storage)
Cannot mount EBS Volume to multiple EC2 instances; use EFS instead
Termination Protection is turned off by default, you must turn it on
On EBS-backed instance, default action is for the root EBS volume to be deleted when the instance is terminated
EBS Root volumes of your DEFAULT AMI's cannot be encrypted (but third party tools can be used to encrypt)
EBS Volumes can also be copied and then encrypted at that time
Additional volumes can be encrypted

Know Volumes Vs Snapshots

Volumes exist on EBS, virtual hard disk in the cloud
Snapshots exist on S3
You can take a snapshot of a volume, the snapshot will be stored on S3
Snapshots are point-in-time copies of volumes
Snapshots are incremental
First snapshot takes a while
Security --- Snapshots of encrypted volumes are encrypted automatically --- Volumes restored from encrypted snapshots are encrypted automatically --- Snapshots can be shared, but only if they are not encrypted
Snapshots of ROOT Device Volumes --- Stop instances before taking snapshot of ROOT volume
EBS Vs. Instance Store (Ephemeral Storage) --- Instance Store volumes cannot be stopped --- If underlying host in Instance Store fails, you lose your data --- EBS can be stopped --- Both can be rebooted, you won't lose your data --- Both will be deleted on termination, but EBS offers option to keep
Snapshotting RAID Array --- Freeze the file system --- Unmount RAID Array --- Shutdown EC2 Instance

Know Amazon Machine Images (AMI)

Regional, but can be copied to other regions

Know CloudWatch (monitoring)

Standard (5minutes)
Detailed (1minute)
CloudWatch is for performance monitoring
Unlike CloudTrail, which is for auditing AWS
Dashboards
Alarms
Events
Logs

Know Roles

More secure than storing access key and secret access keys on instances
Easier to Manage
Can be assigned to EC2 instance AFTER provisioning
Universal to region

Know Instance Meta-data

Used to get information about an instance
curl http://169.254.169.254/latest/meta-data/
curl http://169.254.169.254/latest/user-data/

Know EFS Features

Supports NFSv4.1 protocol
Only pay for storage used
Can scale to petabytes
Can support thousands of concurrent NFS connections
Data stored across multiple AZ's
Read After Write Consistency

Know Lambda

Event-driven compute service
Compute service, run code in response to requests

Know Placement Groups (assume clustered is implied, if not mentioned)

Clustered Placement Groups --- Always in one AZ, used for Big Data (low latency, high throughput)
Spread Placement Groups --- Important EC2 instance on separate hardware

KNOW Elastic Container Service (ECS)

S3 Exam Tips

S3 is object-based
Files can be 0B to 5TB
Unlimited storage
Files are stored in Buckets
Universal namespace, names must be unique
Read after Write consistency for PUTS of new objects --- Immediately able to read object
Eventual Consistency for overwrite PUTS and DELETES (take time to propagate) --- If you update object and then try to read you may get the old object
Writing to S3 returns HTTP200 for successful write
Loading files is faster when multipart upload is enabled

Route53 exam tips

You can only resolve an ELB by going to it's DNS name
ELB never has IPv4 address, only DNS names
Understand difference between Alias Record and CName Record
Given the choice, always choose Alias Record over a CName Record
Understand routing policies and their use cases

VPC exam tips

Think of VPC as logical datacenter in AWS
Consists of IGW's (or virtual private gateways), route tables, NACL's, Subnets, Security Groups
1 subnet = 1 AZ
Security Groups are stateful; NACL's are Stateless
NO TRANSITIVE PEERING
If you need to access resources from another AWS account, you need to perform a VPC peering between both accounts.

Load balancer tips

3 types of Load Balancers --- Application Load Balancers (layer 7) --- Network Load Balancers (layer 4) --- Classic Load Balancers (layer 7 and layer 4)
504 means the gateway has timed out. This means the application not responding within the idle timeout period. --- Troubleshoot. Is it the web server or the database server?
If you need IPv4 address of end user, look for the X-Forwarded-For header
Instances monitored by ELB are reported as InService or OutofService
Healthchecks instance by talking to it
ELB'as have their own DNS name
Read FAQ

Exam Tips

ELB do not have pre-defined IPv4 addresses, must resolve using DNS name
Understand the difference between Alias Record and CName Record
Given the choice, always choose an Alias Record over CNameRoles are not tied to specific region (neither are users) Can apply roles to running instances If you apply a role to an instance, there's no need to configure the Access Keys / Secret keys to get permissions to use AWS Services (Ie, to access an private S3 bucket) --> MORE SECUREIdentity Access Management (IAM) - Allows you to manage users and their level of access to the AWS Console.
Centralized control of AWS account
Shared access to AWS account
Granular permissions
Identify Federation (AD, FB, LinkedIn, etc.)
Multifactor Authentication
Provide temporary access for users/devices/services
Allows you to setup password rotation policy
Integrates with many services
Supports PCI DSS Compliance

Critical Terms
Users - End users
Groups - Collection of users under one set of permissions (Admins, HR, etc.)
Roles - Create roles and assign them to AWS resources (i.e. giving EC2 instance role for writing to EC2)
Policies - Document that defines one or more permissions. Apply policies to users, groups, and roles

IAM does not use region concept.

You can create cross-account roles (ie, you hire a company to perform audit, the user that you provide to the auditor can be cross-account)

Never use your root account for daily use. ALWAYS create new users

Remember: Add user confirmation window (where the security and access key is shown) is only displayed ONCE. If you lose access, you will have to regenerate the keys.
Kinesis

Kinesis Stream
Kinesis Firehose
Kinesis Analytics

Kinesis Streams

data stored for 24 hours by default
data stored in shards
data consumers (ec2 instances) turn shards into data to analyze
5 transactions per second for reads, maximum total rate of 2 MB/second up to 1,000 records for writes

Kinesis Firehose

Automated
no dealing with shards

Kinesis Analytics

Way of analyzing data in Kinesis using SQL-like queriesExam Tips
Lambda scales out (not up) automatically
Lambda functions are independent
Lambda is serverless
Know which AWS services are serverless
Lambda functions can trigger other lambda functions
Lambda is event driven (runs code in response to events)
Architecture can get complicated, AWS X-ray helps to debug
Lambda can do things globally
Know your triggers --- API Gateway, Alex Skills Kit, IoT, S3, DynamoDB, Cloudwatch, Cloudfront, DynamoDB, etc.
Code supported: JS - Java - Python, C#, C++
Pricing: first 1 million hits -> Free. 0.20 USD per million after
Duration: can't run more than 15 mins (was recently raised, it was 5 mins before)
The more memory/duration you need the function running, the higher the costLoad Balancers

Types of Load Balancers:

Application load balancer (http / https level)
Classic Network load balancer (TCP))
Network load balancer (TCP/UDP)

Can be external (accessible via internet) or internal (balancing backend instances behind a subnet, for example)

Performs health checks

Unhealthy threadhold: number of consecutive checks failed
Healthy threadhold: number of consecutive OK to consider healthy

Load balancers only have HOSTNAMES, not IP address. This is because if a AZ goes down, it can move to another without problems
Routing Policies:

Simple -- default routing policy when you create a new record set. Most commonly used when you have a single resource that performs a given function for your domain (e.g. one web server that serves content for violetfamily.com) can point to a ELB that will later balance the load between N servers, but it's still pointing to a single item
Weighted -- allows you to split your traffic based on different weights assigned
Latency -- allows you to route your traffic based on the lowest latency for your end user (region with fastest response time)
Failover --- used when you want to create an active/passive setup (e.g. use primary site in US-EAST-1 and secondary DR Site in US-WEST-1).
Geolocation

Aliases can point to:

ELB
cloudfront
S3 buckets

CNAME: Charged $$$
ALIASES: FreeS3 - Simple Storage Service, provides developers and IT teams with secure, durable, highly-scalable, flat object storage.

Object-based storage: --- Key --- Value --- Version ID --- Metadata --- Subresources: --- --- Access Control Lists --- --- Torrent
Unlimited storage
Files can be 0 Bytes to 5 Terabytes
Files stored in buckets (basically just folders/logical separation)
Bucket names have to be unique globally
Successful upload will receive HTTP200 code
Read after Write consistency for PUTS of new objects
Eventual Consistency for overwrite PUTS and DELETES (can take some time)
Built for 99.99% availability
Amazon guarantees 99.999999999% durability (unlikely to ever lose a file) (11 9s)
Tiered storage available
Lifecycle management
Versioning
Encryption
Secure your data using Access Control Lists and Bucket Policies
Bucket tags are not inherited to files

S3 Storage Tiers

S3 Standard: --- 99.99% available, 99.999999999% durable
S3 IA (Infrequently Accessed): --- Data that is accessed less often, but requires rapid access. Lower fee than S3 but charged retrieval fee.
S3 One Zone IA: --- Lower-cost option for IA but doesn't require multiple AZ resiliance
Glacier: --- Super cheap, used for archival only. Retrieval time takes 3-5 hours

S3 Charges

Storage
Requests
Storage Management Pricing (tags)
Data Transfer Pricing (cross-region replication)
Transfer Accelleration - fast transfers over long distances using CloudFront
Can configure bucket as Request Pays if you use multiple AWS accounts and multiple buckets that transfer info between them

S3 Versioning

Stores all versions of an object (even if you delete an object)
Once enabled, versioning cannot be disabled, only suspended
Integrates with Lifecycle rules
Versioning's MFA Delete capability, which uses MFA, can be used to provide additional layer of security

S3 Cross Region Replication

Versioning must be enabled on both the source and destination buckets
Regions must be unique
Files in an existing bucket are not replicated automatically, all new and updated files will be replicated automatically
You cannot replicate multiple buckets or daisy chain replication
Delete markers are replicated
Deleted individual versions or markers will not be replicated

S3 Lifecycle Management

Can be used with versioning
Can be applied to current and previous versions
Transition to IA after 30 days is possible, if file is larger than 128k
Archive to Glacier after 30 days is possible
Can permanently delete after N days

S3 Security & Encryption

All newly created buckets are private by default
You can setup access control for buckets using bucket policies and ACL
Buckets can be configured to create access logs which log requests made to the bucket.
Methods of Encryption --- In Transit --- --- SSL/TLS --- At Rest --- --- Server Side --- --- --- S3 Manged Keys SSE-S3 --- --- --- AWS Key Management Service, Manged Keys SSE-KMS --- --- --- Server Side Encryption with Customer Provided Keys SSE-C --- --- Client Side Encryption

S3 Transfer Acceleration

Uses CloudFront Edge Network to accelerate yoru uploads to S3

S3 Static Website Hosting

[bucketname].s3-website-[region].amazonaws.com
CORS: you can enable cors on the bucket to allow other sites to get the files from the bucket
If you want to host a static website in S3, just create a bucket name with the URL (IE, if you want to host something.com, create a bucket name with that name) and create an alias to that bucket.

Dualstack: support for IPV4 and IPV6

Storage Tiers
--- S3 Standard
--- --- 99.99 available, 99.999999999 durable, designed to sustain loss of 2 facilities concurrently
--- S3 IA (Infrequently Accessed)
--- --- Accessed less frequently, requires rapid access when needed. Lower fee than S3 but charged for retrieval.
--- S3 One Zone IA
--- --- Want lower-cost for infrequent data but doesn't require multiple AZ resiliency
--- Glacier
--- --- Cheap, used for archival only. 3-5 hour retrieval time

Core Fundamentals of S3:
--- Key
--- Value
--- Version ID
--- Metadata
--- Subresources
---- Access Control Lists
---- Torrent file

Versioning
--- Objected based storage (files only, not OS or db)
--- All version of object are stored, writes and deletes
--- Once enabled, versioning cannot be disabled, only suspended
--- Integrates with Lifecycle rules
--- Versioning's MFA Delete capability can be used to provide additional layer of security
--- Cross Region Replication, requires versioning on source and destination buckets
Lifecycle Management
--- Can be used in conjunction with versioning
--- Can be applied to current/previous versions
--- Actions that can be done:
--- --- Transition to Standard S3 IA after 30 days
--- --- Archive to Glacier after 30 days
--- --- Permanently Delete

CloudFront
--- Edge Location - location where content will be cached
--- Origin - Origin of all files that CDN will distribute
--- Distribution - name given to CDN which consists of collection of Edge Locations
--- --- Web Distribution - typically used for websites
--- --- RTMP Distributions - media streaming/flash files
--- Edge locations are not just read only, you can write to them too
--- Objects are cached for life of TTL (default 24 hours)

Securing Buckets
--- Newly created buckets are private by default
--- You can setup access control using:
--- --- Bucket Policies
--- --- Access Control Lists
--- Buckets can be configured to create access logs

Encryption
--- In Transit
--- --- SSL/TLS
--- At Rest
--- --- Server Side Encryption
--- --- --- S3 Manged Keys SSE-S3
--- --- --- AWS Key Management Service, Manged Keys SSE-KMS
--- --- --- Server Side Encryption with Customer Provided Keys SSE-C

Storage Gateways

File Gateway - flat files, directly on S3
--- Volume Gateway
--- --- Stored Volumes - Entire dataset stored on site, async backed up to S3. Stores data as Amazon EBS snapshots in S3
--- --- Cached Volumes - Entire dataset stored in S3, most recent data stored onsite
--- Gateway Virtual Tape Library (VTL) - Used for backup and uses popular backup applications like NetBackup, Backup Exec, Veeam, etc.
-- Network requirements: Port 443, 80 (activation only) , 3260 (iSCSI targets), UPD53 (dns)

Snowball
--- Import to S3 or Export from S3
--- Snowball
--- --- 80TB, no compute
--- Snowball Edge
--- --- 100TB, has compute
--- Snowmobile
--- --- 100PB, semi-truck, only available in USA

S3 Transfer Acceleration
--- Speed up transfers to S3 using S3 transfer acceleration. Costs extra, great impact for people in distant locations.

S3 Static websites

Serverless
Cheap, scales automatically
Static only, no compute Security Token Service (SKS)

Grants users limited and temporary access to AWS services

Federation (typically Active Directory) -- No need to create IAM accounts. Single sign on the AWS console. Combine users from 1 domain with users from another domain
Federation with mobile apps - FB, AMZ, Google or other OpenID providers
Cross Account Access: users from other AWS accounts

Identity Broker: Service that allows to take identity from A and join it (federate it) to B. Impersonate.

Identity Store: FB, Google, Active Directory, etc, all store the identity.

Identity: the user itself.

Identity broker needs to be programmed.
The temp token returned by IAM Policy is valid between 1 to 36 hours
STS returns 4 values if successful:
1) access key
2) secret access key
3) token
4) duration

Steps:

Develop identity broken to communicate with LDAP & AWS
Identity broker (IB) ALWAYS authenticate with LDAP first, then AWS STS
1. App gets temporary access to AWS Resources

SAML:

Secure Assertive Markup Language

Web Identity Federation with mobile apps: you can auth app using things like FB, you need to code it of course

ARN:

Amazon Resource Name

AssumeRoleWithWebIdentity: You need to call this method after auth with FB. After that you can access the AWS resouces.

Snowball - Petabyte-scale data transport solution that uses secure appliances to transfer large amounts of data into and out of AWS. Addresses challenges with large-scale data transfers including high network costs, long transfer times, and security concerns.

Storage only, up to 80TB Snowball Edge - Snowball but with onboard compute functionality
Storage up to 100TB Snowmobile - Snowball for petabyte/exobyte amounts of data

- Storage up to `100PB`

Exam Notes

Understand what Snowball is
Understand what Import Export is (old name for Snowball)
Snowball can:
--- import to S3
--- export from S3SNS
Notification service. Data type: JSON
Pub/Sub paradigm
PUSH mechanism. Instant
Push notification
Deliver SMS or Email, SQS, or any HTTP endpoint
Message stored reduntly across multiple AZ
Topic: access point
Pay as you go
-- 0.50 per 1 million request
-- 0.75 per 100 sms
-- 0.06 per 100,000 HTTP requests
-- 2 usd per 100,000 emails
Can use different format for different protocols (http/s, email, email json, SQS, lambda)
Each message contains:
-- Name
-- Type
-- Value

Topic -> Subscriber 1 (http)
-> Subscriber 2 (email)
--> Subscriber 3 (SQS)

You can apply a filter policy in a topic subscription (IE, only send critical errors to the managers)

SNS Vs SQS

SNS is push
SQS is poll

SQS

Message queue system. Message retained up to 14 days
256K of text. Billed at 64K chunks
Does NOT guarantee first in, first out. If you need that, use a FIFO SQS queue
SQS pulls information
Supports auto scalling
Visibility timeout window: 12 hours max, default is 30 secs
"At least once" -- each message is delivered at least once but maybe more. Keep that in mind
First million SQS hits: free. 0.50 USD per million, per month, after that
Single request can have from 1 to 10 messages.
Change visibility timeout with the "ChangeMessageVisibility" method
Enable LONG POLLING (20 secs) to wait for a message to become available. Raise an event as soon as a message arrives, good for saving money/requests.
Has SNS integration, SQS subscribes to SNS topic. When SNS msg arrives, distribute msg to suscribed SQS queues. 1 SNS -> N SQS
Storage Gateway - Service that connects an on-premise software appliance with cloud-based storage to provide seamless and secure integration between an organizations IT environment and AWS storage infrastructure.

Types of Storage Gateways

File Gateway (NFS)
Volumes Gateway (iSCSI) --- Stored Volumes --- Cached Volumes
Tape Gateway (VTL)

File Gateway - Files are stored as objects in your S3 buckets, accessed through NFS mount point. Once files are transferred they can be managed as native S3 objects.
Volume gateway - Presents your applications with disk volumes using iSCSI block protocol.
--- Can be asynchronously backed up as point-in-time snapshots of your volumes
--- Stored in cloud as EBS snapshots
--- Stored Volumes
--- --- Store entire copy locally, asynchronouslybackup to AWS. Complete copy of data kept on-site.
--- Cached Volumnes
--- --- S3 is primary data storage and store only most recent copy locally. Don't need large storage arrays locally.
Tape Gateway - Durabled, cost-effective solution to use existing tape-based backup solution, virtual tapes are sent to and stored in S3.

Exam Study Notes
File Gateway - for flat files, stored directly on S3
Volume Gateway:

Stored Volumes - Entire dataset stored onsite and async backed up to S3
Cached Volumes - Entire dataset stored in S3 and most recent data cached onsite Gateway Virtual tape Library (VTL):
Used for backup and uses popular backup applications like NetBackup, Backup Exec, Veeam, etc.SWF (Simple WorkFlow)

Workflow system

Actor:
application to start/initiate workflow
could be website or mobile app, for example

Worker: program/person that interacts with WF
Get task
Process recieved task
Return result

Decider:
control coordination tasks
Ordering, concurrency, scheduling

A task is designated once and never duplicated

Domain:
Container where your WF runs
Isolate set of types, executions, and task lists from others in same account
Format: JSON

A workflow can run for ONE YEAR (measures in seconds)

Difference between SQS and SWF

SWF:
* Task oriented API
* Task runs 1 (never duplicated)
* Keeps track tasks and events
* Human interaction if needed (ie, "Pick item from the storage")

SQS
* Message oriented API
* Message might be duplicated
* Implement manual app trackingSolutions Architect - Associate (Understand VPC's inside and out)

Analytics
Management Tools
Migration
Compute
Desktop & App Streaming
Application Integration
Security & Identify & Compliance
Networking & Content Delivery
Storage
Databases

Security Group

operates at the instance level
supports allow rules only
stateful
evaluate all rules before deciding whether to allow traffic
applies to an instance onlyExam Tips
Cannot enable flow logs for VPC's that are peered with your VPC unless the peer VPC is in your account
you cannot tag a flow log
after flow log is created, you canot change its configuration
not all IP traffic is monitored
traffic from 169.254.169.254 not monitored
DHCP traffic not monitored
traffic to reserved IP address for VPC router is not monitoredExam Tips
NAT is used to provide internet traffic to EC2 instances in private subnets
Bastion is used to securely administer EC2 in private subnets (jump box)NAT gateways
Preferred by the enterprise
Scale automatically up to 10Gbps
No need to patch
Not associated with security groups
Automatically assigned a public ip address
Remember to update route tables and point to NAT Gateways
No need to disable source/destination checks
More secure than a NAT instanceVPC
Logically isloated section of the AWS Cloud where you can launch AWS resources in a virtual network that you define.
Virtual Data Center in the cloud
Amazon provides you a default VPC in every region when you create your account
Can create hardware VPN between corporate datacenter and your VPC, make AWS your extension

What can be done:

Launch instances into subnet of choice
Assign custom IP address range to each subnet
Configure route tables between subnets
Create internet gateway and attach it to our VPC, one per VPC
Better security control over AWS resources
Instance security groups
Subnet Access Control Lists (ACLs)

Route Tables determines if a subnet is REACHEABLE
Network ACL determines if traffic CAN ENTER a subnet

Default VPC Vs. Custom VPC

Default VPC is user friendly, allows you to immediately deply
All subnets in default VPC have a route out to the internet
Each EC2 instance has public and private IP address

VPC Peering

Allows you to connect one VPC with another via a direct network route using private IP Addresses
Instances behave as if they were on the same private network
You can peer VPC's with other AWS accounts as well as with other VPC's in same account
Peering is in star configuration: 1 central VPC peer with 4 others---no transitive peering
Use C5 or M5 instances for VPC peering

Bastion hosts
Bastion hosts are used to security administer EC2 instances via SSH or RDP. Can also be called jump box. More secure than opening all your servers to the world to SSH/RDP

NAT Instances

When creating a NAT instance, disable source/destination check on the instance
NAT instances must be in a public subnet
There must be a route out of the private subnet to the NAT instance for this to work
The amount of traffic that NAT instances can support depends on the instance size
You can create HA using:
- Autoscaling Groups
- Multiple subnets in different AZs
- Script to automate failover
- Behind Security groups
- NAT gateways
Preferred by the enterprise
Scale automatically up to 10Gbps
No need to patch
Not associated with security groups
Automatically assigned a public ip address
Remember to update route tables and point to NAT Gateways
No need to disable source/destination checks
More secure than a NAT instance

Difference between NAT Gateway and Internet Gateway

Both are highly available architectures
Both are used to enable instances in a private subnet to connect to the internet or other AWS services
An Internet Gateway (IGW) allows resources within your VPC to access the internet, and vice versa.
- In order for this to happen, there needs to be a routing table entry allowing a subnet to access the IGW.
NAT Gateway is only from instance to internet (you can download things from the EC2, but internet can't access the server).
- The internet at large cannot get through your NAT to your private resources unless you explicitly allow it.
Nat Gateway can only scale up to 45GB. Keep in mind if bandwidth is an issue.

Network ACL's

VPC automatically comes with default network ACL and by default allows all in/outbound traffic
You can create custom network ACL's. By default each network ACL denies all in/outbound traffic
Each subnet in VPC must be associated with a network ACL, uses default ACL by default
You can associate network ACL with multiple subnets, however subnet can only associate with one ACL at a time
Adding subnet to a second ACL will automatically remove it from the previous ACL
network ACL contains numbered list of rules that is evaluated in order, starting with lowest number
network ACL always have separate inbound and outbound rules
network ACL's are stateless
- VPC Interface Endpoints -- API Gateway, Cloudwatch, Config, Kinesis, SNS, etc.
- VPC Gateway Endpoints This is used so the traffic does not go out to the Internet and back in (remains in the private network, faster and more secure) -- S3 -- DynamoDb

CapitalOne AWS breach & AWS security discussion

Troy — Tue, 30 Jul 2019 17:04:56 +0000

CapitalOne was recently victim to a leak that may have affected 100 million individuals in the US. This post will not go into the unethical intent of the accused or the affect of private information being leaked, but rather the technical aspects and securing your AWS environment. If you'd like to read into it further, this document does a fair job at explaining it.

The diagram below is my understanding of how the CapitalOne S3 data was compromised. Background information about the accused also states that she worked for Amazon Web Services from 2015-2016 in the S3 division. Knowing that she worked for Amazon Web Services for a period of time very well may have affected her ability to compromise CapitalOne.

There seems to be still unanswered questions, such as:

What credentials were exploited for the accused to assume the STS role to gain access?
Why wasn't AWS GuardDuty's trusted list being maintained?
Why was the STS role that was assumed allowed access to the S3 bucket? A strict bucket policy with allowed ARN's should've been used.

What is your opinion on the technical aspect of the exploitation and securing an AWS environment?

draw.io with Git source control management?

Troy — Thu, 25 Jul 2019 13:56:28 +0000

I'm a huge fan of draw.io - a tool used to draw network topology, UML, architecture & more. Our team uses the tool pretty heavily, albeit independent to each other and not necessarily sharing.

I had a thought.. you have the ability to export as .xml. Each user edits their diagram, exports as .xml (to device) and adds to the SCM repository. If another user wants to edit it, they check-out the branch.. make their edits and check in/add a pull request/etc.

Quick diagram using draw.io:

draw.io + SCM = success?

Has anyone else used this type of combination?

Discussion: Why doesn't Netflix, one of the most shared streaming services fail to offer 2FA (two factor authentication)?

Troy — Mon, 22 Jul 2019 01:33:18 +0000

Netflix, one of the largest streaming services in the world maintains millions of subscribers a year. This post doesn't cover the content or the subscribers, but rather poses an excellent question.

Netflix: why no 2FA for the login process?!

2FA, also known as multi-factor authentication or two factor authentication provides an additional layer of security for an authentication mechanism.
WIKI definition:

Multi-factor authentication (MFA) is an authentication method in which a computer
user is
granted access only after successfully presenting two or more pieces
of evidence (or factors) to an
authentication mechanism: knowledge (something the user and only
the user knows), possession (something the user and only the user
has), and inherence (something the user and only the user
is).[1][2]

Two-factor authentication (also known as 2FA) is a type, or subset, of multi-factor authentication. It is a method of confirming
users' claimed identities by using a combination of two different
factors: 1) something they know, 2) something they have, or 3)
something they are.

A good example of two-factor authentication is the withdrawing of
money from an
ATM; only the correct combination of a bank
card (something
the user possesses) and a
PIN (something the user knows) allows
the transaction to be carried out.

Two other examples are to supplement a user-controlled password with a
one-time password (OTP) or code generated or received by an
authenticator (e.g. a security token or smartphone) that only the
user
possesses.[3]

Two-step verification or two-step authentication is a method of confirming a user's claimed identity by utilizing something
they know (password) and a second factor other than something they
have or something they are. An example of a second step is the user
repeating back something that was sent to them through an
out-of-band
mechanism. Or, the second step might be a six digit number generated
by an app
that is common to the user and the authentication
system.[4]

Netflix does not currently offer any forms of the above security. Why? Many claim that the engineering effort would not be worth it, or that their is not private information to protect. I'd argue these points and state that your:

Mailing address
Billing address
Last four of your credit card or PayPal (or billing method)

would be considered private information among many. A problem of unauthorized login sharing of Netflix credentials is rampant; 2FA would assist with preventing this.

The question is -- what's your take on why Netflix has yet to implement increased security measures for its users? Why no 2FA?

Diagramming your AWS VPC & IAM structure

Troy — Mon, 22 Jul 2019 00:52:01 +0000

The snowball effect for organizations & startups using AWS is a real thing. You may start experimenting with using one of the cloud platforms (AWS in this article) and soon to find out you have quite the labyrinth of policies, groups, users, access keys and more. This handy tool developed by the security company DUO (now owned by Cisco) will help you untangle the ball of yarn that started with a back & forth between you and the developers.

DUO CloudMapper is a tool that has grown from originally diagramming your AWS Virtual Private Cloud to now including IAM reports and much more.

A few examples of my personal favorite components of the CloudMapper tool are the resource aggregation along with the IAM reporting for identifying policy best-practices and potential problems.

Here are a few screenshots from the tool and it's components:

For installation, code repository and more visit the official DUO CloudMapper GitHub page. Cheers!

AWS Lambda backups of Route53 -> S3

Troy — Fri, 19 Jul 2019 17:21:48 +0000

Need a way to automatically back up your AWS Route53 public DNS zones? Look no further, as a combination of the following AWS products can fit the need:

Lambda
Route53
CloudWatch
S3

Here's a diagram of the process:

This will execute a Lambda function every 6 hours (or whichever you set the CloudWatch event to). It will use the IAM role to export your Route53 public zones as a CSV & JSON to the S3 bucket of your choice.

Create a S3 private bucket, as it will be your destination for the backups.
Set the s3_bucket_name variable to your AWS S3 bucket name.
Set the s3_bucket_region variable to your AWS S3 region.
Create an IAM role with an attached policy for Route53 read-only and S3 read/write to your S3 Bucket. I have provided an example here.
Create a CloudWatch event for every 6 hours (or desired recurring duration).
Upload the below Lambda Python function (copy and save it as aws_s3_route53.py for example).
Assign the execution role to the IAM role created in step 4, and use the scheduled CloudWatch event trigger created in step 5.
Check the S3 bucket for your backups and verify.

Now you can sleep a bit more peacefully knowing that you when\if you blow out a record-set in your hosted public zone, you'll have a backup!

aws_s3_route53_backups.py

"""AWS Route 53 Lambda Backup"""

import os
import csv
import json
import time
from datetime import datetime
import boto3
from botocore.exceptions import ClientError


# Set environmental variables

s3_bucket_name = ''
s3_bucket_region = ''

try:
    s3_bucket_name = os.environ['s3_bucket_name']
    s3_bucket_region = os.environ['s3_bucket_region']
except KeyError as e:
    print("Warning: Environmental variable(s) not defined")


# Create client objects

s3 = boto3.client('s3', region_name='us-east-1')
route53 = boto3.client('route53')


# Functions

def create_s3_bucket(bucket_name, bucket_region='us-east-1'):
    """Create an Amazon S3 bucket."""
    try:
        response = s3.head_bucket(Bucket=bucket_name)
        return response
    except ClientError as e:
        if(e.response['Error']['Code'] != '404'):
            print(e)
            return None
    # creating bucket in us-east-1 (N. Virginia) requires
    # no CreateBucketConfiguration parameter be passed
    if(bucket_region == 'us-east-1'):
        response = s3.create_bucket(
            ACL='private',
            Bucket=bucket_name
        )
    else:
        response = s3.create_bucket(
            ACL='private',
            Bucket=bucket_name,
            CreateBucketConfiguration={
                'LocationConstraint': bucket_region
            }
        )
    return response


def upload_to_s3(folder, filename, bucket_name, key):
    """Upload a file to a folder in an Amazon S3 bucket."""
    key = folder + '/' + key
    s3.upload_file(filename, bucket_name, key)


def get_route53_hosted_zones(next_zone=None):
    """Recursively returns a list of hosted zones in Amazon Route 53."""
    if(next_zone):
        response = route53.list_hosted_zones_by_name(
            DNSName=next_zone[0],
            HostedZoneId=next_zone[1]
        )
    else:
        response = route53.list_hosted_zones_by_name()
    hosted_zones = response['HostedZones']
    # if response is truncated, call function again with next zone name/id
    if(response['IsTruncated']):
        hosted_zones += get_route53_hosted_zones(
            (response['NextDNSName'],
            response['NextHostedZoneId'])
        )
    return hosted_zones


def get_route53_zone_records(zone_id, next_record=None):
    """Recursively returns a list of records of a hosted zone in Route 53."""
    if(next_record):
        response = route53.list_resource_record_sets(
            HostedZoneId=zone_id,
            StartRecordName=next_record[0],
            StartRecordType=next_record[1]
        )
    else:
        response = route53.list_resource_record_sets(HostedZoneId=zone_id)
    zone_records = response['ResourceRecordSets']
    # if response is truncated, call function again with next record name/id
    if(response['IsTruncated']):
        zone_records += get_route53_zone_records(
            zone_id,
            (response['NextRecordName'],
            response['NextRecordType'])
        )
    return zone_records


def get_record_value(record):
    """Return a list of values for a hosted zone record."""
    # test if record's value is Alias or dict of records
    try:
        value = [':'.join(
            ['ALIAS', record['AliasTarget']['HostedZoneId'],
            record['AliasTarget']['DNSName']]
        )]
    except KeyError:
        value = []
        for v in record['ResourceRecords']:
            value.append(v['Value'])
    return value


def try_record(test, record):
    """Return a value for a record"""
    # test for Key and Type errors
    try:
        value = record[test]
    except KeyError:
        value = ''
    except TypeError:
        value = ''
    return value


def write_zone_to_csv(zone, zone_records):
    """Write hosted zone records to a csv file in /tmp/."""
    zone_file_name = '/tmp/' + zone['Name'] + 'csv'
    # write to csv file with zone name
    with open(zone_file_name, 'w', newline='') as csv_file:
        writer = csv.writer(csv_file)
        # write column headers
        writer.writerow([
            'NAME', 'TYPE', 'VALUE',
            'TTL', 'REGION', 'WEIGHT',
            'SETID', 'FAILOVER', 'EVALUATE_HEALTH'
            ])
        # loop through all the records for a given zone
        for record in zone_records:
            csv_row = [''] * 9
            csv_row[0] = record['Name']
            csv_row[1] = record['Type']
            csv_row[3] = try_record('TTL', record)
            csv_row[4] = try_record('Region', record)
            csv_row[5] = try_record('Weight', record)
            csv_row[6] = try_record('SetIdentifier', record)
            csv_row[7] = try_record('Failover', record)
            csv_row[8] = try_record('EvaluateTargetHealth',
                try_record('AliasTarget', record)
            )
            value = get_record_value(record)
            # if multiple values (e.g., MX records), write each as its own row
            for v in value:
                csv_row[2] = v
                writer.writerow(csv_row)
    return zone_file_name


def write_zone_to_json(zone, zone_records):
    """Write hosted zone records to a json file in /tmp/."""
    zone_file_name = '/tmp/' + zone['Name'] + 'json'
    # write to json file with zone name
    with open(zone_file_name, 'w') as json_file:
        json.dump(zone_records, json_file, indent=4)
    return zone_file_name


## HANDLER FUNCTION ##

def lambda_handler(event, context):
    """Handler function for AWS Lambda"""
    time_stamp = time.strftime("%Y-%m-%dT%H:%M:%SZ",
        datetime.utcnow().utctimetuple()
    )
    if(not create_s3_bucket(s3_bucket_name, s3_bucket_region)):
        return False
    #bucket_response = create_s3_bucket(s3_bucket_name, s3_bucket_region)
    #if(not bucket_response):
        #return False
    hosted_zones = get_route53_hosted_zones()
    for zone in hosted_zones:
        zone_folder = (time_stamp + '/' + zone['Name'][:-1])
        zone_records = get_route53_zone_records(zone['Id'])
        upload_to_s3(
            zone_folder,
            write_zone_to_csv(zone, zone_records),
            s3_bucket_name,
            (zone['Name'] + 'csv')
        )
        upload_to_s3(
            zone_folder,
            write_zone_to_json(zone, zone_records),
            s3_bucket_name,
            (zone['Name'] + 'json')
        )
    return True


if __name__ == "__main__":
    lambda_handler(0, 0)