DEV Community: Prabusah

AWS Parameter and Secrets Lambda extension - Node.js example

Prabusah — Sat, 26 Nov 2022 15:18:59 +0000

TLDR;

This blog walks through how to access values stored in AWS Systems Manager Parameter Store via Lambda extension using Node.js code.

What is Lambda extension:

AWS releases Lambda extensions as Layer to make developers life easier by helping them integrate Lambda with other AWS Services features (like AppConfig, AWS Systems Manager Parameter Store etc.).

How Lambda extension works:

Lambda lifecycle has 3 phases: init, invoke and shutdown.
Init phase - Combination of Extension INIT, Runtime INIT and Function INIT. Extension setup happens during Extension INIT phase.
Invoke phase - Extension exposes HTTP endpoint that can be called from Lambda function runtime.
Shutdown phase - Extension runtime shutdown along with Lambda function runtime.

Why use AWS Systems Manager Parameter Store:

To store connection details, credentials or keys etc.

How AWS Parameters and Secrets Lambda extension works:

Provides in-memory cache for parameters and secrets. Upon Lambda requesting a parameter, the extension fetches the parameter data from local cache, if available. If data not in cache or stale, the extension fetches parameter value from AWS Systems Manager service. This reduces aws-sdk initialization, API calls, reduces cost and improves application performance.

Nodejs example:

const http = require('http');

let getParameterValue = function(paramName) {
    const headers = {
        "X-Aws-Parameters-Secrets-Token': process.env.AWS SESSION TOKEN
    }

    let options = {
        host: "localhost',
        port: '2773',
        path: `/systemsmanager/parameters/get?name=${paramName}`,
        method: 'GET',
        headers: headers
    }

    return new Promise((resolve, reject) => {
        const req = http.get(options, (res) => {
            if (res.statusCode < 200 || res.statusCode >= 300) {
                return reject(new Error('statusCode=' + res.statusCode));
            }
            var body = [];
            res.on('data', function(chunk) {
                body.push(chunk);
            });
            res.on('end', function() {
                resolve(Buffer.concat(body).toString());
            });
        });
        rea.on('error', (e) => {
            reject(e.message);
        });
        req.end();
    });
};

exports.handler = async (event) => {
    let pass = await getParameterValue('/serivce/password');
    let passValue = JSON.parse(pass).Parameter.Value;
    //passValue has the password value
};

Code walkthrough:

AWS Parameters and Secrets Lambda extension exposes HTTP endpoint localhost under 2773 port to Lambda function runtime. AWS SESSION_TOKEN is an in-built environment variable populated by AWS internally. If this secret token not passed to HTTP endpoint - a 401 error will occur.

Parameter store Securestring value retrieval using extension:

Just add '&withDecryption=true' to the suffix of options objects path field-given below:

let options = {
        host: 'localhost',
        path: `/systemsmanager/parameters/get?name=${paramName}&withDecryption=true`,
        port: '2773', 
        headers: headers
        method: 'GET',
}

Image by Radosław Kulupa from Pixabay

AWS Lambda (Node.js) calling SOAP service

Prabusah — Tue, 22 Nov 2022 16:30:30 +0000

REST service:

Uses HTTP for exchanging information between systems in several ways such as JSON, XML, Text etc.

SOAP service:

A Protocol for exchanging information between systems over internet only using XML.

Requirement:

Calling SOAP services from Lambda.
We'll use this npm package: https://www.npmjs.com/package/soap
With this npm package - its a 3 step process:

Create a soap client
Call SOAP method by passing JSON input.
Convert XML output to JSON

Create SOAP client:

const soap = require('soap');
let client = await soap.createClientAsync(url);

url - is the wsdl url. For example: http://sampledomainname/services/sampleserice?wsdl (just for representation - this wsdl url does not works).

wsdl doc has all the details. But if you are like me who can best understand JSON data then use client.describe() to get soap service details in JSON format.

console.log(client.describe());
Output log below:

{
  "SomeService": {
    "SomeServicePort": {
      "someServiceMethod": {
        "input": {
          "serviceRequest": {
            "field1": "xsd:string"
          }
        }
      }
    }
  }
}

Convert JSON to XML (pass input to SOAP): soap npm package takes care of this conversion. Pass JSON, & converts to XML.

Call SOAP method by passing JSON input:

client.SomeService.SomeServicePort_http.someServiceMethod(args, function(err, result) {
    if(err) console.error("Error - ", err); 
    console.log(result); // is a javascript object
}, { postProcess: function(_xml) { 
    console.log('XML - ', _xml); //this prints the input XML
    return _xml.replace('test', 'newtest'); // any mapping or string conversion, add here.
}});

Consider this args JSON as input to soap method

Let args = {
"one":"1"
}

The args JSON would be converted by soap npm package as below XML:

<?xml version=\"1.0\" encoding=\"utf-8\"?>
 <soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\"    xmlns:xsi=\"http://www.w3.org/2001/XMLSchema- instance\">
    <soap:Body>
        <ns1:someServiceMethod>
            <ns1:one>1</ns1:one>
        </ns1:someServiceMethod>
    </soap:Body>
 </soap:Envelope>

Before we discuss about response XML to JSON (result of soap call). Let's see how to use async/await way of calling soap method.

let resultArr = await client.someServiceMethodAsync(args);

Just add suffix Async to the soap method you wish to call, would return a promise. Note the above call results an array with 4 elements in both success and failure scenarios.

Array element 0: result is a javascript object.
Array element 1: rawResponse is the raw xml response string.
Array element 2: soapHeader is the response soap header as a javascript object (contains WS Security info-so let us not log this!).
Array element 3: rawRequest is the raw xml request string.

So the Array element O is already converted the XML response to JSON. Let's put altogether below:

const soap= require('soap');
let client = await soap.createClientAsync("http://sampledomainname/services/sampleserice?wsdl");
let resultArr= await client.someServiceMethodAsync(args); console.log("Response JSON-, resultArr[0]);

Secured Soap Call:

All enterprise soap services would require authentication and in below section let's discuss about WSSecurity.

let wsSecurity = new soap. WSSecurity('username', 'password", options); client.setSecurity(wsSecurity):

Putting altogether:

const soap = require('soap');
let client = await soap.createClientAsync('http://sampledomainname/services/sampleserice?wsdl'); 
let options = {
  hasNonce: true,
  actor: 'actor'
}
let wsSecurity = new soap.WSSecurity('username', 'password', options); 
client.setSecurity(wsSecurity); 
let resultArr = await client.someServiceMethodAsync(args); 
console.log('Response JSON-', resultArr[0]);

Image by SplitShire from Pixabay

Amazon EBS - Primer

Prabusah — Fri, 18 Nov 2022 17:10:40 +0000

Amazon Elastic Block Storage:

This is raw storage in storage devices such as Hard disk drives (HDDs), Solid state drives (SSDs) and Non-Volatile Memory Express (NVMe) that uses disk or volume.

HDD: Legacy technology. Data stored in spinning disks. Read/write speed of 80MB/s to 160 MB/s.
SSD: Data stored in integrated circuits. Read/write speed of 600 MB/s.
NVMe: Data stored in integrated circuits. Read/write speed of 3.5 GB/s.

Blocks and Volumes:

Disk or volume to be formatted as continuous blocks. Block - Fixed storage unit to store data. Volume - Block storage devices can be combined into larger logical units called volumes.

Components of Block storage:

Contains 3 components: Block storage, compute system and operating system (OS). Block storage attached to compute system. OS identifies the block storage and formats to make it ready for use.

AWS block storage:

Instance storage: Ephemeral (temporary) storage that is non-persistent and terminated when associated EC2 instance is terminated. Use for buffers, caches or other temporary content.

Amazon EBS storage: Persistent store. If an EC2 instance goes down, volume and data on volume remain available to attach to different EC2 instance.

Block storage service designed for use with Amazon EC2. EBS volumes suited for file systems, databases or any application that requires access to raw, block- level storage. Best for random reads/writes (DBs) and long, sequential, reads/writes as well.

Snapshots:

Point-in-time copies of data in EBS volumes. Backup from EBS to S3. Incremental copies (only blocks on EBS volumes that changed after most recent snapshot are saved).
Delete snapshot-only data unique to that snapshot removed.
Snapshot of encrypted volumes auto encrypted. Copy snapshots across Regions.

Snapshot Use Cases:

Host Microsoft Sharepoint, SAP, Exchange server etc. Bring your relational/non-relational DB into EBS attached to EC2. Bring your file system.

Amazon Data Lifecycle Manager (DLM):

DLM used to automate creation, retention and deletion of snapshots to backup your EBS volumes.

EBS Availability:

AWS auto replicates EBS volume within the AZ to prevent failure of single hardware component. But what if that AZ itself down? It is recommended to create snapshots of EBS volumes frequently. Snapshot replicated across all AZs within a Region. Snapshots can also be copied to other Regions.

EBS Types:

SSD: (gp2, gp1, io1, io2) and HDD: (st1, sc1).
iops: Input Output per second.
gp: general purpose;
io: Provisioned IOPS.
st1: Throughput Optimized HDD.
sc1: Cold HDD.

EBS Pricing:

Pay for Provisioned volume size, IOPS and throughput performance.
Snapshot Pricing: Actual amount of storage space consumed (not provisioned).

Basic Architecture:

An EC2 instance can have multiple EBS volumes (and different EBS volume types as well) attached. EC2 & EBS must reside in same AZ. Each EBS volume --> snapshots stored in S3 within same Region where EBS volume resides.

Advanced Architecture - Multi Attach:

Multiple EC2 instances connected to a single EBS volume. Data consistency to be managed by your application or OS environment. Multi-Attach supported only with Provisioned IOPS SSD (io1, io2) EBS volume types. EC2 & EBS must reside in same AZ. EC2 & EBS must reside in same AZ. Each EBS volume --> snapshots stored in S3 within same region where EBS volume resides.

Advanced Architecture-Striped volumes:

Multiple EBS volumes operate as single EBS volume attached to a single EC2 instance. EC2 & EBS must reside in same AZ. Each EBS volume--> snapshots stored in S3 within same Region where EBS volume resides.

Security:

IAM:

Policy created to allow users, groups, roles to access EC2 and EBS resources.

Encryption:

Occurs on servers that host EC2 instances, both data at rest and data in transit between EC2 and EBS are encrypted. Both encrypted/unencrypted volumes can be attached to an EC2 instance.
Data inside EBS volume, Data moving between EBS and EC2 instance. Snapshots created out of EBS volume and EBS volumes created out of snapshots-all can be encrypted.

AWS Backup:

Deploy backup policy across AWS accounts in Organization for services like EC2, EBS, RDS etc. Like snapshots AWS Backup also stores data backups of EBS in S3 bucket.
AWS Backup backups many services including EBS... whereas snapshot is to backup only EBS volume data. AWS Backup offers more features compared to snapshots.

AWS Compute Optimizer:

EBS sends data points (metrics) to Amazon CloudWatch. 1-minute metrics. Compute Optimizer uses Amazon CloudWatch metrics to analyze your EBS volumes and provide recommendations to assist you in optimizing your Amazon EBS costs.
(CloudWatch notifies events based on EBS changes like creation of volumes or snapshot etc.)

Image by lisa runnels from Pixabay

Amazon S3 - Pricing

Prabusah — Sat, 12 Nov 2022 13:19:17 +0000

Storage:

Pay for storing the data based on Object size, How long stored during a month, storage classes, monitoring & automation fee per object for S3 Intelligent-Tiering, per-request ingest fees while PUT, COPY or lifecycle rules moving to any S3 storage class.

Requests and data retrieval

Pay for API calls such as GET, LIST and browsing Amazon console.

Data transfer

Charged for bandwidth into and out of S3.
Except:
Data transferred in from internet.
Data transferred between S3 buckets in same region.
Data transferred out from S3 bucket to any AWS Services in same region.
Data transferred out to CloudFront.

Management

Pay for Inventory, Analytics, S3 Lens, S3 Batch Operations, S3 Select and Object tagging.
Pay for Replication and Object Lambda.

Image by Ja! from Pixabay

Amazon S3 - Managing

Prabusah — Sat, 12 Nov 2022 07:26:28 +0000

AWS S3 Managing:

Managing object tagging:

Tagging used to Manage/search/filter resources (including S3 object).
10 tags per object. 50 tags per bucket.
Tag Key - 128 chars; Tag Value - 256 chars.

Tagging in Access control - In IAM policy use tag and its value as condition for fine-grained permission to allow/deny actions.

Tagging in Lifecycle management - use tag to filter subset of objects to apply rule.

Tagging in Replication - filter by tag

Amazon S3 Inventory:

Get list of objects and corresponding metadata from a bucket or prefix on a daily/weekly basis. A less costly alternative is S3 inventory compared to List API. Use it to audit the replication/encryption/compliance/regulatory status of objects.
It may take up to 48 hrs for inventory to delivery first report. Inventory can be queried using Amazon Athena. Create a table, load inventory and query.

Amazon S3 Select:

Filter contents of S3 objects and retrieve subset of data instead of downloading entire object using SQL statements.
Reduces transfer costs.
Works on only CSV, JSON, Apache Parquet objects

Amazon S3 Event Notification:

Notifies object creation regardless of API used PUT, POST, COPY. Similarly event notification can be sent for DELETE, REPLICATION events etc.

Amazon S3 Batch Operations:

To perform a single API action like Copy/Delete/Tags related etc. or invoke a Lambda to perform on a list of S3 objects (can be billions of objects) you specify.
Auto retries on failure.
S3 tracks progress, notifies and stores report of actions.
Fully managed.

Amazon CloudWatch:

Which objects are accessed the most or the least and who is accessing.

Amazon S3 server access logging provides info about customer base.

Amazon S3 - Business continuity and Disaster recovery

Prabusah — Fri, 11 Nov 2022 18:31:41 +0000

Business Continuity:

Keeps business functioning despite significant disruptive events.

Disaster Recovery:

Natural or Human made event that causes an impact to business.

S3 for Business Continuity and Disaster Recovery:

S3 provides 99 point 11 nine's durability. Stored across min of 3 AZs (except S3 One Zone-IA storage class).

S3 Object Lock:

Immutable data (regulatory requirement). Replication - Increase availability.

Versioning:

Multiple variants of object.
Recovery from unintended user actions and application failures.
Overwrite creates new version. Deletion creates a delete marker instead of removing object.
Default - unversioned. But once enabled, can't return to unversioned state. Versioning can be suspended to stop accruing new versions.

Even in unversioned (default state)- all objects have version ID (null). Upon enabling versioning, the existing objects unchanged .ie. their version ID remains same (null). Delete Object (without versionID)-delete marker is set. And when we retrieve (current version) - 404 returned.

Removing delete markers:

Delete (Object + versionId)

S3 Lifecycle management:

Transition actions, when objects transition to another S3 storage class.
Expiration actions, when objects expire (versioning enabled)-S3 expires objects by adding delete marker.
Best practice: Move non-current version to Glacier class then delete after 1 year.

S3 Object lock:

Only in versioned buckets.
WORM-Write Once Read Many model. Prevent objects from deleted/overwritten for fixed time/indefinitely.
Retention period - time object can't be overwritten/deleted. Legal holds - No expiration date.
Configure bucket for Object Lock. Both can be at object level.

Versioning auto enabled when you create bucket with S3 Object Lock. S3 Object lock protection also moved between storage classes during Lifecycle transitions.

Indefinite locking-use Legal holds (because no retention period). Apply/change object lock operations for even billions of objects using $3 Batch operations.

Object Lock retention modes:

Compliance mode - Immutable until retention period. No one can delete/overwrite including root user. Also retention period cannot be edited. Delete entire AWS account to delete the file.
Governance mode - Specific users given permission to alter retention settings/delete objects.

Object Replication:

Replicate all objects or subset (use prefix/tags).
Replicates objects in same storage class as source object (default settings - but can specify different storage class for replicas).
Default, replicates tags, Object Lock settings. 99.99% of objects replicated in seconds.

S3 Multi-Region Access Points:

Request --> Multi-Region Access Points --> Request routed to less latency (closest) region (enable cross-region replication)
Region1
Region2

Amazon S3 - Scaling

Prabusah — Fri, 11 Nov 2022 17:30:33 +0000

Amazon S3 Performance Optimization

S3 Bucket Prefixes - Scale for high request rates. Amazon S3 supports up to 3,500 PUT/POST/DELETE and 5,500 GET transactions per second (TPS) per partitioned prefix.

Insurance-bucket/Auto = 3,500 PUT/POST/DELETE and 5,500 GET TPS.

Insurance-bucket/Life = 3,500 PUT/POST/DELETE and 5,500 GET TPS.

If request to the prefix exceed the supported request rate would result in HTTP 503 errors.

Avoid date based prefixes as that might result in exceeding request rate while the business grows per day basis.

Scaling connections horizontally:

S3 is a large-scale distributed system. So we can make parallel requests. No limits on number of connections made to a bucket Write/retrieve data from S3 in parallel using multipart uploads.

Use Amazon CloudFront:

Browser ->>> CF->>> S3

Amazon S3 - Data Lake

Prabusah — Fri, 11 Nov 2022 17:08:39 +0000

Data Lake:

A centralized repository that allows to migrate, store and manage all structured/unstructured data at unlimited scale. Once centralized, we can extract value and gain insights from data through analytics and ML. This makes the data available to more users across more lines of business - enables them to get insights they need.
S3 is ideal for Data Lake as it provides unlimited scalability.

Data Cataloging:

On put of S3-> Use lambda to extract metadata -> DynamoDB and ElasticSearch -> then query the data.

AWS Glue:

Fully managed ETL service. It can organize, cleanse, validate and format data.

In-Place data querying:

Without provisioning and managing servers/clusters we can transform/query the data. So no need to copy and load data into separate analytics platforms. Athena and Redshift Spectrum provide in-place querying of S3 data lake.

Amazon Athena:

Interactive query service that analyze data directly in S3 using SQL Serverless. Pay for scanned data while running queries. Integrates with QuickSight for easy visualization.

Redshift Spectrum:

More complex queries with large number of data lake users can run concurrent workloads.

Amazon S3 Storage Classes

Prabusah — Tue, 01 Nov 2022 14:06:46 +0000

Amazon S3 Storage classes broadly classified as below:

Frequently accessed tier
Infrequently accessed tier
Unknown or Changing access tier
Archive tier
Deep Archive tier

Amazon S3 Standard (Frequently accessed):

Default storage class. Data can be accessed in milliseconds and
usually most frequently accessed data stored.
Supports Resilient/Availability as data persisted in multiple AZs.
No min/max storage duration. No min size. No retrieval fees.

Amazon S3 Standard-Infrequent Access (Infrequently accessed):

Data can be accessed in milliseconds.
Supports Resilient/Availability as data persisted in multiple AZs.
To be stored minimum 30 days in this class. If an object deleted before 30 days, still charged for 30 days. Minimum file size 128 KB size. If uploading <128KB file, still charged for 128KB.
Charged for data retrieval.

Amazon S3 One Zone-Infrequent Access (Infrequently accessed):

Data can be accessed in millisecond access.
Less Resilient / Less Availability as data persisted only in Single AZ (so less cost).
To be stored minimum 30 days in this class. If an object deleted before 30 days, still charged for 30 days. Minimum file size 128 KB size. If uploading <128KB file, still charged for 128KB.
Charged for data retrieval.

Amazon S3 Intelligent-Tiering (Unknown or Changing Access):

Automated cost savings by monitoring and analyzing access patterns, moves the data to appropriate storage class.
Supports Resilient/Availability as data persisted in multiple AZs.
At first, uses the frequent access tier. Monitors access patterns and moves objects not accessed for 30 (min) consecutive days to Infrequent tier.
Moves objects not accessed for 90 (min) consecutive days to Archive access tier (only when activated). Retrieval between 3-5 hrs.
Moves objects not accessed for 180 (min) consecutive days to Deep Archive access tier (only when activated). Retrieval within 12 hrs.
No min size. If uploading < 128KB file - not charged but eligible only for frequent access tier. No retrieval fees. No lifecycle fees. But monthly object monitoring & automation fees applies.

Amazon S3 Glacier Instant Retrieval (Archive tier):

Data can be accessed in milliseconds.
Supports Resilient/Availability as data persisted in multiple AZs.
Minimum data needs to be stored for 90 days. If an object deleted before 90 days, pro-rated charge for remaining days applies. Min 128 KB size. If uploading < 128KB file, still charged for 128KB
Charged for data retrieval.

Amazon S3 Glacier Flexible Retrieval (Archive tier):

Expedited, then data can be retrieved in 1-5 mins.
Standard, then data can be retrieved in 3-5 hours.
Bulk, then data can be retrieved in 5-12 hours for free.
Supports Resilient/Availability as data persisted in multiple AZs.
Minimum data needs to be stored for 90 days. If an object deleted before 90 days, pro-rated charge for remaining days applies.
Minimum object 40KB size expected. If uploading < 40KB file, still charged for 40KB.
Charged for data retrieval.

Amazon S3 Glacier Deep Archive (Deep Archive tier):

Default retrieval time of 12 hrs.
Supports Resilient/Availability as data persisted in multiple AZs.
Minimum data needs to be stored for 180 days. If an object deleted before 180 days, pro-rated charge for remaining days applies.
Minimum object 40KB size expected. If uploading < 40KB file, still charged for 40KB.

Image by wineguide101 from Pixabay

Amazon S3 Encryption

Prabusah — Tue, 01 Nov 2022 14:02:14 +0000

Encryption:

While in-transit .ie. data traveling to and from Amazon S3-use
SSL/TLS and while at rest .ie. data stored on disks in Amazon S3 data centers - use SSE, CSE.

Server Side Encryption options (SSE):

SSE-S3: AWS S3 Managed keys. Each object encrypted with unique key. Cost effective.
SSE-KMS: Customer Master Keys stored in AWS Key Management Service.
SSE-C: Customer-Provided keys

Client Side Encryption (CSE):

Encrypting data before sending it to Amazon S3.

Image by Markus Spiske from Pixabay

Amazon S3 Security

Prabusah — Tue, 01 Nov 2022 14:00:12 +0000

Amazon S3 Bucket accessible only to user who created or account owner. How to grant access to other users?. Follow any one the methods below:

IAM:

Create user and manage access to buckets/objects.
Contains permission for other than S3 as well.

Bucket policy (resource policy):

Using tags/prefixes configure permissions to all / set of objects. This must have principal.

Pre-signed URLs:

Grant time-limited access with temp URLs

ACL (resource policy):

This makes individual object accessible to users. This is Legacy, use bucket policies/IAM policies.

Block Public Access:

By default, any bucket created has "block all" public access.

Amazon S3 Object Ownership

Object usually owned by the account or user that uploaded it ("Bucket owner"). If other AWS account uploads an object then only that account is the owner. To overcome this, use "Amazon S3 Object Ownership" option.("Bucket owner preferred").

Image by Michal Jarmoluk from Pixabay

Amazon S3 Primer

Prabusah — Tue, 01 Nov 2022 12:12:30 +0000

Amazon S3 uses object storage type. Object data usually stored in bucket. Prefixes (pseudo folder structure) are used to group objects like folder structure in the user interface (AWS Console) but in reality, the object storage type is still a flat structure.

By default, we can create up to 100 buckets in a AWS account (by submitting service ticket this can be increased up to 1000 buckets). Bucket sizes are unlimited so user do not have to allocate or predetermine bucket size.

Few Bucket Info:

Cannot be transferred to other accounts.
Bucket names are globally unique within entire AWS S3 infrastructure. Once deleted from an AWS account, the name becomes available for reuse by any AWS account after 24 hours.
Bucket names cannot be renamed.
Bucket cannot be nested.
Bucket names can be 3-63 characters long.

Terminologies:

Prefixes - pseudo folder.
Key - name of the object.
Object - A file contains data, metadata (optional) and permissions. All 3 usually provided while uploading a file to bucket.

Bucket can have up to 50 tags and an Object can have up to 10 tags.
Any number of objects can be stored in a bucket.
Each object at max can hold 5 TB data.

Region:

In AWS Console - S3 is a globally viewable service. Bucket creation requires region that decides where the data resides.

Cross Region Replication:

Cross Region replication replicates bucket to other region. Entire bucket or use tags to replicate only the objects with the tags we choose.

Same Region Replication:

Source and Target buckets reside in same region.

Strong data consistency:

Strong read-after-write consistency. After list or write of new object or overwrite of existing object, any subsequent read request receives latest version of the object.

Versioning:

Enables recovery of objects from accidental deletion or overwrite.

GET operation:

Retrieve object. To retrieve a part of object, use the Range HTTP header in GET request.

DELETE operation:

No versioning then permanently deleted.
if version enabled then either permanently delete (Key + Version ID) or create delete marker for the object which can be recovered later (only Key name used without VersionID).
Recovery by removing the delete marker.
Retrieving an object that has delete marker returns 404 NOT FOUND.

PUT operation:

Adds an object to bucket. No partial write. Always completely writes the entire object.
In a single PUT operation - Upload up to 5 GB. (Max object size 5 TB). For >5GB size, then use multipart upload API.

Multipart upload API:

Uploads up to 5 TB data part by part up to 5 GB.

Best practice: To upload more than 100MB size of object, then use multipart upload.
S3 retains all parts on server until complete multipart upload is complete or discontinued. So if upload is incomplete then storage costs occurs for parts of data stored in S3. Use lifecycle rules to clean up incomplete multipart uploads automatically.

Online Data Transfer services:

AWS Data Sync, AWS Transfer family, Amazon Kinesis Firehose (direct to S3), Amazon Kinesis Data Streams (process streaming data).

Offline Data Transfer services:

AWS Snowcone up to 8 TB space.
AWS Snowball Storage optimized 40vCPUs; Compute optimized 52 VCPUs may be rack mounted to build larger installations.
AWS Snowmobile up to 100 PB. 45 foot long container with semi trailer-security personnel.

Hybrid cloud storage services:

On premise applications needs rapid data transfer/access to cloud.
AWS Direct Connect Dedicated network connection (without passing through internet) between on-premise to AWS. Uses VLAN.
AWS Storage Gateway NFS/SMB protocol connect to S3 bucket.

Image by Alexa from Pixabay