DEV Community

Adeline Makokha for AWS Community Builders

Posted on

3

AMAZON STORAGE GATEWAY

YOU HAVE DATA AND YOU WANT TO STORE IT IN THE CLOUD AND ACCESS IT ON-PREM

a

WE`LL BE TALKING ABOUT DATA THAT IS:

  • CONNECTED TO ON-PREM APPLICATIONS
  • PART OF CONTINUOUS HYBRID WORKFLOWS
  • BACKUP FOR CRITICAL SYSTEMS
  • IN BULK STORAGE

MOVING DATA TO THE CLOUD?

b

WHAT IS STORAGE GATEWAY?
A HYBRID STORAGE VIRTUAL APPLIANCE DESIGNED TO RUN IN A VIRTUAL ENVIRONMENT ON-PREM.
A HARDWARE APPLIANCE CAN ALSO BE USED
Connects an on-premises software appliance with cloud-based storage to provide seamless integration between your on-premises IT environment and the AWS storage infrastructure.

STORAGE GATEWAY ARCHITECTURE

c

THE THREE KEY HYBRID USE CASE
1.Move backups and archives to the cloud.
2.Reduce on-premises storage with cloud-backed file shares.
3.Provide on-premises applications low latency access to data stored in AWS

d

STORAGE GATEWAY MODES

STORAGE GATEWAY CAN BE IMPLEMENTED IN THREE WAYS:

  • FILE MODE(FILE STORAGE BACKE DBY S3) – NFS & SMB
  • VOLUME MODE(CACHE/STORED – CLOUD VOLUMES RUNNING ON-PREM) - iSCSI
  • TAPE GATEWAY MODE(VIRTUAL TAPE LIBRARY MODE) - iSCSI

FILE GATEWAY

  • File Gateway supports a file interface into Amazon Simple Storage Service (Amazon S3) and combines a service and a virtual software appliance.
  • By using this combination, you can store and retrieve objects in Amazon S3 using industry-standard file protocols such as Network File System (NFS) and Server Message Block (SMB).

  • The software appliance, or gateway, is deployed into your on-premises environment as a virtual machine (VM) running on VMware ESXi, Microsoft Hyper-V, or Linux Kernel-based Virtual Machine (KVM) hypervisor.

  • The gateway provides access to objects in S3 as files.

FILE GATEWAY ARCHITECTURE
m

FILE GATEWAY
n

h

WITH FILE GATEWAY

  • You can store and retrieve files directly using the NFS version 3 or 4.1 protocol.
  • You can store and retrieve files directly using the SMB file system version, 2 and 3 protocol.
  • You can access your data directly in Amazon S3 from any AWS Cloud application or service.
    -You can manage your S3 data using lifecycle policies, cross-region replication, and versioning. You can think of a file gateway as a file system mount on S3.
    HOW TO USE THE FILE GATEWAY

  • TO USE A FILE GATEWAY, YOU START BY DOWNLOADING A VM IMAGE FOR THE FILE GATEWAY.

  • YOU THEN ACTIVATE THE FILE GATEWAY FROM THE AWS MANAGEMENT CONSOLE OR THROUGH THE STORAGE GATEWAY API.

  • AFTER THE FILE GATEWAY IS ACTIVATED, YOU CREATE AND CONFIGURE YOUR FILE SHARE AND ASSOCIATE THAT SHARE WITHYOUR AMAZON SIMPLE STORAGE SERVICE (AMAZON S3) BUCKET.

  • DOING THIS MAKES THE SHARE ACCESSIBLE BY CLIENTS USING EITHER THE NETWORK FILE SYSTEM (NFS) OR SERVER MESSAGE BLOCK (SMB) PROTOCOL.

  • FILES WRITTEN TO A FILE SHARE BECOME OBJECTS IN AMAZON S3.

  • OBJECTS ARE ENCRYPTED WITH AMAZON S3–SERVER-SIDE ENCRYPTION KEYS (SSE-S3). ALL DATA TRANSFER IS DONE THROUGH HTTPS.

VOLUME GATEWAY

  • A VOLUME GATEWAY PROVIDES CLOUD-BACKED STORAGE VOLUMES THAT YOU CAN MOUNT AS INTERNET SMALL COMPUTER SYSTEM INTERFACE (ISCSI) DEVICES FROM YOUR ON-PREMISES APPLICATION SERVERS.
  • THE VOLUME GATEWAY IS DEPLOYED INTO YOUR ON-PREMISES ENVIRONMENT AS A VM.

VOLUME GATEWAY IS IMPLEMENTED IN TWO MODES:

  1. STORED VOLUMES
  2. CACHED VOLUMES

STORED VOLUMES

  • BY USING STORED VOLUMES, YOU CAN STORE YOUR PRIMARY DATA LOCALLY, WHILE ASYNCHRONOUSLY BACKING UP THAT DATA TO AWS.

  • STORED VOLUMES PROVIDE YOUR ON-PREMISES APPLICATIONS WITH LOW-LATENCY ACCESS TO THEIR ENTIRE DATASETS. AT THE SAME TIME, THEY PROVIDE DURABLE, OFFSITE BACKUPS.

  • YOU CAN CREATE STORAGE VOLUMES AND MOUNT THEM AS ISCSI DEVICES FROM YOUR ON-PREMISES APPLICATION SERVERS.

  • DATA WRITTEN TO YOUR STORED VOLUMES IS STORED ON YOUR ON-PREMISES STORAGE HARDWARE.

  • THIS DATA IS ASYNCHRONOUSLY BACKED UP TO AMAZON S3 AS AMAZON ELASTIC BLOCK STORE (AMAZON EBS) SNAPSHOTS
    HOW TO IMPLEMENT

  • After you install the AWS Storage Gateway software appliance—the VM—on a host in your data center and activate it, you can create gateway storage volumes.

  • You then map them to on-premises direct attached storage (DAS) or storage area network (SAN) disks.

  • You can then mount these storage volumes to your on-premises application servers as iSCSI devices.

  • As your on-premises applications write data to and read data from a gateway's storage volume, this data is stored and retrieved from the volume's assigned disk. To prepare data for upload to Amazon S3, your gateway also stores incoming data in a staging area, referred to as an upload buffer. You can use on-premises DAS or SAN disks for working storage.

  • Your gateway uploads data from the upload buffer over an encrypted Secure Sockets Layer (SSL) connection to the AWS Storage Gateway service running in the AWS Cloud.

  • The service then stores the data encrypted in Amazon S3.

STORED VOLUMES ARCHITECTURE-MIGRATION AND DISASTER RECOVERY

m

STORED VOLUMES UNDER THE HOOD

n

STORED VOLUME GATEWAY
x

CACHED GATEWAY MODE

  • YOU STORE YOUR DATA IN AMAZON SIMPLE STORAGE SERVICE (AMAZON S3) AND RETAIN A COPY OF FREQUENTLY ACCESSED DATA SUBSETS LOCALLY.
  • CACHED VOLUMES OFFER A SUBSTANTIAL COST SAVINGS ON PRIMARY STORAGE AND MINIMIZE THE NEED TO SCALE YOUR STORAGE ON-PREMISES.
  • YOU ALSO RETAIN LOW-LATENCY ACCESS TO YOUR FREQUENTLY ACCESSED DATA.

CACHED VOLUMES-EXTEND LIMITED STORAGE CAPACITY INTO AWS

y

UNDER THE HOOD

g

CACHED MODE
s

TAPE GATEWAY MODE

  • A TAPE GATEWAY PROVIDES CLOUD-BACKED VIRTUAL TAPE STORAGE. THE TAPE GATEWAY IS DEPLOYED INTO YOUR ON-PREMISES ENVIRONMENT AS A VM
  • WITH A TAPE GATEWAY, YOU CAN COST-EFFECTIVELY AND DURABLY ARCHIVE BACKUP DATA IN GLACIER OR DEEP_ARCHIVE.
  • A TAPE GATEWAY PROVIDES A VIRTUAL TAPE INFRASTRUCTURE THAT SCALES SEAMLESSLY WITH YOUR BUSINESS NEEDS AND ELIMINATES THE OPERATIONAL BURDEN OF PROVISIONING, SCALING, AND MAINTAINING A PHYSICAL TAPE INFRASTRUCTURE.

TAPE GATEWAY ARCHITECTURE-MIGRATE A PHYSICAL TAPE ARCHITECTURE INTO AWS

b

TAPE GATEWAY
f

SUMMARY
g

v

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read more

Top comments (0)

Best Practices for Running  Container WordPress on AWS (ECS, EFS, RDS, ELB) using CDK cover image

Best Practices for Running Container WordPress on AWS (ECS, EFS, RDS, ELB) using CDK

This post discusses the process of migrating a growing WordPress eShop business to AWS using AWS CDK for an easily scalable, high availability architecture. The detailed structure encompasses several pillars: Compute, Storage, Database, Cache, CDN, DNS, Security, and Backup.

Read full post