How does AWS storage gateway work?

AWS Storage Gateway helps in connecting on-premises software appliance with cloud-based storage which helps in provisioning seamless integration of data security features between on-premises IT environment and AWS storage infrastructure. This service can be used to store data in the AWS Cloud which helps in scalability and cost-effectiveness as well as providing data security. 

It offers file-based, volume-based, and tape-based storage solutions. They have been listed below: 

File Gateway

A file gateway helps in storing a file interface inside Amazon S3. It combines a service and a virtual software appliance. With the help of this, objects can be stored and retrieved from Amazon S3 using industry-standard file protocols like Network File System (NFS), AND Server Message Block (SMB). It is deployed into the on-premise environment in the form of a virtual machine that runs on VMware ESXi or Microsoft Hyper-V hypervisor. It also provides access to objects in Amazon S3. 

With the help of a file gateway the following actions can be performed:  

• Files can be stored and retrieved directly using the NFS version 3 or 4.1 protocol.  
• The files can be stored and retrieved directly using the SMB file system version, 2 and 3 protocol.  
• Files can be accessed directly in Amazon S3 from AWS cloud application or service.  
• The data in Amazon S3 can be managed with the help of life cycle policies, cross-region replication, and versioning. The file Gateway can be visualized as a file system Mount on S3. 

It helps simplify the file storage method in Amazon S3 since it integrates with the applications that are available currently, with the help of industry-standard file system protocols. It also provisions cost-effective storage in comparison to the on-premise data storage facility. It gives low latency data access with the help of transparent local caching.  

It can also be used to manage data transfer to and from AWS, buffering applications with network congestions, optimizing stream data in parallel and managing the consumption of bandwidth.  

AWS services which can be integrated with file gateway include: 

• Common access management system with the help of AWS IAM (Identity and Access Management). 
• Encrypting data using AWS KMS (Key Management Service). 
• AWS CloudTrail to audit data. 
• Monitor data with the help of Amazon CloudWatch.   
• Billing and cost management.  
• Operations which need to be performed with the help of AWS Management Console and AWS CLI (Command Line Interface).  

Using file gateways

When a file gateway is used, the VM image is downloaded, and it is activated for file gateway via the AWS Management Console or through the Storage Gateway API. The file gateway needs to be activated. The file share needs to be created and configured to associate it with the Amazon S3 bucket. This ensures that share can be accessed by clients using NFS or SMB protocol. There is a one-to-one mapping between the files and the objects, wherein the gateway updates the objects in Amazon S3 asynchronously. Objects are encrypted with the help of Amazon S3–server-side encryption keys (SSE-S3) and data transfer takes place via HTTPS. 

The below image shows how file gateway works:

Volume Gateway

It helps provision cloud backed storage volumes which can be mounted as Internet Small Computer System Interface (iSCSI) devices form the on-premises application servers. It supports the below mentioned volume configurations:

• Cached volumes – Data can be stored in Amazon Simple Storage Service (Amazon S3) and a copy of the frequently accessed datasets can be retained locally. Cached volumes help save costs when using primary storage, and also minimizes the need to scale the storage which is on premise. Low latency feature can be retained to access frequently required data.
• Stored volumes – When low-latency is required to access the entire dataset, the on-premise gateway needs to be configured to store all the data locally. Then snapshots need to be asynchronously backed up at every point in time to Amazon S3. This configuration helps in provisioning durability and cost effectiveness so that data can be recovered to the user’s local data center or Amazon EC2.

Usage of Volume Gateways

Volume gateways can be used with cached volumes and stored volumes. Stored volume deployment has been shown in the below image: 

Primary data is stored locally, and this data is asynchronously backed up to AWS. It helps in provisioning low-latency access to the entire data as well as providing durable and offsite backups. Volume storages can be maintained on premise in the data center itself.

Cached volumes deployment has been shown in the below image. Frequently accessed data is stored locally in the storage gateway. Cached volumes help in minimizing the requirement to scale out based on the infrastructure, as well as providing low-latency access to data objects.

Tape Gateway

It can be used to cost efficiently and durably archive backup data in GLACIER or DEEP_ARCHIVE. It provides a virtual tape infrastructure that helps in seamlessly scaling the business requirements and eliminating the operational overhead of providing, scaling, and maintaining physical tape infrastructure.  

Tape gateway is used when the user requires a cost-effective, durable, long-term, and offsite way to archive data. It consists of a virtual tape library (VTL) interface that can be used with the existing tape-based backup software infrastructure in order to store data on virtual tape cartridges which is created by the user. When archive tapes are used, the overhead of managing these tapes on premise and provisioning shipments of tapes offsite is eliminated.  

AWS Storage gateway can be run on premise or as a VM appliance or as a hardware appliance or in AWS as an Amazon EC2 instance. The gateway is then deployed on an EC2 instance so as to provide the iSCSI storage volumes in AWS. These gateways can be hosted on EC2 instances that would help in disaster recovery, data mirroring, and providing storage for applications which are hosted on Amazon EC2. Before a storage gateway can be deployed, the storage solution and the hosting option must be decided upon.  

Tape gateway deployment has been shown in the image below: 

It provisions durable and cost-effective solution in archiving the data stored on AWS Cloud. With the help of its VTL (Virtual tape library) interface, the tape data can be backed up, which will be available on iSCSI devices.

Storage Solution 

It can be chosen from file gateway, volume gateway or tape gateway. 

• File gateway: File gateway can be used to ingest files to Amazon S3 which can be used with object-based workloads, and to provide cost-effective storage for traditional backup applications. They can also be used to bind on premise file storage with S3. Data can be cost-effectively, and durably stored and retrieved from Amazon S3 with the help of industry-standard file protocols.  
• Volume Gateway: Storage volumes can be created in the AWS Cloud. The on premise applications can be used to access the data in these volumes in the form of iSCSI targets. This can be done using cached and stored volumes. 
• Cached volumes are used to store volume data in AWS that will have a portion of most recently accessed data in the on-premise cache. This method makes sure that the low-latency access feature is enabled to access the frequently accessed dataset. Seamless access to the entire dataset stored in AWS is provisioned. Storage resources can be scaled without having to provide additional hardware.  

Note: In stored volumes, the entire dataset/ data volume is stored on premise along with storing backups at certain points in time (also known as snapshots, which are useful in data backup and recovery). The data can be accessed with low latency.  

Hosting options in Storage Gateway 

Storage gateway can be run on premise as a VM appliance or as a hardware appliance or in AWS as an Amazon EC2 instance. If the data center where data is stored goes offline, and no host is available, the gateway can be deployed on an EC2 instance. Storage gateway provides AMI (Amazon Machine Image) which holds the gateway VM image.  

When a host is configured to be deployed to a gateway software appliance, sufficient storage needs to be allocated for the gateway VM.  

Note: When a tape gateway is deployed, the client backup software has to be installed.  

Pre-requisites 

When a gateway is deployed on premise, the type of host has to be chosen beforehand- VMware ESXi Hypervisor or Microsoft Hyper-V. This has to be set up based on the user requirements. If the gateway is deployed behind a firewall, it should be ensured that the ports can be accessed to the gateway VM.  

Conclusion 

In this post, we understood how storage gateway can be used to connect on premise software appliance with cloud-based storages.