Guide to FTP/SFTP access to an Amazon S3 Bucket

The significant majority of IT pros trust SFTP. They have been using this protocol for a while. SFTP is easy to use, works well with CLI, is free, runs anywhere, and can be configured with automation. But on the other hand, times are changing; public cloud storage brings new opportunities and innovation. Having access to a cloud-based Amazon S3 bucket storage that works well with different transfer protocols can be all you need.

So how can you use the best of both worlds? How are FTP/SFTP protocols designed to transfer files with Amazon S3 buckets, object-based storage?

In this guide, we’ll walk you through three different ways to FTP/SFTP access an AWS S3 bucket. First, we’ll set up an FTP server in AWS Transfer Family and transfer internally, and then we’ll use either a client like FileZilla or a third-party service such as Files.com to synchronize with the AWS S3 bucket.

1. Storage Units: FTP/SFTP vs. Amazon S3 Buckets

The FTP and SFTP protocols were designed to transfer files, while Amazon S3 buckets were designed to store objects. Although both are popular for sharing and storing data remotely, they work differently.

Files vs. Objects Storage

The transactional-based units for file-based storage are files. File transfer protocols such as FTP or SFTP (SSH File Transfer Protocol) access data in storage at the file level. They access files stored in a hierarchical folder and file structure. Another type of storage unit is objects, which contain the data itself, its associated (expandable and customizable) metadata, and a GUI (Globally Unique Identifier). Objects may be a file, a subfile, or a collection of unrelated bits and bytes. Amazon S3 buckets use objects. 

There is still another type of file format, known as blocks, and used in structured database storage and Virtual Machine File System (VMFS) volumes. Blocks are out of the scope in this guide.

	Object	File
Client	Via App (Amazon S3)	Via OS
Attributes	Custom Metadata	Fixed FS attributes
Sharing files	Shared semi-static file	Shared changing file
Scalability	Multi-site and highly scalable	Single-site. Simple access

Object outperforms file storage when it comes to storing massive amounts of unstructured data. Regardless of the amount of storage, things are easy to access and provide high stream throughput.

Generally, cloud-based object storage like Amazon S3 buckets will scale much better than SFTP or FTP server storage.

So, why would you store files when you can store objects?

• File storage is generally better for more minor storage requirements. For example, instead of storing billions of files, file-based storage like what SFTP provides is better for storing millions.
• Files are better for sharing and collaborating. Files are friendlier than objects for the casual cloud storage user. In the same way, configuring access to folders, subfolders, and files is much easier.

2. How does Amazon S3 Transfers files?

Amazon’s AWS S3 (Amazon Simple Storage Service) is the best example of an object storage service. A single object within S3 may range from a few Kilobytes to a couple of Terabytes. In AWS S3, objects are organized and stored in a “buckets” structure.

S3 is designed with a simple web services interface to store and get any amount of data from anywhere at any time. S3 is accessed using the secure web-based protocol HTTPS and a REST Application Programming Interface (API), also known as RESTful API.  

You can upload files, folders, or data into S3 with its simple drag and drop function.

All interactions with S3 happen at the application level via its RESTful API. S3 uses commands like PUT, GET, COPY, LIST, and DELETE to interact with the storage and coded as HTTP requests. That means to transfer files in the AWS console, you would need to use their built-in Amazon file transfer interface based on HTTPS.

3. Ways to FTP/SFTP to an AWS S3 Bucket?

Fortunately, AWS does provide full support for either FTP, FTPS, and SFTP, but not precisely over AWS S3. There are three options.

• The first option is using Amazon’s AWS Transfer Family, a fully managed service that allows file transfers directly into and out of Amazon S3. This option is easier to configure but gives you less control.
• The second option is to use a third-party service such as Files.com to mount S3 as you would with any file system. With this option, you can FTP/SFTP access all your files to AWS S3. This last option gives you more flexibility and control but requires additional configuration.
• The third option is to use a file transfer client that supports Amazon S3, like FileZilla or CyberDuck.

A. Using AWS Transfer Family to SFTP/FTP to S3

To use AWS Transfer for SFTP, follow these steps: Create and configure your S3 bucket > Create an FTP Server > Set up user accounts with credentials. > Link the FTP server to one or more S3 Buckets.

• If you don't have an S3 bucket, go ahead and create one.

• Configure your bucket. Give it a name, define its AWS region, access, versioning, and encryption. Once done, you should see your S3 bucket ready to be used.

• Go to your AWS console and open AWS Transfer for SFTP and Create a New Server.

• When configuring your new server, choose the protocol (SFTP, FTPS, or FTP). Preferably use SFTP.

• In Step 2, you’ll be able to set up the identity provider manager for user authentication and authorization. The user permissions will be enforced by an associated AWS role in the IAM (Identity & Access Management) service, but you can also provide a custom identity provider via the API.
• Next, configure endpoints (whether the resources will be reachable from the Internet or via a VPC and its hostname).
• Next, choose S3 as the default AWS Storage service. Notice that you also have access to Amazon’s EFS.

• Configure additional services, review, and create your server.
• It should look something like this.

• Although the SFTP server is up and running, it still can’t be accessed because it has no users. Click on the server ID, and click on “Add User.”

• Enter the name, policy, role, and select your S3 bucket as Home Directory when creating the user. Here, you’ll also need to provide the SSH public key.

Now that you have created an SFTP server using the AWS Transfer Family service, you can connect from an SFTP client, like FileZilla, using the server ID (hostname), the username/password, and the private SSH key. All files transferred from an SFTP endpoint via the SFTP server we created will be uploaded to the S3 bucket (pcwdld-01).

B. Using a file-sharing service such as Files.com – FREE TRIAL

Files.com is a cloud-based intelligent file-sharing solution designed for any size of business. It can help small, medium-large enterprises share files, manage and collaborate, and automate different file-sharing processes. Users upload files to the cloud-based storage, share them via links or inbound inboxes.

Files.com allows you to connect via FTP, SFTP, and WebDAV. In addition, it also supports AWS S3, along with other cloud storage providers.

With Files.com, you can mount an Amazon S3 bucket directly into Files.com so that you can access it like a folder. Alternatively, Files.com can be configured to sync individual folders and files with an S3 bucket. It can be configured to automatically push or pull files to/from your Amazon S3 in any region.

How to FTP to S3 via Files.com?

• Open your Files.com dashboard, go to “Integrations,” and find AWS S3.

• Select “Add Amazon S3 as a remote server.”

• When adding Amazon S3 as a remote server, you’ll need to provide the following information: Internal name for this connection, S3 bucket name, AWS Region, AWS access key ID, and AWS secret key. 

• The internal name for this connection: A custom identifier within Files.com that lets you find your connection later.
• S3 bucket name: A globally unique name. This namespace is shared by all AWS accounts. It cannot be used in another account and any AWS Region. To find this name, go to your Amazon S3, under Buckets, find the S3 bucket name. In my case, “pcwdld-01”.

• AWS Region: Enter the AWS Region where your bucket is located. Find this in the same Amazon S3 Bucket console. In my case, “EU (Paris) eu-west-3”.
• AWS Access Key ID and AWS Secret Key: Access Keys are the credentials for the IAM user or the AWS account root user. Access keys consist of an access key ID and a secret access key. These two keys must be used together to authenticate requests to AWS S3 resources. To find it, go to AWS IAM. If you don’t have an access key, create one. Download the rootkey.txt and retrieve your keys.

• If you input all the correct information, Files.com should be able to synchronize with your S3 Bucket.

Now you can use your remote server (Amazon S3) to add “remote sync” functions to your site’s folder. You can either push from Files.com to Amazon S3 or pull from S3 to Files.com. Additionally, you can configure two-way sync where new or updated files are pulled or pushed between Files.com and Amazon S3.

Configuring your Remote SFTP Server

To FTP/SFTP access to an Amazon S3 Files.com, you would also need to add and synchronize the FTP/SFTP server. Follow the similar steps as above: Go to Integrations > find SFTP > and enter the SFTP’s server information.

• If you input all the correct information, Files.com should synchronize with your SFTP server

Now, with both the SFTP server and the S3 bucket, you can easily synchronize files (pull or push) via Files.com.

How to get Files.com?

The integration with Amazon S3 and Files.com requires the Enterprise Connectivity Add-on, which is included on the Power or Premier plans. Files.com is a subscription-based service charged per month/user. The three plans are Starter, Power, and Premier. Files.com offers a full 7-day free trial.

C. Using FTP/SFTP clients that support Amazon S3

The last best way to FTP/SFTP to Amazon S3 is to use an FTP/SFTP client that supports Amazon S3. Some examples of these clients are FileZilla, WinSCP, Cyberduck. As long as the FTP client is also an S3 client, you will not have to configure anything on the server.

Using an open-source platform such as FileZilla

Use free, open-source, and cross-platform FTP server/client solutions such as FileZilla. The FileZilla client supports FTP, SFTP, and FTPS, while the server only supports FTP and FTPS. With the Pro edition, FileZilla also offers support for AWS S3 (among other cloud storage), allowing FTP to Amazon S3 without a problem.

When configuring the FileZilla client, you’ll be able to set the host and use S3 as a protocol

• Bear in mind that you are configuring a client and not a server. So, you’ll be able to access but not modify.
• FTP is not recommended for transferring outside public networks. If you intend to use an FTP server for S3, you’ll need to set it up inside a VPC (Virtual Private Cloud) or via a VPN for security reasons. Choose FTPS (for FileZilla) instead if you want to access files using the Internet.

Which method is best for you?

Why can’t you simply get FTP/SFTP access into an Amazon S3 bucket? Both technologies use different storage units, one uses files, and the other is based on objects. In addition, one is a file transfer method, and the other is a storage service.

Fortunately, Amazon S3 supports FTP, SFTP, and FTPS via their AWS Transfer Family service (but it is not cheap). In addition, third-party services and products can also provide seamless integration. For example, cloud-based services like Files.com or a file transfer client that supports S3, such as FileZilla, will allow you to interact with AWS S3 at a much better price and control.

FTP/SFTP access to an Amazon S3 Bucket FAQs