Download S3 Folder Aws Cli

6 min read Oct 05, 2024

Downloading an Entire S3 Folder Using AWS CLI: A Step-by-Step Guide

The AWS Command Line Interface (AWS CLI) is a powerful tool for interacting with Amazon Web Services (AWS) from your terminal. One of the many tasks you can accomplish with the AWS CLI is downloading an entire folder from Amazon S3.

This guide provides a step-by-step walkthrough on how to download an entire S3 folder using the AWS CLI.

Prerequisites

Before we begin, make sure you have the following:

AWS Account: You need an AWS account to access and interact with AWS services like S3.
AWS CLI Installed: Install the AWS CLI on your local machine. You can download it from the official AWS website.
Configured AWS Credentials: Your AWS CLI should be configured with your AWS access key ID and secret access key. This allows the CLI to authenticate with your AWS account. You can configure your credentials using the aws configure command.

Downloading an S3 Folder

The AWS CLI provides the aws s3 sync command to synchronize data between your local machine and an S3 bucket. To download an entire S3 folder, you can use this command with the --delete flag.

Here's how to download an S3 folder using the AWS CLI:

Specify the S3 path: Determine the exact path of the folder you want to download from your S3 bucket. This path includes the bucket name, the folder name, and any subfolders within the folder. For example: s3://my-bucket/my-folder/.
Specify the local path: Choose a local directory where you want to download the S3 folder. For example, you can use the Downloads folder on your machine: /Users/your-username/Downloads/.
Execute the command: Run the following command in your terminal, replacing the placeholders with your specific S3 and local paths.
```
aws s3 sync s3://my-bucket/my-folder/ /Users/your-username/Downloads/ --delete
```

Important Notes:

The --delete flag ensures that any files or folders present in the local directory that don't exist in the S3 folder are deleted.
The command will download all objects within the specified folder, including any subfolders.
You can use wildcards in your S3 path to download specific files or folders within the folder. For example: s3://my-bucket/my-folder/*.

Example: Downloading an S3 Folder with Subfolders

Let's say you want to download the folder my-folder located in the S3 bucket my-bucket. This folder contains subfolders with images:

my-bucket/
    my-folder/
        images/
            image1.jpg
            image2.png
        videos/
            video1.mp4

You can use the following command to download the entire folder:

aws s3 sync s3://my-bucket/my-folder/ /Users/your-username/Downloads/ --delete

This command will download all the images, videos, and subfolders within my-folder to the /Users/your-username/Downloads/ directory on your local machine.

Alternative Methods: Downloading Individual Files

If you only need to download specific files from an S3 folder, you can use the aws s3 cp command. This command allows you to copy a single object from S3 to your local machine.

For example, to download the image1.jpg file from the my-folder folder in the my-bucket bucket:

aws s3 cp s3://my-bucket/my-folder/images/image1.jpg /Users/your-username/Downloads/

Conclusion

Downloading an entire S3 folder using the AWS CLI is a simple and efficient way to manage your S3 data. The aws s3 sync command, combined with the --delete flag, allows you to synchronize your local directory with your S3 folder, ensuring that you have a complete copy of the folder's contents. Remember to specify the correct S3 path and local directory before running the command.