amazon web services – How can I use wildcards to `cp` a group of files with the AWS CLI

amazon web services – How can I use wildcards to `cp` a group of files with the AWS CLI

To download multiple files from an aws bucket to your current directory, you can use recursive, exclude, and include flags.
The order of the parameters matters.

Example command:

aws s3 cp s3://data/ . --recursive --exclude * --include 2016-08*

For more info on how to use these filters: http://docs.aws.amazon.com/cli/latest/reference/s3/#use-of-exclude-and-include-filters

The Order of the Parameters Matters

The exclude and include should be used in a specific order, We have to first exclude and then include. The viceversa of it will not be successful.

aws s3 cp s3://data/ . --recursive  --include 2016-08* --exclude * 

This will fail because order of the parameters maters in this case. The include is excluded by the *

aws s3 cp s3://data/ . --recursive --exclude * --include 2016-08*`

This one will work because the we excluded everything but later we had included the specific directory.

amazon web services – How can I use wildcards to `cp` a group of files with the AWS CLI

Okay I have to say the example is wrong and should be corrected as follows:

aws s3 cp . s3://data/ --recursive --exclude * --include 2006-08* --exclude */*

The . needs to be right after the cp. The final --exclude is to make sure that nothing is picked up from any subdirectories that are picked up by the --recursive (learned that one by mistake…)

This will work for anyone struggling with this by the time they got here.

Leave a Reply

Your email address will not be published.