Using the AWS CLI (S3)

With the AWS CLI you can easily interact with your Object Storage. Here you will find practical examples of how to use the CLI for your buckets.

Prerequisites

You have a User with an Access Key and Secret Key.
The AWS CLI is installed on your system. If not, download it from the AWS CLI documentation.

Configuration of the AWS CLI

1. Configure credentials

Configure your credentials:

aws configure set aws_access_key_id YOUR_ACCESS_KEY
aws configure set aws_secret_access_key YOUR_SECRET_KEY

2. Configure endpoint and additional options

Set the following options:

aws configure set default.endpoint_url https://storage.netways.cloud
aws configure set default.region de
aws configure set default.response_checksum_validation when_required
aws configure set default.request_checksum_calculation when_required
aws configure set default.s3.addressing_style virtual

Basic Examples

1. List Buckets

Show all buckets in your account:

aws s3 ls

2. Upload a File

Upload a local file to your bucket:

aws s3 cp ~/my-file.txt s3://my-bucket/

3. Show Bucket Contents

Show all files in a bucket:

aws s3 ls s3://my-bucket/

4. Sync a Directory

Synchronize a local directory with your bucket:

aws s3 sync ~/my-directory s3://my-bucket/backup

5. Download a File

Download a file from the bucket:

aws s3 cp s3://my-bucket/my-file.txt ~/Downloads/

6. Delete Files

Delete a file from the bucket:

aws s3 rm s3://my-bucket/my-file.txt

7. Show File Statistics

Show details for a file:

aws s3api head-object --bucket my-bucket --key my-file.txt

8. Create a Public Link

Create a temporary download link (valid for 3600 seconds by default):

aws s3 presign s3://my-bucket/my-file.txt --expires-in 86400

9. Create a Bucket

aws s3 mb s3://my-bucket2

Basic Commands for Bucket Policies

To manage public (anonymous) access you need to use bucket policies:

1. Show current policy

aws s3api get-bucket-policy --bucket my-bucket

2. Set a policy

aws s3api put-bucket-policy --bucket my-bucket --policy file://policy.json

3. Delete a policy

aws s3api delete-bucket-policy --bucket my-bucket

Examples of Common Policy Scenarios

Policy Types

AWS S3 supports complex policies with various permissions:

s3:GetObject (read only)
s3:PutObject (write only)
s3:* (full access)

1. Allow public read access

Create a policy.json file:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::my-bucket/*"]
    }
  ]
}

Apply it:

aws s3api put-bucket-policy --bucket my-bucket --policy file://policy.json

2. Allow public write access

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["s3:PutObject"],
      "Resource": ["arn:aws:s3:::my-bucket/uploads/*"]
    }
  ]
}

3. Allow full public access

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["s3:*"],
      "Resource": ["arn:aws:s3:::my-bucket/*"]
    }
  ]
}

4. Set policy for a specific prefix

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::my-bucket/public/*"]
    }
  ]
}

5. Read access for specific IP ranges

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::my-bucket/*"],
      "Condition": {
        "IpAddress": {"aws:SourceIp": ["192.0.2.0/24"]}
      }
    }
  ]
}

6. Time‑limited access

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::my-bucket/public/*"],
      "Condition": {
        "DateLessThan": {"aws:CurrentTime": "2023-12-31T23:59:59Z"}
      }
    }
  ]
}

Tips & Hints

Security Tips

Public write permissions should only be used in exceptional cases
Regularly review policies for necessity
Combine policies with bucket quotas to limit unwanted usage
Always validate JSON policies
Test new policies first in non‑production environments

Tip

Incremental policy changes:

aws s3api get-bucket-policy --bucket my-bucket --output text > current.json
# Edit the file
aws s3api put-bucket-policy --bucket my-bucket --policy file://current.json

Debugging

Add the --debug flag to your commands to get detailed information if something does not work as expected.

Bucket Versioning

Enable versioning

# Enable versioning
aws s3api put-bucket-versioning --bucket my-bucket --versioning-configuration Status=Enabled

# Check status
aws s3api get-bucket-versioning --bucket my-bucket

Working with versions

# List all versions of an object
aws s3api list-object-versions --bucket my-bucket --prefix path/file.txt

# Download a specific version
aws s3api get-object --bucket my-bucket --key path/file.txt --version-id VERSION_ID ~/target/file.txt

# Delete a version
aws s3api delete-object --bucket my-bucket --key path/file.txt --version-id VERSION_ID

Disable versioning

aws s3api put-bucket-versioning --bucket my-bucket --versioning-configuration Status=Suspended

Object Lifecycle Management

Create lifecycle rules

1. Create a JSON configuration file (`lifecycle.json`):

{
  "Rules": [
    {
      "ID": "AutoDeleteTempFiles",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "temp/"
      },
      "Expiration": {
        "Days": 7
      }
    },
    {
      "ID": "DeleteOldLogs",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "logs/"
      },
      "Expiration": {
        "Days": 30
      }
    }
  ]
}

2. Apply the rule:

aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://lifecycle.json

Attention

From AWS CLI version 2.23.0 there is an incompatibility with our Object Storage. Error: Missing required header for this request: Content-MD5. Work around by using an older AWS CLI version (≤ 2.22.35).

Manage lifecycle rules

# Show existing rules
aws s3api get-bucket-lifecycle-configuration --bucket my-bucket

# Delete a rule
aws s3api delete-bucket-lifecycle --bucket my-bucket

Alternative: Delete non‑current versions

{
  "Rules": [
    {
      "ID": "ExpireNonCurrentVersions",
      "Status": "Enabled",
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 90
      }
    }
  ]
}

Advanced expiration options

{
  "Rules": [
    {
      "ID": "ExpireAtSpecificDate",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "project-archive/"
      },
      "Expiration": {
        "Date": "2024-12-31T00:00:00Z"
      }
    }
  ]
}

Object Locking

Create a bucket with Object Lock

aws s3api create-bucket --bucket locked-bucket --object-lock-enabled-for-bucket

Retention settings

# Set retention for an object
aws s3api put-object-retention --bucket locked-bucket --key path/file.txt --retention '{ "Mode": "GOVERNANCE", "RetainUntilDate": "2025-01-01T00:00:00" }'

# Get retention info
aws s3api get-object-retention --bucket locked-bucket --key path/file.txt

Legal Hold

# Enable legal hold
aws s3api put-object-legal-hold --bucket locked-bucket --key path/file.txt --legal-hold Status=ON

# Check status
aws s3api get-object-legal-hold --bucket locked-bucket --key path/file.txt

Notes

Object Lock

Once enabled, Object Lock cannot be disabled
Retention periods can only be extended, not shortened

Lifecycle Management

Rules are typically executed within 24 h
Always test new rules with non‑critical data

Best Practices

Combine versioning with lifecycle rules for automatic cleanup
Use Object Lock for compliance‑critical data
Document all lifecycle rules within the team