Skip to content

Using the Boto3 Library (S3)

With the Python library Boto3 you can easily interact with your Object Storage at storage.netways.cloud. Here you will find practical examples of how to use Boto3 with your S3 bucket.

Prerequisites

  • You have a User with an S3 key (Access Key and Secret Key).
  • The Boto3 library is installed on your system. If not, install it with: 
    pip install boto3

Configuring Boto3

1. Configure AWS credentials

Configure your credentials to connect Boto3 to storage.netways.cloud. You can store the credentials in ~/.aws/credentials:

[nws]
aws_access_key_id = ACCESS_KEY
aws_secret_access_key = SECRET_KEY

Alternatively, you can provide the credentials directly in code (not recommended for production environments):

import boto3

session = boto3.Session(
    aws_access_key_id='ACCESS_KEY',
    aws_secret_access_key='SECRET_KEY'
)
s3_client = session.client('s3', endpoint_url='https://storage.netways.cloud')

Replace ACCESS_KEY and SECRET_KEY with your S3 credentials.

Basic Examples

1. List buckets

Show all buckets in your project:

import boto3

s3_client = boto3.client('s3', endpoint_url='https://storage.netways.cloud')
response = s3_client.list_buckets()
for bucket in response['Buckets']:
    print(bucket['Name'])

2. Upload a file

Upload a local file to your bucket:

s3_client.upload_file('~/my-file.txt', 'my-bucket', 'my-file.txt')

3. List bucket contents

Show all objects in a bucket:

response = s3_client.list_objects_v2(Bucket='my-bucket')
for obj in response.get('Contents', []):
    print(obj['Key'])

4. Sync a directory

Synchronize a local directory with your bucket (example using os and upload_file):

import os

local_dir = '~/my-directory'
bucket_name = 'my-bucket'
prefix = 'backup/'

for root, _, files in os.walk(os.path.expanduser(local_dir)):
    for file in files:
        local_path = os.path.join(root, file)
        relative_path = os.path.relpath(local_path, os.path.expanduser(local_dir))
        s3_path = os.path.join(prefix, relative_path).replace('\\', '/')
        s3_client.upload_file(local_path, bucket_name, s3_path)

5. Download a file

Download a file from the bucket:

s3_client.download_file('my-bucket', 'my-file.txt', '~/Downloads/my-file.txt')

6. Delete files

Delete a file from the bucket:

s3_client.delete_object(Bucket='my-bucket', Key='my-file.txt')

7. Show file statistics

Display details about a file:

response = s3_client.head_object(Bucket='my-bucket', Key='my-file.txt')
print(response)

Create a temporary download link (valid for 24 hours):

url = s3_client.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-bucket', 'Key': 'my-file.txt'},
    ExpiresIn=24*3600
)
print(url)

9. Create a bucket

Create a new bucket:

s3_client.create_bucket(Bucket='my-bucket2')

Basic Commands for Bucket Policies

To manage public access (anonymous access) you can set bucket policies with Boto3.

1. Show current policy

try:
    response = s3_client.get_bucket_policy(Bucket='my-bucket')
    print(response['Policy'])
except s3_client.exceptions.ClientError as e:
    if e.response['Error']['Code'] == 'NoSuchBucketPolicy':
        print("No policy found")
    else:
        raise e

2. Set a policy

Set a policy for the bucket:

import json

policy = {
    "Version": "2012-10-17",
    "Statement": []
}
s3_client.put_bucket_policy(Bucket='my-bucket', Policy=json.dumps(policy))

3. Delete a policy

s3_client.delete_bucket_policy(Bucket='my-bucket')

Examples for Common Policy Scenarios

1. Allow public read access

policy = {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": ["s3:GetObject"],
            "Resource": ["arn:aws:s3:::my-bucket/*"]
        }
    ]
}
s3_client.put_bucket_policy(Bucket='my-bucket', Policy=json.dumps(policy))

2. Allow public write access

policy = {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": ["s3:PutObject"],
            "Resource": ["arn:aws:s3:::my-bucket/*"]
        }
    ]
}
s3_client.put_bucket_policy(Bucket='my-bucket', Policy=json.dumps(policy))

3. Allow full public access

policy = {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": ["s3:GetObject", "s3:PutObject"],
            "Resource": ["arn:aws:s3:::my-bucket/*"]
        }
    ]
}
s3_client.put_bucket_policy(Bucket='my-bucket', Policy=json.dumps(policy))

4. Set policy for a specific prefix

policy = {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": ["s3:GetObject"],
            "Resource": ["arn:aws:s3:::my-bucket/public/*"]
        }
    ]
}
s3_client.put_bucket_policy(Bucket='my-bucket', Policy=json.dumps(policy))

5. Policy for specific IP ranges

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::my-bucket/*"],
      "Condition": {
        "IpAddress": {"aws:SourceIp": ["192.0.2.0/24"]}
      }
    }
  ]
}

6. Policy for time‑limited access

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::my-bucket/public/*"],
      "Condition": {
        "DateLessThan": {"aws:CurrentTime": "2025-12-31T23:59:59Z"}
      }
    }
  ]
}

Tips & Notes

Security notes

  • Public write permissions (s3:PutObject) should only be used in exceptional cases.
  • Review policies regularly for necessity.
  • Combine policies with bucket quotas to limit unwanted usage.
  • Always validate JSON policies.
  • Test new policies first in non‑production environments.
Debugging

Enable debugging in Boto3 to get detailed information: 

import logging
logging.basicConfig(level=logging.DEBUG)

Bucket Versioning

Enabling versioning

Versioning allows storing multiple versions of an object:

s3_client.put_bucket_versioning(
    Bucket='my-bucket',
    VersioningConfiguration={'Status': 'Enabled'}
)

# Check status
response = s3_client.get_bucket_versioning(Bucket='my-bucket')
print(response.get('Status'))

Working with versions

# List all versions of an object
response = s3_client.list_object_versions(Bucket='my-bucket', Prefix='path/file.txt')
for version in response.get('Versions', []):
    print(version['VersionId'], version['Key'])

# Download a specific version
s3_client.download_file('my-bucket', 'path/file.txt', '~/target/file.txt', ExtraArgs={'VersionId': 'VERSION_ID'})

# Delete a version
s3_client.delete_object(Bucket='my-bucket', Key='path/file.txt', VersionId='VERSION_ID')

Disabling versioning

s3_client.put_bucket_versioning(
    Bucket='my-bucket',
    VersioningConfiguration={'Status': 'Suspended'}
)

Object Lifecycle Management

Create lifecycle rules

Configure automatic deletion of objects:

Code (not supported) 
lifecycle_config = {
    'Rules': [
        {
            'ID': 'AutoDeleteTempFiles',
            'Status': 'Enabled',
            'Filter': {'Prefix': 'temp/'},
            'Expiration': {'Days': 7}
        },
        {
            'ID': 'DeleteOldLogs',
            'Status': 'Enabled',
            'Filter': {'Prefix': 'logs/'},
            'Expiration': {'Days': 30}
        }
    ]
}
s3_client.put_bucket_lifecycle_configuration(
    Bucket='my-bucket',
    LifecycleConfiguration=lifecycle_config
)

Attention

In current Boto3 versions, setting lifecycle rules for third‑party S3 providers is no longer supported.
Calling put_bucket_lifecycle_configuration results in the following error: 

botocore.errorfactory.InvalidRequest: An error occurred (InvalidRequest) when calling the PutBucketLifecycleConfiguration operation: Missing required header for this request: Content-MD5
More information can be found here.

Object Locking

Create a bucket with Object Lock

s3_client.create_bucket(
    Bucket='locked-bucket',
    ObjectLockEnabledForBucket=True
)

Retention settings

# Set retention for an object
s3_client.put_object_retention(
    Bucket='locked-bucket',
    Key='path/file.txt',
    Retention={
        'Mode': 'GOVERNANCE',
        'RetainUntilDate': '2025-12-31T00:00:00Z'
    }
)

# Retrieve retention info
response = s3_client.get_object_retention(Bucket='locked-bucket', Key='path/file.txt')
print(response['Retention'])

# Enable legal hold
s3_client.put_object_legal_hold(
    Bucket='locked-bucket',
    Key='path/file.txt',
    LegalHold={'Status': 'ON'}
)

# Check status
response = s3_client.get_object_legal_hold(Bucket='locked-bucket', Key='path/file.txt')
print(response['LegalHold'])

Notes

Object Lock

  • Once enabled, Object Lock cannot be disabled.
  • Retention periods can only be extended, not shortened.

Lifecycle Management

  • Rules are typically executed within 24 h.
  • Always test new rules with non‑critical data.
Best Practices
  • Combine versioning with lifecycle rules for automatic cleanup.
  • Use Object Lock for compliance‑critical data.
  • Document all lifecycle rules within the team.