Skip to content

AWS S3 Adapter

The file_keeper:s3 adapter allows you to use Amazon S3 for storing and retrieving files. This adapter leverages the boto3 Python library.

Overview

This adapter provides a convenient way to integrate AWS S3 with file-keeper. You'll need to have the boto3 library installed and configure it with the appropriate credentials for your AWS account.

pip install 'file-keeper[s3]'

## or

pip install boto3

Initialization Example

Here's an example of how to initialize the S3 adapter:

storage = make_storage("my_s3_storage", {
    "type": "file_keeper:s3",
    "key": "***",
    "secret": "***",
    "bucket": "file-keeper",
    "initialize": True,
})

storage = make_storage("my_s3_storage", {
    "type": "file_keeper:s3",
    "key": "minioadmin",
    "secret": "minioadmin",
    "bucket": "file-keeper",
    "endpoint": "http://127.0.0.1:9000/",
    "initialize": True,
})

Set environment variables mentioned in boto3 documentations. In this way you don't need to use hardcoded secrets.

export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export AWS_ENDPOINT_URL=http://127.0.0.1:9000

Then storage will automatially pick any missing options from envvars

storage = make_storage("my_s3_storage", {
    "type": "file_keeper:s3",
    "bucket": "file-keeper",
})

Just like environment variables, you can use configuration file. It will be located at ~/.aws/credentials or a different location specified by AWS_CONFIG_FILE.

Add default profile there

[default]
aws_access_key_id = minioadmin
aws_secret_access_key = minioadmin
endpoint_url = http://127.0.0.1:9000

Or, if you want to use different profile name, set AWS_PROFILE with the name of prefered credentials profile. And now you can initialize the storage without specifying parameters that are set by the profile:

storage = make_storage("my_s3_storage", {
    "type": "file_keeper:s3",
    "bucket": "file-keeper",
})

If you already have initialized boto3.client("s3"), you can just reuse it instead of specifying key and secret:

s3_client = boto3.client("s3", ...) # somehow you've initialized it

storage = make_storage("my_s3_storage", {
    "type": "file_keeper:s3",
    "bucket": "file-keeper",
    "client": s3_client
})

Important Notes:

  • Replace the placeholder values (bucket, region, access_key_id, secret_access_key) with your actual AWS account credentials and configuration.
  • Ensure that you have created a bucket in your AWS S3 account to store the files.
  • For enhanced security, consider using IAM roles instead of hardcoding access keys.
  • Refer to the AWS S3 documentation for more information about AWS S3.