Uploading files — Boto3 Docs 1.18.3 documentation

The AWS SDK for Python provides a pair of methods to upload a file to an S3 bucket.

The upload_file method accepts a file name, a bucket name, and an object name. The method handles large files by splitting them into smaller chunks and uploading each chunk in parallel.

import loggingimport boto3from botocore.exceptions import ClientErrordef upload_file(file_name, bucket, object_name=None): """Upload a file to an S3 bucket :param file_name: File to upload :param bucket: Bucket to upload to :param object_name: S3 object name. If not specified then file_name is used :return: True if file was uploaded, else False """ # If S3 object_name was not specified, use file_name if object_name is None: object_name = file_name # Upload the file s3_client = boto3.client('s3') try: response = s3_client.upload_file(file_name, bucket, object_name) except ClientError as e: logging.error(e) return False return True

The upload_fileobj method accepts a readable file-like object. The file object must be opened in binary mode, not text mode.

s3 = boto3.client('s3')with open("FILE_NAME", "rb") as f: s3.upload_fileobj(f, "BUCKET_NAME", "OBJECT_NAME")

The upload_file and upload_fileobj methods are provided by the S3Client, Bucket, and Object classes. The method functionality provided by each class is identical. No benefits are gained by calling one class's method over another's. Use whichever class is most convenient.

The Callback parameter

Both upload_file and upload_fileobj accept an optional Callbackparameter. The parameter references a class that the Python SDK invokes intermittently during the transfer operation.

Invoking a Python class executes the class's __call__ method. For each invocation, the class is passed the number of bytes transferred up to that point. This information can be used to implement a progress monitor.

The following Callback setting instructs the Python SDK to create an instance of the ProgressPercentage class. During the upload, the instance's __call__ method will be invoked intermittently.

s3.upload_file( 'FILE_NAME', 'BUCKET_NAME', 'OBJECT_NAME', Callback=ProgressPercentage('FILE_NAME'))

An example implementation of the ProcessPercentage class is shown below.

import osimport sysimport threadingclass ProgressPercentage(object): def __init__(self, filename): self._filename = filename self._size = float(os.path.getsize(filename)) self._seen_so_far = 0 self._lock = threading.Lock() def __call__(self, bytes_amount): # To simplify, assume this is hooked up to a single filename with self._lock: self._seen_so_far += bytes_amount percentage = (self._seen_so_far / self._size) * 100 sys.stdout.write( "\r%s%s / %s (%.2f%%)" % ( self._filename, self._seen_so_far, self._size, percentage)) sys.stdout.flush()