How to Get the User to Choose a File and Upload It in Aws S3

How to Upload And Download Files From AWS S3 Using Python (2022)

Acquire how to use deject resource in your Python scripts

Photo past Raj Steven from Pexels

I am writing this post out of sheer frustration.

Every post I've read on this topic assumed that I already had an business relationship in AWS, an S3 bucket, and a mound of stored data. They just show the code but kindly shadow over the well-nigh important part — making the code work through your AWS account.

Well, I could've figured out the code easily, thank you very much. I had to sift through many SO threads and the AWS docs to go rid of every nasty authentication error forth the way.

So that you lot won't experience the aforementioned and do the difficult piece of work, I will share all the technicalities of managing an S3 bucket programmatically, right from account creation to calculation permissions to your local machine to access your AWS resources.

Stride 1: Setup an business relationship

Correct, let's start with creating your AWS business relationship if you haven't already. Nothing unusual, merely follow the steps from this link:

GIF by the writer

Then, we will get to the AWS IAM (Identity and Access Management) console, where we will be doing about of the work.

GIF by the author

You lot tin hands switch betwixt different AWS servers, create users, add policies, and allow access to your user account from the panel. Nosotros will practise each ane by one.

Step 2: Create a user

For ane AWS account, you tin create multiple users, and each user can have various levels of admission to your account's resources. Let'south create a sample user for this tutorial:

GIF by the author

In the IAM console:

1. Go to the Users tab.
2. Click on Add users.
3. Enter a username in the field.
4. Tick the "Access key — Programmatic admission field" (essential).
5. Click "Side by side" and "Attach existing policies directly."
6. Tick the "AdministratorAccess" policy.
7. Click "Side by side" until you see the "Create user" button
8. Finally, download the given CSV file of your user'southward credentials.

It should await like this:

Past me🥱

Shop information technology somewhere rubber because we will be using the credentials later.

Step 3: Create a saucepan

Now, allow's create an S3 bucket where we can shop data.

GIF by the author

In the IAM panel:

1. Click services in the top left corner.
2. Scroll downward to storage and select S3 from the right-paw list.
3. Click "Create saucepan" and give information technology a name.

You tin choose whatever region yous want. Leave the rest of the settings and click "Create bucket" once more.

Step 4: Create a policy and add it to your user

In AWS, access is managed through policies. A policy can be a set of settings or a JSON file fastened to an AWS object (user, resources, group, roles), and it controls what aspects of the object you lot tin can use.

Beneath, we will create a policy that enables us to collaborate with our saucepan programmatically — i.eastward., through the CLI or in a script.

GIF by the writer

In the IAM console:

1. Become to the Policies tab and click "Create a policy."
2. Click the "JSON" tab and insert the code beneath:

replacing your-bucket-name with your own. If you pay attention, in the Action field of the JSON, nosotros are putting s3:* to allow any interaction to our saucepan. This is very broad, and then you may only allow specific actions. In that case, check out this page of the AWS docs to learn to limit admission.

This policy is only attached to the bucket, and we should connect it to the user equally well so that your API credentials piece of work correctly. Here are the instructions:

GIF past the writer

In the IAM console:

1. Become to the Users tab and click on the user we created in the last section.
2. Click the "Add permissions" push button.
3. Click the "Attach existing policies" tab.
4. Filter them by the policy we just created.
5. Tick the policy, review it and click "Add" the final time.

Step 5: Download AWS CLI and configure your user

We download the AWS command-line tool considering it makes authentication so much easier. Kindly go to this page and download the executable for your platform:

GIF past the author

Run the executable and reopen any active last sessions to permit the changes take effect. Then, type aws configure:

GIF past the author

Insert your AWS Cardinal ID and Secret Admission Fundamental, along with the region yous created your saucepan in (use the CSV file). You lot tin find the region name of your saucepan on the S3 folio of the console:

By me.

Simply click "Enter" when you lot reach the Default Output Format field in the configuration. At that place won't exist whatever output.

Pace half dozen: Upload your files

We are virtually there.

Now, we upload a sample dataset to our bucket so that nosotros can download it in a script after:

GIF past the author

Information technology should exist easy in one case you lot go to the S3 page and open your saucepan.

Step 7: Cheque if hallmark is working

Finally, pip install the Boto3 parcel and run this snippet:

If the output contains your bucket name(south), congratulations — you at present have full access to many AWS services through boto3, not just S3.

Using Python Boto3 to download files from the S3 bucket

With the Boto3 bundle, you lot have programmatic admission to many AWS services such as SQS, EC2, SES, and many aspects of the IAM console.

However, every bit a regular data scientist, yous will mostly need to upload and download data from an S3 bucket, so we will only cover those operations.

Let's start with the download. After importing the packet, create an S3 form using the customer function:

To download a file from an S3 bucket and immediately save it, nosotros tin can use the download_file function:

In that location won't be whatever output if the download is successful. You should pass the exact file path of the file to be downloaded to the Primal parameter. The Filename should incorporate the pass y'all desire to salve the file to.

Uploading is too very straightforward:

The function is upload_file and you just accept to change the club of the parameters from the download function.

Conclusion

I advise reading the Boto3 docs for more advanced examples of managing your AWS resource. It covers services other than S3 and contains code recipes for the about common tasks with each i.

Thanks for reading!

You tin can become a premium Medium member using the link beneath and become access to all of my stories and thousands of others:

Or only subscribe to my email listing:

Yous tin can reach out to me on LinkedIn or Twitter for a friendly chat about all things data. Or y'all tin can just read another story from me. How about these:

ostbyiver1960.blogspot.com

Source: https://towardsdatascience.com/how-to-upload-and-download-files-from-aws-s3-using-python-2022-4c9b787b15f2?source=post_internal_links---------6-------------------------------

Share this post