Importing Postgres Heroku from AWS S3

[Drone4four]

I’m learning how to handle Postgres instances by backing them up and restoring them on Heroku for a Django project (a small rudimentary CMS). The amount of data is a few hundred kilobytes because it's just text that I am storing in my my db.

I downloaded the binary data to my local machine using this particular section of the Heroku doc.

The next step was to create an AWS account, including setting up Access Keys which I located in the dashboard and entered them into my local dev environment. I named my bucket. I uploaded the binary to S3.

I’ve made it all the way to the end of Heroku’s import Postgres guide.

I install the `awscli` package with pip which enabled me to presign my s3 bucket (which succeeded).

I am right at the final step of importing my backup to Heroku Postgres. I am so close!

My traceback at this point indicates that Heroku is expecting an HTTP 200 (request has succeeded) but instead it receives an HTTP 400 (can’t process) ‘due to the source URL being inaccessible’. This points towards the restrictive permissions in place on my AWS S3 bucket.

You can find my traceback in full at the bottom of this post.

With regards to my AWS S3 bucket, in the dashboard, the main Permissions switch relevant here is the “Block all public access” option. Whether this checkbox is enabled or disabled (I carefully tried both), I encountered the same HTTP 200/400 in my traceback. This is where I believe the issue is.

I’m not sure what else to try. I’m also a little concerned that with the vast number of variables available for Amazon’s S3 service, I don’t know how I might share or export my configuration nicely for you people to take a closer look. What other information could I provide to better help you people help me?

$ heroku pg:backups:restore 'https://postgres-restore-<project-name>.s3.amazonaws.com/2021May<hash redacted>aa9376\?X-Amz-Algorithm\=AWS4-HMAC-SHA256\&X-Amz-Credential\=AKIAY<hash redacted>0527%2FOhio%2Fs3%2Faws4_request\&X-Amz-Date\=2021<hash redacted>33Z\&X-Amz-Expires\=3600\&X-Amz-SignedHeaders\=host\&X-Amz-Signature\=19<hash redacted>17' HEROKU_POSTGRESQL_PUCE -a <project-name>
› Warning: heroku update available from 7.50.0 to 7.51.0.
▸ WARNING: Destructive Action
▸ This command will affect the app <project-name>
▸ To proceed, type <project-name> or re-run this command with
▸ --confirm <project-name>

> <project-name>
Starting restore of https://postgres-restore-<project-name>.s3.amazonaws.com/<redacted>-802a-4409-8624-<redacted>\?X-Amz-Algorithm\=<redacted>\&X-Amz-Credential\=<redacted>%2F20210527%2FOhio%2Fs3%2Faw s4_request\&X-Amz-Date\=20210527T101733Z\&X-Amz-Expires\=3600\&X-Amz-SignedHeaders\=host\&X-Amz-Signature\=199<hash redacted>d86a0317 to postgresql-animate-93816... done

Use Ctrl-C at any time to stop monitoring progress; the backup will continue restoring.
Use heroku pg:backups to check progress.
Stop a running restore with heroku pg:backups:cancel.

Restoring... !
▸ An error occurred and the backup did not finish.
▸
▸ waiting for restore to complete
▸ pg_restore finished with errors
▸ waiting for download to complete
▸ download finished with errors
▸ please check the source URL and ensure it is publicly accessible
▸
▸ Run heroku pg:backups:info r019 for more details.

$ heroku pg:backups:info r019 -a <project-name>
› Warning: heroku update available from 7.50.0 to 7.51.0.
=== Backup r019
Database: BACKUP
Started at: 2021-05-27 10:29:02 +0000
Finished at: 2021-05-27 10:29:02 +0000
Status: Failed
Type: Manual
Backup Size: 0.00B (0% compression)

=== Backup Logs
2021-05-27 10:29:02 +0000 2021/05/27 10:29:02 aborting: could not write to output stream: Expected HTTP Status 200, received: "400 Bad Request"
2021-05-27 10:29:02 +0000 pg_restore: error: could not read from input file: end of file
2021-05-27 10:29:02 +0000 waiting for restore to complete
2021-05-27 10:29:02 +0000 pg_restore finished with errors
2021-05-27 10:29:02 +0000 waiting for download to complete
2021-05-27 10:29:02 +0000 download finished with errors
2021-05-27 10:29:02 +0000 please check the source URL and ensure it is publicly accessible
$