How to do a Pre-signed POST upload to AWS S3 in Go?

I would like to do a pre-signed POST to upload files to an AWS S3 bucket - how would this be done in Go?

Please note that this is not the same as Pre-signed upload with PUT.

So in order to help others I will answer the question myself and provide some code to help others who might have the same problem.

Example web app for Google App Engine rendering a pre-signed POST form can be found here.

And a small library I created doing the pre-signed POST in Go.

In short, doing a presigned POST to a public-read Amazon S3 bucket you need to:

1. Configure the S3 bucket to only allow public download.

Example bucket policy that allow only public read.

{
    "Version": "2012-10-17",
    "Id": "akjsdhakshfjlashdf",
    "Statement": [
	    {
		    "Sid": "kjahsdkajhsdkjasda",
		    "Effect": "Allow",
		    "Principal": {
			    "AWS": "*"
		    },
		    "Action": "s3:GetObject",
		    "Resource": "arn:aws:s3:::BUCKETNAMEHERE/*"
	    }
    ]
}

2. Create a policy for the HTTP POST that allows the upload.

AWS S3 docs

Example POST policy template with expiration to upload a specific key, into a specific bucket and allow public-read access.

{ "expiration": "%s",
	"conditions": [
		{"bucket": "%s"},
		["starts-with", "$key", "%s"],
		{"acl": "public-read"},

		{"x-amz-credential": "%s"},
		{"x-amz-algorithm": "AWS4-HMAC-SHA256"},
		{"x-amz-date": "%s" }
	]
}

3. Generate and sign the policy using the S3 bucket owner's credentials.

AWS docs

• Fill in the correct values for expiration, bucket, key, credentials and date.
• base64 encode the policy.
• HMAC-SHA256 the policy to get a signature.
• hex encode the signature.

4. Construct and POST the multipart form data

AWS S3 docs

Now either you would generate an HTML form and automatically get the correct multipart form data request like described in the above link.

I wanted to do this by hand in Go so here's how to do that.

Either way you need to provide all the parts that are specified in the POST policy you created in steps 2 and 3. You can also not have additional fields in the request except for the mandatory ones (not in the policy).

The order of the fields is also specified and all of them are multipart fields in the HTTP POST request.

func Upload(url string, fields Fields) error {
    var b bytes.Buffer
    w := multipart.NewWriter(&b)
    for _, f := range fields {
            fw, err := w.CreateFormField(f.Key)
            if err != nil {
                    return err
            }
            if _, err := fw.Write([]byte(f.Value)); err != nil {
                    return err
            }
    }
    w.Close()

    req, err := http.NewRequest("POST", url, &b)
    if err != nil {
            return err
    }
    req.Header.Set("Content-Type", w.FormDataContentType())

    client := &http.Client{}
    res, err := client.Do(req)
    if err != nil {
            return err
    }
    if res.StatusCode != http.StatusOK {
            err = fmt.Errorf("bad status: %s", res.Status)
    }
    return nil
}

Here is an alternative approach from https://github.com/minio/minio-go
that you might like for a full programmatic way of generating presigned post policy.

package main

import (
	"fmt"
	"log"
	"time"

	"github.com/minio/minio-go"
)

func main() {
    policy := minio.NewPostPolicy()
	policy.SetKey("myobject")
	policy.SetBucket("mybucket")
	policy.SetExpires(time.Now().UTC().AddDate(0, 0, 10)) // expires in 10 days
	config := minio.Config{
		AccessKeyID:     "YOUR-ACCESS-KEY-HERE",
		SecretAccessKey: "YOUR-PASSWORD-HERE",
		Endpoint:        "https://s3.amazonaws.com",
	}
	s3Client, err := minio.New(config)
	if err != nil {
		log.Fatalln(err)
	}
	m, err := s3Client.PresignedPostPolicy(policy)
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Printf("curl ")
	for k, v := range m {
		fmt.Printf("-F %s=%s ", k, v)
	}
	fmt.Printf("-F file=@/etc/bashrc ")
	fmt.Printf(config.Endpoint + "/mybucket\n")
}

Step 1:

    policy := minio.NewPostPolicy()
	policy.SetKey("myobject")
	policy.SetBucket("mybucket")
	policy.SetExpires(time.Now().UTC().AddDate(0, 0, 10)) // expires in 10 days

Instantiate a new policy structure, this policy structure implements following methods.

func NewPostPolicy() *PostPolicy
func (p *PostPolicy) SetBucket(bucket string) error
func (p *PostPolicy) SetContentLength(min, max int) error
func (p *PostPolicy) SetContentType(contentType string) error
func (p *PostPolicy) SetExpires(t time.Time) error
func (p *PostPolicy) SetKey(key string) error
func (p *PostPolicy) SetKeyStartsWith(keyStartsWith string) error
func (p PostPolicy) String() string

Step 2:

    m, err := s3Client.PresignedPostPolicy(policy)
	if err != nil {
		fmt.Println(err)
		return
	}

Now PresignedPostPolicy() takes the PostPolicy structure and returns back a map of "key/values" which can be used in your HTML form or curl command to upload data to s3.

At a glance it looks like POST works with an attached policy and signature -- designed for browser based uploads. See the AWS Docs for details.

Specifically, you need to generate a policy and sign that -- then include them in the HTML form, and thereby the POST request -- along with the rest of the required information. Or let the browser do it for you.

In the case of HTML form POST uploads you are only signing the policy string. The final URL to be posted to can vary based on the form contents: https://bucket.s3.amazonaws.com/<depends-on-form-content>. So you can't presign that URL because you don't know what it is.

This is different than a signed URL to which you PUT a file. You can sign that because you know the full URL: https://bucket.s3.amazonaws.com/known-key

You could build a POST request with the appropriate policy and parameters and upload via POST that way. However, you would need to know the contents of the form to know the URL beforehand. In which case you may as well use a presigned PUT URL.

At least that is how it appears at a glance...

Came across this problem and faced the "InvalidAccessKeyId" error when attempting to use the solution provided by @murrekatt.

Later I found out this issue was because I was generating the presigned POST inside a lambda and not including x-amz-security-token in the form data & policy.

So here's what I wrote with the help from @murrekatt and the boto3 library:

import (
	"crypto/hmac"
	"crypto/sha256"
	"encoding/base64"
	"encoding/hex"
	"encoding/json"
	"fmt"
	"time"

	"github.com/aws/aws-sdk-go-v2/aws"
)

type PresignedPOST struct {
	URL           string `json:"url"`
	Key           string `json:"key"`
	Policy        string `json:"policy"`
	Credential    string `json:"credential"`
	SecurityToken string `json:"securityToken,omitempty"`
	Signature     string `json:"signature"`
	Date          string `json:"date"`
}

func NewPresignedPost(input *NewPresignedPostInput) (*PresignedPOST, error) {

	// expiration time
	expirationTime := time.Now().Add(time.Second * time.Duration(input.ExpiresIn)).UTC()
	dateString := expirationTime.Format("20060102")

	// credentials string
	creds := fmt.Sprintf("%s/%s/%s/s3/aws4_request", input.Credentials.AccessKeyID, dateString, input.Region)

	// policy
	policyDoc, err := createPolicyDocument(expirationTime, input.Bucket, input.Key, creds, &input.Credentials.SessionToken, input.Conditions)
	if err != nil {
		return nil, err
	}

	// create signature
	signature := createSignature(input.Credentials.SecretAccessKey, input.Region, dateString, policyDoc)

	// url
	url := fmt.Sprintf("https://%s.s3.amazonaws.com/", input.Bucket)

	// expiration time
	dateTimeString := expirationTime.Format("20060102T150405Z")

	// post
	post := &PresignedPOST{
		Key:           input.Key,
		Policy:        policyDoc,
		Signature:     signature,
		URL:           url,
		Credential:    creds,
		SecurityToken: input.Credentials.SessionToken,
		Date:          dateTimeString,
	}

	return post, nil
}

type NewPresignedPostInput struct {
	// Key name
	Key string

	// Creds
	Credentials aws.Credentials

	// Region
	Region string

	// The name of the bucket to presign the post to
	Bucket string

	// Expiration -  The number of seconds the presigned post is valid for.
	ExpiresIn int64

	// A list of conditions to include in the policy. Each element can be either a list or a structure.
	// For example:
	// [
	// 		{"acl": "public-read"}, ["content-length-range", 2, 5], ["starts-with", "$success_action_redirect", ""]
	// ]
	Conditions []interface{}
}

// helpers
func createPolicyDocument(expirationTime time.Time, bucket string, key string, credentialString string, securityToken *string, extraConditions []interface{}) (string, error) {

	doc := map[string]interface{}{}
	doc["expiration"] = expirationTime.Format("2006-01-02T15:04:05.000Z")

	// conditions
	conditions := []interface{}{}
	conditions = append(conditions, map[string]string{
		"bucket": bucket,
	})

	conditions = append(conditions, []string{
		"starts-with", "$key", key,
	})

	conditions = append(conditions, map[string]string{
		"x-amz-credential": credentialString,
	})

	if securityToken != nil {
		conditions = append(conditions, map[string]string{
			"x-amz-security-token": *securityToken,
		})
	}

	conditions = append(conditions, map[string]string{
		"x-amz-algorithm": "AWS4-HMAC-SHA256",
	})

	conditions = append(conditions, map[string]string{
		"x-amz-date": expirationTime.Format("20060102T150405Z"),
	})

	// other conditions
	conditions = append(conditions, extraConditions...)

	doc["conditions"] = conditions

	// base64 encoded json string
	jsonBytes, err := json.Marshal(doc)
	if err != nil {
		return "", err
	}

	return base64.StdEncoding.EncodeToString(jsonBytes), nil
}

func createSignature(secretKey string, region string, dateString string, stringToSign string) string {

	// Helper to make the HMAC-SHA256.
	makeHmac := func(key []byte, data []byte) []byte {
		hash := hmac.New(sha256.New, key)
		hash.Write(data)
		return hash.Sum(nil)
	}

	h1 := makeHmac([]byte("AWS4"+secretKey), []byte(dateString))
	h2 := makeHmac(h1, []byte(region))
	h3 := makeHmac(h2, []byte("s3"))
	h4 := makeHmac(h3, []byte("aws4_request"))
	signature := makeHmac(h4, []byte(stringToSign))
	return hex.EncodeToString(signature)
}

Usage

// credentials
conf, _ := config.LoadDefaultConfig(c.Context)
awsCreds, _ := conf.Credentials.Retrieve(c.Context)

// generate presigned post
post, err := s3util.NewPresignedPost(&s3util.NewPresignedPostInput{
	Key:         <file-name>,
	Credentials: awsCreds,
	Region:      <region>,
	Bucket:      <bucket-name>,
	ExpiresIn:   <expiration>,
	Conditions: []interface{}{
		[]interface{}{"content-length-range", 1, <size-limit>},
	},
})

Then on the frontend, use the returned json in a POST form data

key:					<key>
X-Amz-Credential:		<credential>
X-Amz-Security-Token:	<securityToken> 	// if provided
X-Amz-Algorithm:		AWS4-HMAC-SHA256
X-Amz-Date:				<date>
Policy:					<policy>
X-Amz-Signature:		<signature>
file: 					<file>