How to serialize `LastEvaluatedKey` from DynamoDB's Golang SDK?

When working with DynamoDB in Golang, if a call to query has more results, it will set LastEvaluatedKey on the QueryOutput, which you can then pass in to your next call to query as ExclusiveStartKey to pick up where you left off.

This works great when the values stay in Golang. However, I am writing a paginated API endpoint, so I would like to serialize this key so I can hand it back to the client as a pagination token. Something like this, where something is the magic package that does what I want:

type GetDomainObjectsResponse struct {
  Items     []MyDomainObject `json:"items"`
  NextToken string           `json:"next_token"`
}
  
func GetDomainObjects(w http.ResponseWriter, req *http.Request) {
  // ... parse query params, set up dynamoIn ...

  dynamoIn.ExclusiveStartKey = something.Decode(params.NextToken)

  dynamoOut, _ := db.Query(dynamoIn)

  response := GetDomainObjectsResponse{}
  dynamodbattribute.UnmarshalListOfMaps(dynamoOut.Items, &response.Items)

  response.NextToken := something.Encode(dynamoOut.LastEvaluatedKey)
  
  // ... marshal and write the response ...
}

(please forgive any typos in the above, it's a toy version of the code I whipped up quickly to isolate the issue)

Because I'll need to support several endpoints with different search patterns, I would love a way to generate pagination tokens that doesn't depend on the specific search key.

The trouble is, I haven't found a clean and generic way to serialize the LastEvaluatedKey. You can marshal it directly to JSON (and then e.g. base64 encode it to get a token), but doing so is not reversible. LastEvaluatedKey is a map[string]types.AttributeValue, and types.AttributeValue is an interface, so while the json encoder can read it, it can't write it.

For example, the following code panics with panic: json: cannot unmarshal object into Go value of type types.AttributeValue.

lastEvaluatedKey := map[string]types.AttributeValue{
	"year":  &types.AttributeValueMemberN{Value: "1993"},
	"title": &types.AttributeValueMemberS{Value: "Benny & Joon"},
}

bytes, err := json.Marshal(lastEvaluatedKey)
if err != nil {
  panic(err)
}

decoded := map[string]types.AttributeValue{}
err = json.Unmarshal(bytes, &decoded)
if err != nil {
  panic(err)
}

What I would love would be a way to use the DynamoDB-flavored JSON directly, like what you get when you run aws dynamodb query on the CLI. Unfortunately the golang SDK doesn't support this.

I suppose I could write my own serializer / deserializer for the AttributeValue types, but that's more effort than this project deserves.

Has anyone found a generic way to do this?

OK, I figured something out.

type GetDomainObjectsResponse struct {
  Items     []MyDomainObject `json:"items"`
  NextToken string           `json:"next_token"`
}
  
func GetDomainObjects(w http.ResponseWriter, req *http.Request) {
  // ... parse query params, set up dynamoIn ...

  eskMap := map[string]string{}
  json.Unmarshal(params.NextToken, &eskMap)
  esk, _ = dynamodbattribute.MarshalMap(eskMap)
  dynamoIn.ExclusiveStartKey = esk

  dynamoOut, _ := db.Query(dynamoIn)

  response := GetDomainObjectsResponse{}
  dynamodbattribute.UnmarshalListOfMaps(dynamoOut.Items, &response.Items)

  lek := map[string]string{}
  dynamodbattribute.UnmarshalMap(dynamoOut.LastEvaluatedKey, &lek)
  response.NextToken := json.Marshal(lek)
  
  // ... marshal and write the response ...
}

(again this is my real solution hastily transferred back to the toy problem, so please forgive any typos)

As @buraksurdar pointed out, attributevalue.Unmarshal takes an inteface{}. Turns out in addition to a concrete type, you can pass in a map[string]string, and it just works.

I believe this will NOT work if the AttributeValue is not flat, so this isn't a general solution [citation needed]. But my understanding is the LastEvaluatedKey returned from a call to Query will always be flat, so it works for this usecase.

Inspired by Dan, here is a solution to serialize and deserialize to/from base64

package dynamodb_helpers

import (
	"encoding/base64"
	"encoding/json"

	"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
	"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)

func Serialize(input map[string]types.AttributeValue) (*string, error) {
	var inputMap map[string]interface{}
	err := attributevalue.UnmarshalMap(input, &inputMap)
	if err != nil {
		return nil, err
	}
	bytesJSON, err := json.Marshal(inputMap)
	if err != nil {
		return nil, err
	}
	output := base64.StdEncoding.EncodeToString(bytesJSON)
	return &output, nil
}

func Deserialize(input string) (map[string]types.AttributeValue, error) {
	bytesJSON, err := base64.StdEncoding.DecodeString(input)
	if err != nil {
		return nil, err
	}
	outputJSON := map[string]interface{}{}
	err = json.Unmarshal(bytesJSON, &outputJSON)
	if err != nil {
		return nil, err
	}

	return attributevalue.MarshalMap(outputJSON)
}

Inspired by Dan and ThomasP1988 I would like to give a quick solution by merging above inputs. Heres how you can implement pagination in your API using LastEvaluatedKey and ExclusiveStartKey in GoLang and DynamoDb

// getting lsk map from base64 string
var esk map[string]*dynamodb.AttributeValue
if lastEvaluatedKey != "" {
bytesJSON, err := base64.StdEncoding.DecodeString(lastEvaluatedKey)
if err != nil {
return fmt.Errorf("failed to decode lek : %w", err)
}
var outputJSON map[string]interface{}
err = json.Unmarshal(bytesJSON, &outputJSON)
if err != nil {
return fmt.Errorf("failed to unmarshal lek : %w", err)
}
esk, err = dynamodbattribute.MarshalMap(outputJSON)
if err != nil {
return fmt.Errorf("failed to marshal lek : %w", err)
}
}
// you need to pass this in your Query input with limit
result, err := s.db.Query(
&dynamodb.QueryInput{
TableName:                 aws.String("TableName"),
IndexName:                 aws.String("IndexName"),
ExpressionAttributeNames:  expr.Names(),
ExpressionAttributeValues: expr.Values(),
KeyConditionExpression:    expr.KeyCondition(),
ScanIndexForward:          aws.Bool(false),
ExclusiveStartKey:         esk,
Limit:                     aws.Int64(10),
},
)
// returning LastEvaluatedKey as base 64 encoded string
lekOutPut := ""
if result.LastEvaluatedKey != nil {
lek := map[string]interface{}{}
dynamodbattribute.UnmarshalMap(result.LastEvaluatedKey, &lek) //put last evaluated key in a map
lastKey, err := json.Marshal(lek)                             // convert to byte array
if err != nil {
return fmt.Errorf("failed to marshal lek : %w", err)
}
lekOutPut = base64.StdEncoding.EncodeToString(lastKey) //convert to string to return as a param
}

return lekOutPut in response to client so that client can send it back to you in another API req