Implementing idempotency keys

I'm trying to get my two Golang GRPC endpoints to support idempotency keys. My service will store and read keys from Mongo (because I'm already using it for other data) as a unique index in its own Collection.

I'm thinking of two solutions but each has their weaknesses. I know there's more complex stuff like saving request and response and making the logic ACID. However for my first endpoint, the only-once logic (the endpoint's code which needs to be idempotent) calls a service that sends an email, so it can't be rollbacked. My second endpoint does multiple Inserts in Mongo, which seems can be rollbacked but I'm not sure how and if there's another solution that'd also solve for the first endpoint.

Solution 1

func MyEndpoint(request Request) (Response, error) {
  doesExist, err := doesIdemKeyExist(request.IdemKey)
  if err != nil {
    return nil, status.Error(codes.Internal, "Failed to check idem key.")
  }
  if doesExist {
    return Response{}, nil
  }
  
  // < only-once logic >

  err := insertIdemKey(request.IdemKey)
  if err != nil {
    if mongo.IsDuplicateKeyError(err) {
      return Response{}, nil
    }
    return nil, status.Error(codes.Internal, "Failed to insert idem key.")
  }
 
  return Response{}, nil
}

The weakness here is that client could send first request to my endpoint and lose connection, then retry with second request. First request could process but not reach insertIdemKey, so second request would process too, violating idempotency.

Solution 2

func MyEndpoint(request Request) (Response, error) {
  err := insertIdemKey(request.IdemKey)
  if err != nil {
    if mongo.IsDuplicateKeyError(err) {
      return Response{}, nil
    }
    return nil, status.Error(codes.Internal, "Failed to insert idem key.")
  }

  // < only-once logic >

  return Response{}, nil
}

The weakness here is that only-once logic could have intermittent failures, such as from dependencies. Affected requests that are retried will be ignored.

What's the best solution here? Should I just compromise and go with one of these imperfect solutions?

You should use a document with a state property in MongoDB, with possible values processing and done.

When a request comes in, try to insert the document into the database with the given idemKey and state=processing. If that fails because the key already exists, then either report success (if state is done) or that it is still being processed (if state is processing). Or wait for it to complete and then report success.

If inserting the document succeeds, proceed with executing "only-once logic".

Once the "only-once logic" is done, update the document's state to state=done. If executing the logic fails, you may delete the document from the database so a subsequent request can try to execute it again.

To protect against a server failure during executing the logic or against a failure to delete the document, you should record the start / creation timestamp too, and define an expiration. Let's say when a new request comes in and the document exists with processing sate but the document is older than 30 seconds, you could assume it will never be completed and proceed as if the document didn't exist in the database in the first place: set its creation timestamp to the current time and execute the logic, then update the state to done if logic execution succeeds. MongoDB also supports auto-removal of expired documents, but note that the removal is not timed precisely.

Note that this solution isn't perfect either: if executing the logic succeeds but you can't update the document's state to done afterwards, after expiration you could end up repeating the execution. What you want is the atomic / transactional execution of your logic and a MongoDB operation, which is not possible.

If your "only-once logic" contains multiple inserts, you could use insertOrUpdate() to not duplicate the records if the execution fails and you have to repeat it, or you could insert the documents with idemKey included, so you could identify which documents were previously inserted (and you could remove them first thing, or skip them and just insert the rest).
Also note that starting with MongoDB 5.0, transactions are supported, so you can execute multiple inserts in a single transaction.

See related question: https://stackoverflow.com/questions/50488013/how-to-synchronize-two-apps-running-in-two-different-servers-via-mongodb/50488423#50488423