AWS S3 Lifecyclerules

I've been trying for a few days to set up versioning on AWS S3.
Setting up the versioning itself was simple, but I would like to avoid versioning my objects forever.
I saw that this required the use of LifecycleHooks. Except that they only run once a day, and I can't run them manually, which makes iterations slow.

What I would like to do is:

• Version all objects, and keep the versions, including potential deletion markers, for 15 days.
• Except the objects which starts by health-* that I don't want to version (if not possible, only keep versions during 1 day), and I don't want to keep deletion markers either.

Moreover, my objects are stored with a path per client (/client/app/myFile, /client/app/health-xxx). I don't know if it has an impact on the rules prefix?

I tried by myself, with the help of this documentation, but I experiment difficulties to translate my high level business need to concrete rules. And like I said, since I can iterate only once a day, it's quite slow ^^'

First things first, you can't have versioning enabled for some objects and exclude other. Either you have it, or you don't.

> When you enable versioning in a bucket, all new objects are versioned and given a unique version ID. Objects that already existed in the bucket at the time versioning was enabled will thereafter always be versioned

Also, keep in mind that:

> Objects that are stored in your bucket before you set the versioning state have a version ID of null. When you enable versioning, existing objects in your bucket do not change.

Having said that, you might actually need more than one rule, but I'm not sure, as your use case is not clear.

Anyhow, when working with more than one rule per bucket, remember that:

> When you have multiple rules in an S3 Lifecycle configuration, an object can become eligible for multiple S3 Lifecycle actions. In such cases, Amazon S3 follows these general rules:

• Permanent deletion takes precedence over transition.

• Transition takes precedence over creation of delete markers.

So, summing it up, you might use something like this:

{
  "Rules": [
    {
      "ID": "Delete-Objects-After-15-Days",
      "Filter": {
        "And": {
          "Prefix": "",
          "Tags": [],
          "Not": {
            "Prefix": "PATH/TO/YOUR/OBJECTS/WITH/PREFIX/health-"
          }
        }
      },
      "Status": "Enabled",
      "Transitions": [],
      "NoncurrentVersionTransitions": [],
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 15
      },
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 0
      },
      "NoncurrentVersionDeletions": {
        "NoncurrentDays": 15
      }
    }
  ]
}

Apply it with:

aws s3api put-bucket-lifecycle-configuration  \
--bucket bucketname  \
--lifecycle-configuration file://lifecycle.json

As for since I can iterate only once a day, it's quite slow, well, you can't speed it up on the S3 end, so you might consider creating a Lambda function that basically deletes the objects you don't need.

You can run the function as often as you want / need.

Thanks @Budaker for the quick response, which aswer a some questions I had, and gives useful informations!

> Having said that, you might actually need more than one rule, but I'm not sure, > as your use case is not clear.

Let me reformulate (and updating it accordingly to the fact I can't avoid versionning for the prefixed object /client/app/health- ).

General Case :

• Keep the current version forever
• Keep all non current version for 15 days. Afterward, delete it.
• Keep all delete marker for 15 days. Afterward, delete it

Exception Case : For any files prefixed by /client/app/health-

• Keep the current version for 1 day. Afterward, delete it.
• Keep all non current version for 1 days. Afterward, delete it.
• Keep all delete marker for 1 days. Afterward, delete it

Considering your example and explaination, you gave me a sample for the general case. I don't see how I could cover the exception case tho. But you gave me enough material to be able to find a solution on my own

I will edit this message with the full solution in a few days - After testing it to be sure it works.

Edit : Here is the rules. Thanks again @Budaker!

{
    "Rules": [
        {
            "Expiration": {
                "ExpiredObjectDeleteMarker": true
            },
            "ID": "Delete-All-NonCurrent-Version-After-15-Days",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "NoncurrentVersionExpiration": {
                "NoncurrentDays": 15
            },
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 1
            }
        },
        {
            "Expiration": {
                "Days": 1
            },
            "ID": "Delete-client1-Health-Object-And-Version-After-1-Day",
            "Filter": {
                "Prefix": "PATH/TO/FILE/health-"
            },
            "Status": "Enabled",
            "NoncurrentVersionExpiration": {
                "NoncurrentDays": 1
            },
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 1
            }
        }
    ]
}