Daily load of weekly records in elastic through logstash ignoring repeated records

I'm totally new to elastic and logstash, we are currently evaluating it, expecting to use it for some of our data needings, and I have to say that it's going quite well up to date.

Now I have a doubt and I cannot find a clear answer on the official documentation.

We have an url where we create on demand a json with environmental data for the last 7 days. Those data are updated hourly, so if I check it on monday at 6am it will contain data from monday at 5am to last tuesday at 6am, and if I check it on tuesday at 14pm it will contain data from tuesday at 13pm to last wednesday at 14pm.

I want to load those data on elastic periodically, let's say on a daily basis, but I don't want to repeat values that I've already loaded. I mean, the first time my logstash pipeline executes it will have to transfer data for the last seven days, but if it fires again a day later, it only should transfer records from the last day, ignoring the records of the 6 previous days that already are loaded on the index.

I've been able to load records from the url into elastic index, but I cannot see how to configure the pipeline to avoid those repeated data.

If anyone could point me in the right direction I will appreciate so much.

The best way would be to handle this on the side that serves that URL where you could give a date interval to retrieve instead of needlessly retrieving useless data every time.

But if this is not an option, you have different ways of achieving what you want in Logstash.

Also, you need the exact same thing as provided by the jdbc input plugin, i.e. something that records the last value of a certain field in a local file, unfortunately the http_poller input doesn't support that... yet... because an issue about this very need has been posted in... 2016!!

One way is to use the environment filter in order to load an environment variable that would contain the date of the last load for instance. After each run you need to make sure that the environment variable is updated so that its value can be picked up during the next call.

Another way is to use the memcached filter to store that value, but I think it's a bit far fetched to install and maintain memcached just for a single value.