Serve most recent augmented diff from disk cache only

[mmd-osm]

Today, several clients are requesting current augmented diffs more or less at the same time via a call to /api/augmented_diff?id=xyz. This approach does not scale to a higher number of clients, as each query takes about 5-30 seconds, consumes quite some CPU cycles, but effectively returns exactly the same result.

I'm proposing a few changes to this process:

• Keep the most recent 60-120 augmented diffs precalculated on disk, possibly as compressed files. Maybe also for a longer period of time.
  Compared to the previous approach which kept everything on disk, this approach should keep the disk space requirements low
• Precalculate most current augmented diff right after update_from_disk completes
• Announce newly available diff via augmented_diff_status only after precalculation
• Serve most recent augmented diffs via disk cache only, avoiding the query execution altogether. Older augmented diffs (outside 60-120 buffer range) may still trigger query.

Pros:

• Much better scalability to a larger number of data consumers
• Less overall system load
• Less frustration for data consumers due to 429 error messages

Cons:

• None
• (maybe a few seconds more delay for data consumers)

Thanks to @pa5cal for suggestions.

[tyrasd]

each query takes about 5-30 seconds

in some of my tests (especially when requesting slightly older adiffs), it was even sometimes the case that generating one minutely diff required more than 1 minute of server time.

Keep the most recent 60-120 augmented diffs precalculated on disk.

Why not more? At least having a day or two (better: a week or two) of cache would make sense IMHO. I think a common use-case of adiffs is the updating of statistics that was initially generated from a planet dump (e.g. for something like tyrasd/taghistory#10). Since those planets are typically generated on a daily or weekly basis, that requires fetching adiffs up to more than ~10 days back.

[mmd-osm]

@tyrasd : 60-120 was just an initial starting point for the discussion based on current log files. I guess once we have this as a configurable parameter it could be easily extended to longer periods of time, if disk space permits.

[mmd-osm]

Fixed in ea9bb55 & f77be3a

[mmd-osm]

@tyrasd : I think Roland put in a default of 60 minutes now. For bigger values you could try a bit of lobbying for your use case... ;)

[pa5cal]

Perfecto!
Thank you very much.

[pa5cal]

Could we increase the caching to a min of 3h or better 6h?

In the last seven days, I got many timeouts and 429 response code during processing the augmented diff files.