Add a global lock on security worker iterations, based on the value of
the current pagination token. This is to avoid multilple worker
processes possibly doing work on the same set of manifests.
GlobalLock had a dependency on app, which would cause a circular
dependency if imported from the main app. Workaround this by requiring
to pass the configuration to the GlobalLock instead (this is done by a
classmethod, due to the use of Redlock's factory). This means before
the use of GlobalLock, "configure" will need to be called once, per process.
By default, Redlock creates a new client per instance. Using the
provided factory allows Redlock to reuse a single connection per
instance and avoid running out of connections. e.g When a worker tries
to get a lock, it should not open new connections every time.
Increase sleep duration between queue polls on
WorkerSleepException. This will give more time before retrying after
failing to acquire a lock.
Correctly converts the given ttl from seconds to milliseconds when
passed to Redis (redlock uses 'px', not 'ex'). Also increase the lock
timeout of gc workers to 1 day.
Some iteration, for repos with large numbers of tags (1000s), will
take more than 15 minutes to complete. This change will prevent multiple
workers GCing the same repo, and one possibly preempting
another. GlobalLock's ttl will make the lock available again when
expired, but will not actually stop execution of the current GC
iteration until the GlobalLock context is done. Having a 1 day timeout
should be enough.
NOTE: The correct solution would have GlobalLock should either renew
the lock until the caller is done, or signal that it is no longer
valid to the caller.
