* storage: Disable pushes on registry (PROJQUAY-6870)
The current read-only option for Quay is not sometimes feasible, since it requires an insert of the service key and other manual config changes. For instance, if you want to just recalculate quota on the registry, but would like to allow all registry operations (including UI) without the possibility of pushes until recalculation is done, setting the whole registry `read-only` cannot be done since it makes the database read only as well.
This PR introduces a new flag called `DISABLE_PUSHES` which allows all registry operations to continue (changing tags, repo editing, robot account creation/deletion, user creation etc.) but will disable pushes of new images to the registry (i.e. backend storage will not change). If a registry already contains the image and a new tag is simply being added, that operation should succeed.
The following message would appear in the logs:
~~~
gunicorn-registry stdout | 2024-03-13 20:19:49,414 [369] [DEBUG] [endpoints.v2] sending response: b'{"errors":[{"code":"METHOD NOT ALLOWED","detail":{},"message":"Pushes to the registry are currently disabled. Please contact the administrator for more information."}]}\n'
gunicorn-registry stdout | 2024-03-13 20:19:49,414 [369] [INFO] [gunicorn.access] 172.17.0.1 - - [13/Mar/2024:20:19:49 +0000] "PUT /v2/ibazulic/mariadb/manifests/sha256:c4694ba424e0259694a5117bbb510d67340051f0bdb7f9fa8033941a2d66e53e HTTP/1.1" 405 169 "-" "skopeo/1.9.3"
nginx stdout | 172.17.0.1 (-) - - [13/Mar/2024:20:19:49 +0000] "PUT /v2/ibazulic/mariadb/manifests/sha256:c4694ba424e0259694a5117bbb510d67340051f0bdb7f9fa8033941a2d66e53e HTTP/1.1" 405 169 "-" "skopeo/1.9.3" (0.002 3813 0.002)
~~~
The flag defaults to `False` (pushes enabled), unless set otherwise.
* Removed constraint on storage replication when pushes are disabled
* Rebase
* Fix isort sorting
* Fix isort sorting #2
* Removed constraint on storage replication when pushes are disabled
* Rebase
* Remove constraint on storage replication worker
* Fix linting on config.py
Adds ACCOUNT_RECOVERY_MODE to allow Quay to run with some core
features disabled. When this is set, the instance should only be used
in order by existing users who hasn't linked their account to an
external login service, after database authentication has been
disabled.
GlobalLock had a dependency on app, which would cause a circular
dependency if imported from the main app. Workaround this by requiring
to pass the configuration to the GlobalLock instead (this is done by a
classmethod, due to the use of Redlock's factory). This means before
the use of GlobalLock, "configure" will need to be called once, per process.
Prevents the queueworker from setting the event to stop the poll_queue
job when a WorkerSleepException is raised. On WorkerSleepException,
the worker should instead skip this iteration (go to sleep). e.g when
the NamespaceGCWorker can't acquire a lock because it is already taken
by some other worker.
Reverts the gcworkers job timeout from 24h to 3h. In case of a
deadlock between processes (for example, redeploying the app will not
clear the existing Redis keys), 24h is too long waiting for the locks to
expires so that the workers can resume work.
Add missing Counter increment for on row deletion on the Manifest table.
Correctly converts the given ttl from seconds to milliseconds when
passed to Redis (redlock uses 'px', not 'ex'). Also increase the lock
timeout of gc workers to 1 day.
Some iteration, for repos with large numbers of tags (1000s), will
take more than 15 minutes to complete. This change will prevent multiple
workers GCing the same repo, and one possibly preempting
another. GlobalLock's ttl will make the lock available again when
expired, but will not actually stop execution of the current GC
iteration until the GlobalLock context is done. Having a 1 day timeout
should be enough.
NOTE: The correct solution would have GlobalLock should either renew
the lock until the caller is done, or signal that it is no longer
valid to the caller.
* local-dev: implement local development environment
this commit copies the files in /init into /local-dev, edits those files
to support hot-reload features for local development, and introduces
docker-compose/makefile targets in order to support local dev.
Signed-off-by: ldelossa <ldelossa@redhat.com>
* local-dev: hop quay workers to gunicorn
this commit adds a uwsgi worker application delegate and a factory
function allowing each worker to be ran by gunicorn.
each worker now supports hot-reload and will reload itself when it's
code is updated.
this changes only affects the local dev env.
Signed-off-by: ldelossa <ldelossa@redhat.com>
* local-dev: add docs
Signed-off-by: ldelossa <ldelossa@redhat.com>
While a transaction is obviously safer, with the number of tables
and rows referencing these tables now, a transaction is potentially
locking up a significant chunk of the database. Since we're already
performing cleanup before calling the delete, including disabling
new data being written for the User or Repository, deletion without
a transaction should (usually) be sufficient; if it isn't, an
IntegrityError will be raised, and the workers can retry continuing
the GC operation
* Change storage GC to process a single row at a time
This should remove the deadlock under the transaction and be much less
heavy on the DB
* Ensure we don't select repositories for GC from those already marked
for deletion or those under to-be-deleted namespaces
* Ensure that GC operations occur under global locks, to prevent
concurrent GC of the same repositories, which should reduce lock
contention on the database
