* auth: Implement is_restricted_user for OIDC and allow super users to create content regardless of set restriction (PROJQUAY-8208) Currently, if OIDC is set as an authentication mechanism and restricted users is set, Quay will return a `501 Not Implemented` on invocation. Now, Quay will properly check the restricted user whitelist for federated users. Additionally, if user restriction is in place and super user's username was **not** explicitly whitelisted, super users would not be able to create new content inside the registry. Now, the username is explicitly checked in the UI to allow super users to create both organizations and repos regardless of restricted users whitelist.
* Add tests
* Add tests for usermanager
---------
Co-authored-by: Ivan Bazulic <ibazulic@redhat.com>
* logs: Audit export logs requests (PROJQUAY-7679) We add the ability to audit export logs requests that were previously not tracked.
* Add UI elements to properly render new audit log
* Truncate date/time column on exterme zooms
* Add initdb.py entries
* Fix migration and add test db data
* Add test database and fix migration paths
* Changed logging mechanism to grab raised exceptions
* Fix improper import
* Add date/time timestamp to saved metadata
* Change message on export logs screen in UI
* Changed message in old UI as well
* Change log description in new UI too
* Simplify call logic and add additonal information to logged errors
---------
Co-authored-by: Ivan Bazulic <ibazulic@redhat.com>
* gc: fix fk constraint violation tag delete (PROJQUAY-8006)
* revert fix and call delete tag notifications
* add test
---------
Co-authored-by: Sunandadadi <sunanda.3094@gmail.com>
robots: Add robot federation for keyless auth (PROJQUAY-7652)
adds the ability to configure federated auth for robots by
using external OIDC providers. Each robot can be configured
to have multiple external OIDC providers as the source for
authentication.
* health: Add statement timeout to health check (PROJQUAY-7950)
Currently, our query to check team roles (part of the db health check) does not time out after a certain period but runs indefinitely. This causes worker timeouts after 30 seconds, which ends up in worker being killed by the master gunicorn process.
We limit the maximum execution time of the query to 5000 ms so that proper exception is raised if that timeout is reached.
PostgreSQL logs:
~~~
2024-09-16 09:38:56.431 EDT [115775] testuser@quayclone2 LOG: duration: 0.011 ms statement: BEGIN
2024-09-16 09:38:56.431 EDT [115775] testuser@quayclone2 LOG: duration: 0.034 ms statement: SET statement_timeout=5000
2024-09-16 09:38:56.431 EDT [115775] testuser@quayclone2 LOG: duration: 0.010 ms statement: COMMIT
2024-09-16 09:38:56.432 EDT [115775] testuser@quayclone2 LOG: duration: 0.004 ms statement: BEGIN
2024-09-16 09:38:56.432 EDT [115775] testuser@quayclone2 LOG: duration: 0.300 ms statement: SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT 1
2024-09-16 09:38:56.433 EDT [115775] testuser@quayclone2 LOG: duration: 0.010 ms statement: COMMIT
2024-09-16 09:38:56.433 EDT [115775] testuser@quayclone2 LOG: duration: 0.005 ms statement: BEGIN
2024-09-16 09:38:56.433 EDT [115775] testuser@quayclone2 LOG: duration: 0.012 ms statement: SET statement_timeout=0
2024-09-16 09:38:56.433 EDT [115775] testuser@quayclone2 LOG: duration: 0.006 ms statement: COMMIT
~~~
Quay logs:
~~~
gunicorn-web stdout | 2024-09-16 13:38:56,412 [287] [DEBUG] [peewee.pool] Created new connection 127610088683136.
gunicorn-web stdout | 2024-09-16 13:38:56,417 [287] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2024-09-16 13:38:56,418 [287] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-09-16 13:38:56,431 [287] [DEBUG] [peewee] ('SET statement_timeout=%s', (5000,))
gunicorn-web stdout | 2024-09-16 13:38:56,431 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2024-09-16 13:38:56,432 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-09-16 13:38:56,433 [287] [DEBUG] [peewee] ('SET statement_timeout=0', None)
gunicorn-web stdout | 2024-09-16 13:38:56,434 [287] [DEBUG] [app] Ending request: urn:request:d039265b-414e-4d03-b29f-3e481286bf0f (/health/instance)...
~~~
* Fix generator function
Allows users to specify a regex tag pattern when creating namespace/repository autoprune policies via the new UI. Users will have the option to prune tags that only match the tag pattern or exclude tags that match the tag pattern.
during manifest push, we generate a map of blobs which
are part of the manifest layers. This is done using a
UNION query which can overload the worker if the
number of layers is too large. Instead, run each
query individually to prevent the crash
This allows a more refined search than just the repo name. When two
organizations contain the same name repo, e.g: org1/python and
org2/python, you can now search via org1/python to get the specific
result instead of both.
* storage: Disable pushes on registry (PROJQUAY-6870)
The current read-only option for Quay is not sometimes feasible, since it requires an insert of the service key and other manual config changes. For instance, if you want to just recalculate quota on the registry, but would like to allow all registry operations (including UI) without the possibility of pushes until recalculation is done, setting the whole registry `read-only` cannot be done since it makes the database read only as well.
This PR introduces a new flag called `DISABLE_PUSHES` which allows all registry operations to continue (changing tags, repo editing, robot account creation/deletion, user creation etc.) but will disable pushes of new images to the registry (i.e. backend storage will not change). If a registry already contains the image and a new tag is simply being added, that operation should succeed.
The following message would appear in the logs:
~~~
gunicorn-registry stdout | 2024-03-13 20:19:49,414 [369] [DEBUG] [endpoints.v2] sending response: b'{"errors":[{"code":"METHOD NOT ALLOWED","detail":{},"message":"Pushes to the registry are currently disabled. Please contact the administrator for more information."}]}\n'
gunicorn-registry stdout | 2024-03-13 20:19:49,414 [369] [INFO] [gunicorn.access] 172.17.0.1 - - [13/Mar/2024:20:19:49 +0000] "PUT /v2/ibazulic/mariadb/manifests/sha256:c4694ba424e0259694a5117bbb510d67340051f0bdb7f9fa8033941a2d66e53e HTTP/1.1" 405 169 "-" "skopeo/1.9.3"
nginx stdout | 172.17.0.1 (-) - - [13/Mar/2024:20:19:49 +0000] "PUT /v2/ibazulic/mariadb/manifests/sha256:c4694ba424e0259694a5117bbb510d67340051f0bdb7f9fa8033941a2d66e53e HTTP/1.1" 405 169 "-" "skopeo/1.9.3" (0.002 3813 0.002)
~~~
The flag defaults to `False` (pushes enabled), unless set otherwise.
* Removed constraint on storage replication when pushes are disabled
* Rebase
* Fix isort sorting
* Fix isort sorting #2
* Removed constraint on storage replication when pushes are disabled
* Rebase
* Remove constraint on storage replication worker
* Fix linting on config.py
* increase char length in RepoMirrorConfig for external_registry_password
* increase char length in RepoMirrorConfig for external_registry_password
* increase char length in Repo Mirror Config for external_registry_password
* increase char length in Repo Mirror Config for external_registry_password1
Previous assumption made use of the config media type only, which is not the case
if a manifest's artifact type is explicitly set. i.e the config's media type and
artifact type are different, and the artifact type take precedence for filtering.
* database: adding subject_backfilled index to manifest table (PROJQUAY-7360) (#2963)
adding subject_backfilled index to manifest table
* Rebasing with main
* updating cypress data
* Rebasing with latest
* Forcing an empty commit.
* minor fixes
* Autoprune model changes
* fixing failing tests
* format fix
---------
Co-authored-by: Brandon Caton <bcaton@redhat.com>
* database: adding subject_backfilled index to manifest table (PROJQUAY-7360) (#2963)
adding subject_backfilled index to manifest table
* Rebasing with main
* updating cypress data
* fixing conflicts and rebasing with latest code
* adding tests
* Forcing an empty commit.
* Forcing an empty commit.
* skip_locked test fix
* adding tests
* minor fixes
---------
Co-authored-by: Brandon Caton <bcaton@redhat.com>
* resolving conflicts
* database: adding subject_backfilled index to manifest table (PROJQUAY-7360) (#2963)
adding subject_backfilled index to manifest table
* adding index to last_ran_ms and updating alembic head
* minor fixes
* updating cypress data
---------
Co-authored-by: Brandon Caton <bcaton@redhat.com>
* handle cache value size limit in memcached
* only print cache value for redis in debug logs on redis errors
---------
Signed-off-by: dmesser <dmesser@redhat.com>
* Add sqlite db support on quay start up
* Add batchmode to migration scripts to support sqlite db
* Add sqlite db to config-tool validator + alembic migration
* Fix migration script to prevent db row locking
Added commit statement to ensure previous transaction is
completed before the next one within the same table
* Clean up unused sqlite volume
* Apply black formatting to migration scripts
* Address review comments
* Ensure py39-unit test runs the alembic migration on Sqlite
* Add static type checking for alembic config file name
* alembic remove commit and invalidate during migration
When disconnecting from db, alembic tries to rollback
causing PendingRollbackError
* Bump go version in config-tool Dockerfile
* Explicitly commit transaction to prevent db table locking
* Clean up + remove debug statements
* Undo database secret key change
* Add TEST_DATABASE_URI to py39-unit to run unit test with sqlite db
* Drop index before dropping column to prevent sqlite error
* Add test coverage + address last set of reviews
---------
Signed-off-by: harishsurf <hgovinda@redhat.com>
* secscan: Fix Slack notification creation on initial index (PROJQUAY-7037)
This fixes the Slack and e-mail notifications when images are indexed on initial push, where `tags` information was missing. If an image is a manifest child, instead of the tag, we provide a full SHA digest to the client.
Also adds the ability to filter security vulnerabilites depending on their severity. If the `NOTIFICATION_MIN_SEVERITY_ON_NEW_INDEX` is not set in the `config.yaml` file, we will automatically create notifications only for vulnerabilities marked "high" or "critical". This variable can take values defined here:
3248a72da6/util/secscan/__init__.py (L1)
Example:
```
NOTIFICATION_MIN_SEVERITY_ON_NEW_INDEX: Medium
```
* Removed superfulous debug statements
* Add default to config.yaml, remove extra comment
* Fix isort sorting
* Update vulnerability_found.html
* Update vulnerability_found.html
* Isort formatting fix
* Update vulnerability_found.html
* registry: implements the OCI 1.1 referrers API
Migrations:
- Adds a subject column for lookup
- Adds a subject_backfilled column to track status of the backfilling
of existing manifests
- Adds a manifest_json column making use of postgres' JSONB support,
for future use.
Manifestsubjectbackfillworker: Indexes existing manifests for possible
existing subject field.
* Deprecate IGNORE_UNKNOWN_MEDIATYPES
* Cleanup
* marketplace: change redhatsubscriptions model to return a list of customer ids (PROJQUAY-7129)
* Use list comprehension in model and expect to return None if no rows are
returned
* fix behavior for reconciler and fix unit test for reconciler
Reducing the number of DB calls in the repo list endpoint with quota enabled by:
- Adding the id to RepositoryBaseElement when the repositories are initially fetched, removing the need to fetch the repository ID's again
- Fetching the repository sizes with a single DB call using the IN operator
* db: add config variable for DB pooling (PROJQUAY-6397)
adding `DB_CONNECTION_POOLING` param to enable/disable
connection pooling for primary and replica DBs
* db: use read replica for selected queries (PROJQUAY-6397)
We add a new param `can_use_read_replica` to the `select`
query. This allows us to choose which queries we want to
send to the read replica. This is useful in cases where
the read replica lags behind the primary and some queries
need the latest data
