registry: index manifest artifact type for filtering (PROJQUAY-7471)
Previous assumption made use of the config media type only, which is not the case
if a manifest's artifact type is explicitly set. i.e the config's media type and
artifact type are different, and the artifact type take precedence for filtering.
Co-authored-by: Kenny Lee Sin Cheong <kenny.lee28@gmail.com>
* database: adding subject_backfilled index to manifest table (PROJQUAY-7360) (#2963)
adding subject_backfilled index to manifest table
* Rebasing with main
* updating cypress data
* Rebasing with latest
* Forcing an empty commit.
* minor fixes
* Autoprune model changes
* fixing failing tests
* format fix
---------
Co-authored-by: Brandon Caton <bcaton@redhat.com>
* database: adding subject_backfilled index to manifest table (PROJQUAY-7360) (#2963)
adding subject_backfilled index to manifest table
* Rebasing with main
* updating cypress data
* fixing conflicts and rebasing with latest code
* adding tests
* Forcing an empty commit.
* Forcing an empty commit.
* skip_locked test fix
* adding tests
* minor fixes
---------
Co-authored-by: Brandon Caton <bcaton@redhat.com>
* resolving conflicts
* database: adding subject_backfilled index to manifest table (PROJQUAY-7360) (#2963)
adding subject_backfilled index to manifest table
* adding index to last_ran_ms and updating alembic head
* minor fixes
* updating cypress data
---------
Co-authored-by: Brandon Caton <bcaton@redhat.com>
* handle cache value size limit in memcached
* only print cache value for redis in debug logs on redis errors
---------
Signed-off-by: dmesser <dmesser@redhat.com>
* Add sqlite db support on quay start up
* Add batchmode to migration scripts to support sqlite db
* Add sqlite db to config-tool validator + alembic migration
* Fix migration script to prevent db row locking
Added commit statement to ensure previous transaction is
completed before the next one within the same table
* Clean up unused sqlite volume
* Apply black formatting to migration scripts
* Address review comments
* Ensure py39-unit test runs the alembic migration on Sqlite
* Add static type checking for alembic config file name
* alembic remove commit and invalidate during migration
When disconnecting from db, alembic tries to rollback
causing PendingRollbackError
* Bump go version in config-tool Dockerfile
* Explicitly commit transaction to prevent db table locking
* Clean up + remove debug statements
* Undo database secret key change
* Add TEST_DATABASE_URI to py39-unit to run unit test with sqlite db
* Drop index before dropping column to prevent sqlite error
* Add test coverage + address last set of reviews
---------
Signed-off-by: harishsurf <hgovinda@redhat.com>
* secscan: Fix Slack notification creation on initial index (PROJQUAY-7037)
This fixes the Slack and e-mail notifications when images are indexed on initial push, where `tags` information was missing. If an image is a manifest child, instead of the tag, we provide a full SHA digest to the client.
Also adds the ability to filter security vulnerabilites depending on their severity. If the `NOTIFICATION_MIN_SEVERITY_ON_NEW_INDEX` is not set in the `config.yaml` file, we will automatically create notifications only for vulnerabilities marked "high" or "critical". This variable can take values defined here:
3248a72da6/util/secscan/__init__.py (L1)
Example:
```
NOTIFICATION_MIN_SEVERITY_ON_NEW_INDEX: Medium
```
* Removed superfulous debug statements
* Add default to config.yaml, remove extra comment
* Fix isort sorting
* Update vulnerability_found.html
* Update vulnerability_found.html
* Isort formatting fix
* Update vulnerability_found.html
* registry: implements the OCI 1.1 referrers API
Migrations:
- Adds a subject column for lookup
- Adds a subject_backfilled column to track status of the backfilling
of existing manifests
- Adds a manifest_json column making use of postgres' JSONB support,
for future use.
Manifestsubjectbackfillworker: Indexes existing manifests for possible
existing subject field.
* Deprecate IGNORE_UNKNOWN_MEDIATYPES
* Cleanup
* marketplace: change redhatsubscriptions model to return a list of customer ids (PROJQUAY-7129)
* Use list comprehension in model and expect to return None if no rows are
returned
* fix behavior for reconciler and fix unit test for reconciler
Reducing the number of DB calls in the repo list endpoint with quota enabled by:
- Adding the id to RepositoryBaseElement when the repositories are initially fetched, removing the need to fetch the repository ID's again
- Fetching the repository sizes with a single DB call using the IN operator
* db: add config variable for DB pooling (PROJQUAY-6397)
adding `DB_CONNECTION_POOLING` param to enable/disable
connection pooling for primary and replica DBs
* db: use read replica for selected queries (PROJQUAY-6397)
We add a new param `can_use_read_replica` to the `select`
query. This allows us to choose which queries we want to
send to the read replica. This is useful in cases where
the read replica lags behind the primary and some queries
need the latest data
* feat: Add support for auto pruning at repository level (PROJQUAY-6354)
* Add repositoryautoprunepolicy table to alembic migration script
* Add repository auto-prune policy endpoints
* Add UI for repository auto-pruning policies
* case: apply repo auto-prune policy when no namespace policy given
* case: both namespace and repo pruning policy are given
* Add tests for repository autoprune endpoint
* Add cypress test for repository auto-prune
* Add repo auto-prune policy clean-up for repository deletion
* Add repository auto pruning tables to quay db snapshot for cypress tests
* Address review comments
* Add more tests + fix CI + reformat files
* Address review comments #2
---------
Signed-off-by: harishsurf <hgovinda@redhat.com>
* added base class for OIDC auth + UI
* adding read-only teams page + display sync config + option to remove team sync
* setting page in read only mode fix
* ui tests
* adding validation for group name input
* fixes based on review + fixing test suite
* add backend tests for externalOIDC
* minor fixes
* Adds handling for when a subscription returned from the subscription watch api has a quantity greater than 1. Number of private repos should be correctly calculated using the quantity.
* Updates ui so that subscriptions can only be added to an org as a group, i.e. a subscription with quantity = 2 cannot be split across organizations.
* billing: marketplace UI
adds UI in billing section for managing user and org-bound skus
add more unit tests for org binding
changed endpoint for bulk attaching skus to orgs
* update reconciliationworker to use webCustomerId instead of
ebsAccountNumber
* fix reconciler where it was incorrectly using the ebsAccountNumber to
create subscriptions
* add job to reconciler so that it reconciles different ids between the
database and the user api
* separate skus to be used by billing and skus to be used by reconciler
