Changing logger.info to logger.debug to stop too many INFO log lines from displaying in Quay logs + Adding the missing worker name that print those
The idea of the log is to tell you which block is being procesed by worker that is processing it. These logs are missing the worker names
```
2023-11-27 15:24:05,825 [80] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 210690-213109
2023-11-27 15:24:05,832 [80] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 131657-134076
[sbhavsar-mac:~ sayalibhavsar]$ grep "No candidates, marking entire block completed" quay | wc -l
5300
```
For huge registries above logs could be a bit noisy. We could hide it as a debug message rather than as an info message.
Co-authored-by: sayali bhavsar <sayalibhavsar9009@gmail.com>
Co-authored-by: Marcus Kok <47163063+Marcusk19@users.noreply.github.com>
* database: adding subject_backfilled index to manifest table (PROJQUAY-7360) (#2963)
adding subject_backfilled index to manifest table
* Rebasing with main
* updating cypress data
* Rebasing with latest
* Forcing an empty commit.
* minor fixes
* Autoprune model changes
* fixing failing tests
* format fix
---------
Co-authored-by: Brandon Caton <bcaton@redhat.com>
* database: adding subject_backfilled index to manifest table (PROJQUAY-7360) (#2963)
adding subject_backfilled index to manifest table
* Rebasing with main
* updating cypress data
* fixing conflicts and rebasing with latest code
* adding tests
* Forcing an empty commit.
* Forcing an empty commit.
* skip_locked test fix
* adding tests
* minor fixes
---------
Co-authored-by: Brandon Caton <bcaton@redhat.com>
* Add sqlite db support on quay start up
* Add batchmode to migration scripts to support sqlite db
* Add sqlite db to config-tool validator + alembic migration
* Fix migration script to prevent db row locking
Added commit statement to ensure previous transaction is
completed before the next one within the same table
* Clean up unused sqlite volume
* Apply black formatting to migration scripts
* Address review comments
* Ensure py39-unit test runs the alembic migration on Sqlite
* Add static type checking for alembic config file name
* alembic remove commit and invalidate during migration
When disconnecting from db, alembic tries to rollback
causing PendingRollbackError
* Bump go version in config-tool Dockerfile
* Explicitly commit transaction to prevent db table locking
* Clean up + remove debug statements
* Undo database secret key change
* Add TEST_DATABASE_URI to py39-unit to run unit test with sqlite db
* Drop index before dropping column to prevent sqlite error
* Add test coverage + address last set of reviews
---------
Signed-off-by: harishsurf <hgovinda@redhat.com>
* secscan: Fix Slack notification creation on initial index (PROJQUAY-7037)
This fixes the Slack and e-mail notifications when images are indexed on initial push, where `tags` information was missing. If an image is a manifest child, instead of the tag, we provide a full SHA digest to the client.
Also adds the ability to filter security vulnerabilites depending on their severity. If the `NOTIFICATION_MIN_SEVERITY_ON_NEW_INDEX` is not set in the `config.yaml` file, we will automatically create notifications only for vulnerabilities marked "high" or "critical". This variable can take values defined here:
3248a72da6/util/secscan/__init__.py (L1)
Example:
```
NOTIFICATION_MIN_SEVERITY_ON_NEW_INDEX: Medium
```
* Removed superfulous debug statements
* Add default to config.yaml, remove extra comment
* Fix isort sorting
* Update vulnerability_found.html
* Update vulnerability_found.html
* Isort formatting fix
* Update vulnerability_found.html
Add `Vary: Origin` header to the 404 responses to instruct browsers (and other utilities) to cache various requests properly (based on their origin) in order to avoid cache poisoning. See the [W3C security details](https://www.w3.org/TR/2020/SPSD-cors-20200602/#resource-security) for more information.
* registry: implements the OCI 1.1 referrers API
Migrations:
- Adds a subject column for lookup
- Adds a subject_backfilled column to track status of the backfilling
of existing manifests
- Adds a manifest_json column making use of postgres' JSONB support,
for future use.
Manifestsubjectbackfillworker: Indexes existing manifests for possible
existing subject field.
* Deprecate IGNORE_UNKNOWN_MEDIATYPES
* Cleanup
* marketplace: change redhatsubscriptions model to return a list of customer ids (PROJQUAY-7129)
* Use list comprehension in model and expect to return None if no rows are
returned
* fix behavior for reconciler and fix unit test for reconciler
Changes marketplace api handler to now return a list of all web customer ids for a user.
Allows quay.io to keep track of all web customer ids instead of just personal types.
* db: add config variable for DB pooling (PROJQUAY-6397)
adding `DB_CONNECTION_POOLING` param to enable/disable
connection pooling for primary and replica DBs
* Adds handling for when a subscription returned from the subscription watch api has a quantity greater than 1. Number of private repos should be correctly calculated using the quantity.
* Updates ui so that subscriptions can only be added to an org as a group, i.e. a subscription with quantity = 2 cannot be split across organizations.
* update reconciliationworker to use webCustomerId instead of
ebsAccountNumber
* fix reconciler where it was incorrectly using the ebsAccountNumber to
create subscriptions
* add job to reconciler so that it reconciles different ids between the
database and the user api
* separate skus to be used by billing and skus to be used by reconciler
* chore: pass config to isort as it doesn't always detect it
* chore: mark package "test" as local, not stdlib
* chore: remove "isort: skip_file"
* chore: fix app in test_load_security_information
* chore: fix app in test_notification
* chore: fix app in test_index_report
* add Postgresql client certificate authentication option
* user(robots): disallow robot login and create PROJQUAY-5968
Add a config flag `ROBOTS_DISALLOW` to turn off Robot login and creation
in Quay for all accounts.
with the Flag set existing Robots cannot login anymore
```
$ podman login -u milang+test2 -p W7B...HQA quay.example.com
Error: logging into "quay.example.com": invalid username/password
```
This behavior will be reported in the logs accordingly
```
gunicorn-registry stdout | 2023-08-24 19:08:01,907 [253] [WARNING] [auth.credentials] Failed to validate credentials for robot milang+test2: Robot account has been disabled. Please contact your administrator.
```
as well as in the UI when creating a Robot Account which will display
the `Robot account has been disabled. Please contact your
administrator.` message.
* Revert "add Postgresql client certificate authentication option"
need to branch for PR (stupid me)
This reverts commit f8f54eead2.
* fixed error message to proper english and added schema definition for the option
* add migration for orgrhskus table
* add endpoints for managing and listing skus bound to an org
* create checks in billing flow to look for org-bound skus
* refactor RH marketplace api objects to be more usable in tests
* update cypress test db data and exclude it from pre-commit hook formatting
