* fix(test): prevent MySQL deadlocks in parallel proxy model tests (PROJQUAY-0000)
Mark all registry proxy model test classes to run serially using
pytest-xdist group markers. These tests all use the same "quayio-cache"
organization and were causing MySQL deadlocks when run in parallel
across multiple workers with pytest -n auto.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(test): resolve Flask app naming conflict in quotaregistrysizeworker tests (PROJQUAY-0000)
Import Flask app with alias to avoid conflict with pytest 'app' fixture.
The test was using 'app.config' but 'app' resolved to a pytest fixture
definition instead of the Flask application object.
Follows the same pattern as test_securityscanningnotificationworker.py.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Brady Pratt <bpratt@redhat.com>
---------
Signed-off-by: Brady Pratt <bpratt@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
* Fix race conditions in pull metrics tracking and flushing
Replace non-atomic operations with atomic Redis operations to prevent
data loss when concurrent pulls occur during flush operations.
* fixing tests
* updating tests
* added uuid to the rename factors to ensure unique key at concurrent requests
---------
Co-authored-by: shudeshp <shudeshp@redhat.com>
* Added redis database and redis flush worker
* updated digest validation
* adding test coverage for scan_keys function
* adding test coverage
* added tests for scan function
* added coverage for flush to database function
* added coverage for gnuicorn worker initialization
* Replaced mock methods in the test with actual function calls
---------
Co-authored-by: shudeshp <shudeshp@redhat.com>
* fix: resolve Sentry/OpenTelemetry integration conflicts
Configure Sentry to use minimal integrations when OTEL_TRACING is enabled
to prevent instrumentation conflicts that broke exception capture
---------
Co-authored-by: shudeshp <shudeshp@redhat.com>
* proxycache: Download blob not cached when pulling manifests with blob available locally (PROJQUAY-6708)
* Skip downloading blobs without placeholders
Adds an include_orgs param to the active users query used by the reconciler and sets it to true for reconciler runs
Reconciler is not including orgs as a candidate for creating corresponding RH entitlements. As a result it misses users with stripe billing that are considered orgs.
* notifications: fetch autoprune tags with multiple policies for image expiry notification(PROJQUAY-8117)
* don't fetch notifications if tags expiry is greater than notification days + add tests
Allows users to specify a regex tag pattern when creating namespace/repository autoprune policies via the new UI. Users will have the option to prune tags that only match the tag pattern or exclude tags that match the tag pattern.
* registry: implements the OCI 1.1 referrers API
Migrations:
- Adds a subject column for lookup
- Adds a subject_backfilled column to track status of the backfilling
of existing manifests
- Adds a manifest_json column making use of postgres' JSONB support,
for future use.
Manifestsubjectbackfillworker: Indexes existing manifests for possible
existing subject field.
* Deprecate IGNORE_UNKNOWN_MEDIATYPES
* Cleanup
* marketplace: change redhatsubscriptions model to return a list of customer ids (PROJQUAY-7129)
* Use list comprehension in model and expect to return None if no rows are
returned
* fix behavior for reconciler and fix unit test for reconciler
Changes marketplace api handler to now return a list of all web customer ids for a user.
Allows quay.io to keep track of all web customer ids instead of just personal types.
* billing: marketplace UI
adds UI in billing section for managing user and org-bound skus
add more unit tests for org binding
changed endpoint for bulk attaching skus to orgs
* update reconciliationworker to use webCustomerId instead of
ebsAccountNumber
* fix reconciler where it was incorrectly using the ebsAccountNumber to
create subscriptions
* add job to reconciler so that it reconciles different ids between the
database and the user api
* separate skus to be used by billing and skus to be used by reconciler
* chore: pass config to isort as it doesn't always detect it
* chore: mark package "test" as local, not stdlib
* chore: remove "isort: skip_file"
* chore: fix app in test_load_security_information
* chore: fix app in test_notification
* chore: fix app in test_index_report
* add migration for orgrhskus table
* add endpoints for managing and listing skus bound to an org
* create checks in billing flow to look for org-bound skus
* refactor RH marketplace api objects to be more usable in tests
* update cypress test db data and exclude it from pre-commit hook formatting
Garbage collect manifests no longer referenced in Quay from the
security scanner service.
Also moved quota related code from data/registry_model/ to data/model/
to avoid circular dependencies.
Allows for only unique blobs are counted at the namespace and repository level. Calculation includes manifest list sizes.
Add's the following internal configurations that default to true:
QUOTA_INVALIDATE_TOTALS: Invalidates calculated totals when FEATURE_QUOTA_MANAGEMENT is set to false
RESET_CHILD_MANIFEST_EXPIRATION: Resets the expiry for child manifests on push of the manifest list for immediate GC eligibility
PERMANENTLY_DELETE_TAGS: Enables features related to the permanent deletion of tags outside the configured time machine window
By default, Redlock creates a new client per instance. Using the
provided factory allows Redlock to reuse a single connection per
instance and avoid running out of connections. e.g When a worker tries
to get a lock, it should not open new connections every time.
Increase sleep duration between queue polls on
WorkerSleepException. This will give more time before retrying after
failing to acquire a lock.
Migrate from using boto2 to boto3. Changes include:
- Removes explicit bucket addressing style: Boto3 will initially try virtual-style addressing first then fallback to path-style addressing (https://github.com/boto/boto3/blob/develop/docs/source/guide/configuration.rst)
- GCS workarounds to use boto3:
- Handles CORS config
- Update signed url access key parameter name
- Uses ListBucket V1 API
- On client-side chunks join, copy using non-multipart api: Use copy_from instead of copy when joining chunks client-side. This is because copy assumes multipart upload should be used which GCS and Rados are not compatible with (S3's version. They have their own parallel upload api)
- Update RDS healthcheck to use boto3
* Change verbs to use a DerivedStorageForManifest table instead of DerivedStorageForImage
This allows us to deprecate the DerivedStorageForImage table.
Fixes https://issues.redhat.com/browse/PROJQUAY-519
* Change uploaded blob tracking to use its own table and deprecate
RepositoryTag
* Start recording the compressed layers size and config media type on the
manifest row in the database
NOTE: This change includes a database migration which will *lock* the
manifest table
* Change tag API to return the layers size from the manifest
* Remove unused code
* Add new config_media_type field to OCI types
* Fix secscan V2 test for us no longer writing temp images
* Remove unused uploading field
* Switch registry model to use synthetic legacy images
Legacy images are now (with exception of the V2 security model) read from the *manifest* and sythensized in memory. The legacy image IDs are generated realtime based on the hashids library. This change also further deprecates a bunch of our Image APIs, reducing them to only returning the image IDs, and emptying out the remaining metadata (to avoid the requirement of us loading the information for the manifest from storage).
This has been tested with our full clients test suite with success.
* Add a backfill worker for manifest layers compressed sizes
* Change image tracks into manifest tracks now that we no longer have
manifest-less tags
* Add back in the missing method
* Add missing joins to reduce extra queries
* Remove unnecessary join when looking up legacy images
* Remove extra hidden filter on tag queries
* Further DB improvements
* Delete all Verbs, as they were deprecated
* Add back missing parameter in manifest data type
* Fix join to return None for the robot if not defined on mirror config
* switch to using secscan_v4_model for all indexing and remove most of secscan_v2_model code
* Add a missing join
* Remove files accidentally re-added due to rebase
* Add back hashids lib
* Rebase fixes
* Fix broken test
* Remove unused GPG signer now that ACI conversion is removed
* Remove duplicated repomirrorworker
* Remove unused notification code for secscan. We'll re-add it once Clair
V4 security notifications are ready to go
* Fix formatting
* Stop writing Image rows when creating manifests
* Stop writing empty layer blobs for manifests
As these blobs are shared, we don't need to write ManifestBlob rows
for them
* Remove further unused code
* Add doc comment to _build_blob_map
* Add unit test for synthetic V1 IDs
* Remove unused import
* Add an invalid value test to synthetic ID decode tests
* Add manifest backfill worker back in
Seems to have been removed at some point
* Add a test for cached active tags
* Rename test_shared to not conflict with another same-named test file
Pytest doesn't like having two test modules with the same name
* Have manifestbackfillworker also copy over the config_media_type if present
Co-authored-by: alecmerdler <alecmerdler@gmail.com>
