Adds an include_orgs param to the active users query used by the reconciler and sets it to true for reconciler runs
Reconciler is not including orgs as a candidate for creating corresponding RH entitlements. As a result it misses users with stripe billing that are considered orgs.
* notifications: fetch autoprune tags with multiple policies for image expiry notification(PROJQUAY-8117)
* don't fetch notifications if tags expiry is greater than notification days + add tests
Allows users to specify a regex tag pattern when creating namespace/repository autoprune policies via the new UI. Users will have the option to prune tags that only match the tag pattern or exclude tags that match the tag pattern.
* storage: Disable pushes on registry (PROJQUAY-6870)
The current read-only option for Quay is not sometimes feasible, since it requires an insert of the service key and other manual config changes. For instance, if you want to just recalculate quota on the registry, but would like to allow all registry operations (including UI) without the possibility of pushes until recalculation is done, setting the whole registry `read-only` cannot be done since it makes the database read only as well.
This PR introduces a new flag called `DISABLE_PUSHES` which allows all registry operations to continue (changing tags, repo editing, robot account creation/deletion, user creation etc.) but will disable pushes of new images to the registry (i.e. backend storage will not change). If a registry already contains the image and a new tag is simply being added, that operation should succeed.
The following message would appear in the logs:
~~~
gunicorn-registry stdout | 2024-03-13 20:19:49,414 [369] [DEBUG] [endpoints.v2] sending response: b'{"errors":[{"code":"METHOD NOT ALLOWED","detail":{},"message":"Pushes to the registry are currently disabled. Please contact the administrator for more information."}]}\n'
gunicorn-registry stdout | 2024-03-13 20:19:49,414 [369] [INFO] [gunicorn.access] 172.17.0.1 - - [13/Mar/2024:20:19:49 +0000] "PUT /v2/ibazulic/mariadb/manifests/sha256:c4694ba424e0259694a5117bbb510d67340051f0bdb7f9fa8033941a2d66e53e HTTP/1.1" 405 169 "-" "skopeo/1.9.3"
nginx stdout | 172.17.0.1 (-) - - [13/Mar/2024:20:19:49 +0000] "PUT /v2/ibazulic/mariadb/manifests/sha256:c4694ba424e0259694a5117bbb510d67340051f0bdb7f9fa8033941a2d66e53e HTTP/1.1" 405 169 "-" "skopeo/1.9.3" (0.002 3813 0.002)
~~~
The flag defaults to `False` (pushes enabled), unless set otherwise.
* Removed constraint on storage replication when pushes are disabled
* Rebase
* Fix isort sorting
* Fix isort sorting #2
* Removed constraint on storage replication when pushes are disabled
* Rebase
* Remove constraint on storage replication worker
* Fix linting on config.py
Previous assumption made use of the config media type only, which is not the case
if a manifest's artifact type is explicitly set. i.e the config's media type and
artifact type are different, and the artifact type take precedence for filtering.
* feature flag: add feature flag for image expiry trigger (PROJQUAY-7460)
* add FEATURE_IMAGE_EXPIRY_TRIGGER to CLIENT_WHITELIST
* minor fix
* setting IMAGE_EXPIRY_TRIGGER to true for testing
* fixing config.json
* database: adding subject_backfilled index to manifest table (PROJQUAY-7360) (#2963)
adding subject_backfilled index to manifest table
* Rebasing with main
* updating cypress data
* fixing conflicts and rebasing with latest code
* adding tests
* Forcing an empty commit.
* Forcing an empty commit.
* skip_locked test fix
* adding tests
* minor fixes
---------
Co-authored-by: Brandon Caton <bcaton@redhat.com>
* registry: implements the OCI 1.1 referrers API
Migrations:
- Adds a subject column for lookup
- Adds a subject_backfilled column to track status of the backfilling
of existing manifests
- Adds a manifest_json column making use of postgres' JSONB support,
for future use.
Manifestsubjectbackfillworker: Indexes existing manifests for possible
existing subject field.
* Deprecate IGNORE_UNKNOWN_MEDIATYPES
* Cleanup
* marketplace: change redhatsubscriptions model to return a list of customer ids (PROJQUAY-7129)
* Use list comprehension in model and expect to return None if no rows are
returned
* fix behavior for reconciler and fix unit test for reconciler
Changes marketplace api handler to now return a list of all web customer ids for a user.
Allows quay.io to keep track of all web customer ids instead of just personal types.
* feat: Add support for auto pruning at repository level (PROJQUAY-6354)
* Add repositoryautoprunepolicy table to alembic migration script
* Add repository auto-prune policy endpoints
* Add UI for repository auto-pruning policies
* case: apply repo auto-prune policy when no namespace policy given
* case: both namespace and repo pruning policy are given
* Add tests for repository autoprune endpoint
* Add cypress test for repository auto-prune
* Add repo auto-prune policy clean-up for repository deletion
* Add repository auto pruning tables to quay db snapshot for cypress tests
* Address review comments
* Add more tests + fix CI + reformat files
* Address review comments #2
---------
Signed-off-by: harishsurf <hgovinda@redhat.com>
* billing: marketplace UI
adds UI in billing section for managing user and org-bound skus
add more unit tests for org binding
changed endpoint for bulk attaching skus to orgs
* update reconciliationworker to use webCustomerId instead of
ebsAccountNumber
* fix reconciler where it was incorrectly using the ebsAccountNumber to
create subscriptions
* add job to reconciler so that it reconciles different ids between the
database and the user api
* separate skus to be used by billing and skus to be used by reconciler
* chore: pass config to isort as it doesn't always detect it
* chore: mark package "test" as local, not stdlib
* chore: remove "isort: skip_file"
* chore: fix app in test_load_security_information
* chore: fix app in test_notification
* chore: fix app in test_index_report
* add migration for orgrhskus table
* add endpoints for managing and listing skus bound to an org
* create checks in billing flow to look for org-bound skus
* refactor RH marketplace api objects to be more usable in tests
* update cypress test db data and exclude it from pre-commit hook formatting
Garbage collect manifests no longer referenced in Quay from the
security scanner service.
Also moved quota related code from data/registry_model/ to data/model/
to avoid circular dependencies.
* Aggregate stripe plans and subscriptions.
* Validate end date for subscriptions when fetching from marketplace.
* Check subscription returned from stripe api is non-null value when
finding stripe plan for sku.
Allows superusers to trigger a calculation of the deduplicated registry size. A superuser can go to the organization panel of the superuser page and select Calculate to queue a calculation of the registry total. The total will only be calculated when requested. Includes warning to user of increase of database load when running calculation.
Allow the replication worker to retry immediately without having to
wait and exhaust the queueitem's retries. This addresses transient
unreliable storage service issues.
Allows for only unique blobs are counted at the namespace and repository level. Calculation includes manifest list sizes.
Add's the following internal configurations that default to true:
QUOTA_INVALIDATE_TOTALS: Invalidates calculated totals when FEATURE_QUOTA_MANAGEMENT is set to false
RESET_CHILD_MANIFEST_EXPIRATION: Resets the expiry for child manifests on push of the manifest list for immediate GC eligibility
PERMANENTLY_DELETE_TAGS: Enables features related to the permanent deletion of tags outside the configured time machine window
* repomirror: Use skopeo list-tags to get repo tags
`skopeo inspect foo` returns infromation about the image `foo:latest`,
and repository tags. Quay needs only list of tags, so it should use
`skopeo list-tags` that doesn't fail if `foo:latest` doesn't exist.
* Update type hints
* On local-dev Quay does not provide valid TLS certificate
Currently Quay creates tags for Docker V2 schema 1 manifests in manifest lists. This makes it appear a tag was mirrored successfully when it had actually failed. This change rolls back those failed tags when the sync fails.
Adds the `REPO_MIRROR_ROLLBACK` option to specify whether the mirror will rollback the state of the repo on failure of any one of the tags. Defaults to false. Adds additional `PARTIAL_SYNC` error status which logs the tags that failed to sync to the console.
Previous logic for claiming mirror ("locking") relied on the value
returned from updating the database row. Since this was always being
updated with a new expiration time, it would always succeed, even when
another process had already claimed the same mirror.
Having only one worker index recent manifest doesn't keep up with the
rate new manifests being pushed, given the time it takes for an index
request to complete. Adding the option to bypass the global lock
allows for more workers, but also increase the chance of duplicate work.
