registry: index manifest artifact type for filtering (PROJQUAY-7471)
Previous assumption made use of the config media type only, which is not the case
if a manifest's artifact type is explicitly set. i.e the config's media type and
artifact type are different, and the artifact type take precedence for filtering.
Co-authored-by: Kenny Lee Sin Cheong <kenny.lee28@gmail.com>
* database: adding subject_backfilled index to manifest table (PROJQUAY-7360) (#2963)
adding subject_backfilled index to manifest table
* Rebasing with main
* updating cypress data
* fixing conflicts and rebasing with latest code
* adding tests
* Forcing an empty commit.
* Forcing an empty commit.
* skip_locked test fix
* adding tests
* minor fixes
---------
Co-authored-by: Brandon Caton <bcaton@redhat.com>
* registry: implements the OCI 1.1 referrers API
Migrations:
- Adds a subject column for lookup
- Adds a subject_backfilled column to track status of the backfilling
of existing manifests
- Adds a manifest_json column making use of postgres' JSONB support,
for future use.
Manifestsubjectbackfillworker: Indexes existing manifests for possible
existing subject field.
* Deprecate IGNORE_UNKNOWN_MEDIATYPES
* Cleanup
* marketplace: change redhatsubscriptions model to return a list of customer ids (PROJQUAY-7129)
* Use list comprehension in model and expect to return None if no rows are
returned
* fix behavior for reconciler and fix unit test for reconciler
Changes marketplace api handler to now return a list of all web customer ids for a user.
Allows quay.io to keep track of all web customer ids instead of just personal types.
* feat: Add support for auto pruning at repository level (PROJQUAY-6354)
* Add repositoryautoprunepolicy table to alembic migration script
* Add repository auto-prune policy endpoints
* Add UI for repository auto-pruning policies
* case: apply repo auto-prune policy when no namespace policy given
* case: both namespace and repo pruning policy are given
* Add tests for repository autoprune endpoint
* Add cypress test for repository auto-prune
* Add repo auto-prune policy clean-up for repository deletion
* Add repository auto pruning tables to quay db snapshot for cypress tests
* Address review comments
* Add more tests + fix CI + reformat files
* Address review comments #2
---------
Signed-off-by: harishsurf <hgovinda@redhat.com>
* billing: marketplace UI
adds UI in billing section for managing user and org-bound skus
add more unit tests for org binding
changed endpoint for bulk attaching skus to orgs
* update reconciliationworker to use webCustomerId instead of
ebsAccountNumber
* fix reconciler where it was incorrectly using the ebsAccountNumber to
create subscriptions
* add job to reconciler so that it reconciles different ids between the
database and the user api
* separate skus to be used by billing and skus to be used by reconciler
* chore: pass config to isort as it doesn't always detect it
* chore: mark package "test" as local, not stdlib
* chore: remove "isort: skip_file"
* chore: fix app in test_load_security_information
* chore: fix app in test_notification
* chore: fix app in test_index_report
* add migration for orgrhskus table
* add endpoints for managing and listing skus bound to an org
* create checks in billing flow to look for org-bound skus
* refactor RH marketplace api objects to be more usable in tests
* update cypress test db data and exclude it from pre-commit hook formatting
Garbage collect manifests no longer referenced in Quay from the
security scanner service.
Also moved quota related code from data/registry_model/ to data/model/
to avoid circular dependencies.
* Aggregate stripe plans and subscriptions.
* Validate end date for subscriptions when fetching from marketplace.
* Check subscription returned from stripe api is non-null value when
finding stripe plan for sku.
Allows superusers to trigger a calculation of the deduplicated registry size. A superuser can go to the organization panel of the superuser page and select Calculate to queue a calculation of the registry total. The total will only be calculated when requested. Includes warning to user of increase of database load when running calculation.
Allow the replication worker to retry immediately without having to
wait and exhaust the queueitem's retries. This addresses transient
unreliable storage service issues.
Allows for only unique blobs are counted at the namespace and repository level. Calculation includes manifest list sizes.
Add's the following internal configurations that default to true:
QUOTA_INVALIDATE_TOTALS: Invalidates calculated totals when FEATURE_QUOTA_MANAGEMENT is set to false
RESET_CHILD_MANIFEST_EXPIRATION: Resets the expiry for child manifests on push of the manifest list for immediate GC eligibility
PERMANENTLY_DELETE_TAGS: Enables features related to the permanent deletion of tags outside the configured time machine window
* repomirror: Use skopeo list-tags to get repo tags
`skopeo inspect foo` returns infromation about the image `foo:latest`,
and repository tags. Quay needs only list of tags, so it should use
`skopeo list-tags` that doesn't fail if `foo:latest` doesn't exist.
* Update type hints
* On local-dev Quay does not provide valid TLS certificate
Currently Quay creates tags for Docker V2 schema 1 manifests in manifest lists. This makes it appear a tag was mirrored successfully when it had actually failed. This change rolls back those failed tags when the sync fails.
Adds the `REPO_MIRROR_ROLLBACK` option to specify whether the mirror will rollback the state of the repo on failure of any one of the tags. Defaults to false. Adds additional `PARTIAL_SYNC` error status which logs the tags that failed to sync to the console.
Previous logic for claiming mirror ("locking") relied on the value
returned from updating the database row. Since this was always being
updated with a new expiration time, it would always succeed, even when
another process had already claimed the same mirror.
Having only one worker index recent manifest doesn't keep up with the
rate new manifests being pushed, given the time it takes for an index
request to complete. Adding the option to bypass the global lock
allows for more workers, but also increase the chance of duplicate work.
Index recent manifests in a separate background process, allowing the
main process to correctly select random slabs from the entire table
set and marking them completed in the allocator (rbtree). This avoids
the worker having to start iterating from the beginning of the table
whenever it is restarted.
During a rollback the mirror worker checks for new tags that were created in the repository in the time that the mirror operation has been running. If it encounters older tags that have been updated it will attempt to create a new tag that will point to the previous manifest. Currently for large lists of tags this will fail since we only retrieve the 100 latest tags. The mirror worker will never reach the tags that have been updated and will never recreate them, leading to the behavior of deleting tags during a rollback.
Add a global lock on security worker iterations, based on the value of
the current pagination token. This is to avoid multilple worker
processes possibly doing work on the same set of manifests.
Currently the CI breaks due to a dependency of black, `click`, breaking with it's latest release with `ImportError: cannot import name '_unicodefun' from 'click'`. Since black does not pin it's version of click it pulls in the latest version containing the breaking change and fails the CI check. This updates black with the patch. [See the original issue here.](https://github.com/psf/black/issues/2964) The rest of the changes are format updates introduced with the latest version of black.
Currently when attempting to mirror a registry containing unsigned images the mirror will fail due to not finding the source signature. This is caused by the updated version of Skopeo blocking unsigned images by default. This allows users to specify the ability to pull unsigned images per-repository. The Skopeo version is also now pinned.
