* registry: implements the OCI 1.1 referrers API
Migrations:
- Adds a subject column for lookup
- Adds a subject_backfilled column to track status of the backfilling
of existing manifests
- Adds a manifest_json column making use of postgres' JSONB support,
for future use.
Manifestsubjectbackfillworker: Indexes existing manifests for possible
existing subject field.
* Deprecate IGNORE_UNKNOWN_MEDIATYPES
* Cleanup
* marketplace: change redhatsubscriptions model to return a list of customer ids (PROJQUAY-7129)
* Use list comprehension in model and expect to return None if no rows are
returned
* fix behavior for reconciler and fix unit test for reconciler
Reducing the number of DB calls in the repo list endpoint with quota enabled by:
- Adding the id to RepositoryBaseElement when the repositories are initially fetched, removing the need to fetch the repository ID's again
- Fetching the repository sizes with a single DB call using the IN operator
* feat: Add support for auto pruning at repository level (PROJQUAY-6354)
* Add repositoryautoprunepolicy table to alembic migration script
* Add repository auto-prune policy endpoints
* Add UI for repository auto-pruning policies
* case: apply repo auto-prune policy when no namespace policy given
* case: both namespace and repo pruning policy are given
* Add tests for repository autoprune endpoint
* Add cypress test for repository auto-prune
* Add repo auto-prune policy clean-up for repository deletion
* Add repository auto pruning tables to quay db snapshot for cypress tests
* Address review comments
* Add more tests + fix CI + reformat files
* Address review comments #2
---------
Signed-off-by: harishsurf <hgovinda@redhat.com>
* added base class for OIDC auth + UI
* adding read-only teams page + display sync config + option to remove team sync
* setting page in read only mode fix
* ui tests
* adding validation for group name input
* fixes based on review + fixing test suite
* add backend tests for externalOIDC
* minor fixes
* Adds handling for when a subscription returned from the subscription watch api has a quantity greater than 1. Number of private repos should be correctly calculated using the quantity.
* Updates ui so that subscriptions can only be added to an org as a group, i.e. a subscription with quantity = 2 cannot be split across organizations.
Previously, we weren't returning the total compressed layer size (in bytes) which is part of the manifest table. This makes it cumbersome to identify the real size of an image via `api/v1/repository/{LOCATION}/manifest/{manifestref}` endpoint since only individual layer sizes were returned and the total would then need to be manually summed over each individual layer. With this change, the API endpoint will return the compressed size of all layers if we're referencing a real image, if a reference points to an OCI index or a manifest list, the returned size will be 0.
* billing: marketplace UI
adds UI in billing section for managing user and org-bound skus
add more unit tests for org binding
changed endpoint for bulk attaching skus to orgs
* update reconciliationworker to use webCustomerId instead of
ebsAccountNumber
* fix reconciler where it was incorrectly using the ebsAccountNumber to
create subscriptions
* add job to reconciler so that it reconciles different ids between the
database and the user api
* separate skus to be used by billing and skus to be used by reconciler
* api: add caching for get_repository method (PROJQUAY-6472)
* fixing formatting
* using pickle to serialize and deserialize objects
* adding model_cache parameter to lookup_repository func for interface and registry_proxy_model
* adding test
* formatting fix
Currently if a new manifest is created that re-uses blobs already existing in the registry but not within the namespace it's possible for that manifest to be blocked if the newly referenced blobs exceed the quota limit. The issue being that the rejection only happens after the manifest has been created and tagged. This change blocks the manifest upload before the creation of the tag and prevents that tag from appearing in the UI and being pullable.
If the manifest upload get's rejected before the tag has been created for a new manifest, a temporary tag is created outside the time machine window so it is immediately available for GC.
* chore: pass config to isort as it doesn't always detect it
* chore: mark package "test" as local, not stdlib
* chore: remove "isort: skip_file"
* chore: fix app in test_load_security_information
* chore: fix app in test_notification
* chore: fix app in test_index_report
* georeplication: Enqueue blobs for replication on manifest list pushes (PROJQUAY-5256)
Previously, blob replication was only queued when normal manifests were pushed to the registry. This created problems with manifest lists who contain child manifests whose blobs were never queued. The end result was a huge discrepancy of storage usage between different defined storage engines.
With this fix, blobs will always be enqueued for replication regardless of the image type.
* Update `black` in `requirements-dev.txt` to match the version we use on Github
* add migration for orgrhskus table
* add endpoints for managing and listing skus bound to an org
* create checks in billing flow to look for org-bound skus
* refactor RH marketplace api objects to be more usable in tests
* update cypress test db data and exclude it from pre-commit hook formatting
Garbage collect manifests no longer referenced in Quay from the
security scanner service.
Also moved quota related code from data/registry_model/ to data/model/
to avoid circular dependencies.
* ui: Add custom TOS support
Enable adding of a custom TOS in the Quay footer for on-premise installations via the optional `TERMS_OF_SERVICE` config flag.
If the flag is not defined, the TOS field will not be visible on Quay rendering. Also changes the behaviour of the discovery endpoint to include custom terms of service instead of pointing to Quay.io terms of service for all installations.
* Fix formatting errors
* Fix TOS page for Quay.io deployments
* Change TOS variable name to TERMS_OF_SERVICE_URL for better clarity
* Removed reference to Quay.io from the discovery endpoint description
Add new parameter `allow_hidden` to `lookup_manifest_by_digest` method and set this to true on the manifest v2 endpoint.
Enables manifests to be pulled by digest, and fixes issues with recent versions of conftest being unable to push to quay.
This change allows validation of proxy cache config api to support anonymous pulls
when the registry returns a 401 since repo name is not known during initial proxy config
Signed-off-by: harishsurf <hgovinda@redhat.com>
* Aggregate stripe plans and subscriptions.
* Validate end date for subscriptions when fetching from marketplace.
* Check subscription returned from stripe api is non-null value when
finding stripe plan for sku.
Allows superusers to trigger a calculation of the deduplicated registry size. A superuser can go to the organization panel of the superuser page and select Calculate to queue a calculation of the registry total. The total will only be calculated when requested. Includes warning to user of increase of database load when running calculation.
