fixing CVE-2025-4374 by extending the create_repository method to understand if we are requesting a proxy_cache repository added unittests for create_repository when proxy_cache.
Co-authored-by: Michaela Lang <milang@redhat.com>
* Revert "bug: Adding allow hidden flag while looking up for manifests (PROJQUAY-8536) (#3722)"
This reverts commit f0c153fab5.
* Revert "proxy: moving manifest check to after upstream manifest fetch (PROJQUAY-8536) (#3814)"
This reverts commit 944edd064b.
---------
Co-authored-by: bcaton <bcaton@redhat.com>
* Adding allow hidden flag while looking up for manifests
* added test to validate allow_hidden tag
* minor addition to fix failing test
* testing
* fixed test
---------
Co-authored-by: shudeshp <shudeshp@redhat.com>
Co-authored-by: Brandon Caton <bcaton@redhat.com>
* registry: implements the OCI 1.1 referrers API
Migrations:
- Adds a subject column for lookup
- Adds a subject_backfilled column to track status of the backfilling
of existing manifests
- Adds a manifest_json column making use of postgres' JSONB support,
for future use.
Manifestsubjectbackfillworker: Indexes existing manifests for possible
existing subject field.
* Deprecate IGNORE_UNKNOWN_MEDIATYPES
* Cleanup
* api: add caching for get_repository method (PROJQUAY-6472)
* fixing formatting
* using pickle to serialize and deserialize objects
* adding model_cache parameter to lookup_repository func for interface and registry_proxy_model
* adding test
* formatting fix
Currently if a new manifest is created that re-uses blobs already existing in the registry but not within the namespace it's possible for that manifest to be blocked if the newly referenced blobs exceed the quota limit. The issue being that the rejection only happens after the manifest has been created and tagged. This change blocks the manifest upload before the creation of the tag and prevents that tag from appearing in the UI and being pullable.
If the manifest upload get's rejected before the tag has been created for a new manifest, a temporary tag is created outside the time machine window so it is immediately available for GC.
Garbage collect manifests no longer referenced in Quay from the
security scanner service.
Also moved quota related code from data/registry_model/ to data/model/
to avoid circular dependencies.
(PROJQUAY-5600)
Catch exception thrown when looking up tag `lifetime_start_ms` and set
`created_at` property for a manifest to None.
If `created_at` is None, don't calculate indexing SLI for manifest.
Add new parameter `allow_hidden` to `lookup_manifest_by_digest` method and set this to true on the manifest v2 endpoint.
Enables manifests to be pulled by digest, and fixes issues with recent versions of conftest being unable to push to quay.
Allows superusers to trigger a calculation of the deduplicated registry size. A superuser can go to the organization panel of the superuser page and select Calculate to queue a calculation of the registry total. The total will only be calculated when requested. Includes warning to user of increase of database load when running calculation.
Moves the resetting of child manifest temporary tags to happen on deletion instead of on push/pull. Resetting child manifest temporary tags caused issues in other portions of the code like proxy cache where temporary tags were deleted too early.
Allows for only unique blobs are counted at the namespace and repository level. Calculation includes manifest list sizes.
Add's the following internal configurations that default to true:
QUOTA_INVALIDATE_TOTALS: Invalidates calculated totals when FEATURE_QUOTA_MANAGEMENT is set to false
RESET_CHILD_MANIFEST_EXPIRATION: Resets the expiry for child manifests on push of the manifest list for immediate GC eligibility
PERMANENTLY_DELETE_TAGS: Enables features related to the permanent deletion of tags outside the configured time machine window
* API/UI: Filtering of tags API through query parameter (PROJQUAY-5362)
* Changing syntax of query param to add operation + added propagation of filtering from new UI
* added exception to return 400 on incorrect syntax
* Added tests to test filtering of /tags endpoint
* Minor fixes
The distribution spec does not require the docker-content-digest header
to be set in response to a manifest GET/HEAD request.
This changes both the proxy client and the registry proxy model to
correctly check whether a manifest is up-to-date with the upstream
registry or not when no digest header is received.
NOTE: when checking staleness against registries that do not return the
docker-content-digest header, Quay will make a GET request to the
registry and calculate the digest from the manifest itself. GET requests
usually count towards rate-limiting.
This change also sets the accept-encoding header to 'identity'. The python
requests library seems to automatically set the accept-encoding header to
'gzip'. Dockerhub ignores that header when serving blobs, but some
registries don't (namely registry.access.redhat.com). When Quay receives a
gzipped config blob (have not tested non-config blobs) for some reason it
doesn't know how to handle it. I suspect it has to do wit the fact that in
this case the content-length header will differ from the actual size of
the response body, so when Quay tries to upload the blob it cannot
correctly calculate the actual blob size, so it does a partial upload to
its object storage, which then results in a digest mismatch error
(BlobDigestMismatchException).
Currently the CI breaks due to a dependency of black, `click`, breaking with it's latest release with `ImportError: cannot import name '_unicodefun' from 'click'`. Since black does not pin it's version of click it pulls in the latest version containing the breaking change and fails the CI check. This updates black with the patch. [See the original issue here.](https://github.com/psf/black/issues/2964) The rest of the changes are format updates introduced with the latest version of black.
introduces the possibility to pull images from external registries
through Quay, storing them locally for faster subsequent pulls.
Closes PROJQUAY-3030 and PROJQUAY-3033
* Add dev dependencies mypy and typing
* Add makefile target `types-test`, not yet included in `test` target.
* Generate stubs for imported modules to avoid mypy complaining about missing types.
* Remove generated stubs as there are way too many and they cause tons of mess in the repo. Switched to ignoring untyped modules for now, to concentrate on Quay-only type checking.
* mypy config changed to ignore missing imports
* ignore property decorator as it is not supported by mypy
* mypy annotations for many configuration variables
* re-generate mypy_stubs directory as its necessary in some classes for base classes to prevent mypy errors
* util/registry/queuefile referred to non existent definition of Empty class in multiprocessing.queues
* ignore type checking for things like monkey patching and exported/re-imported objects that
mypy does not allow.
* Adjust mypy config to warn us about unreachable return paths and useless expressions.
* Add the __annotations__ property to INTERNAL_ONLY_PROPERTIES so that it is not part of the config schema testing
* Remove redundant dependencies `typing` and `typing-extensions` which are NOOP after Python 3.5
* Remove mypy-extensions which only provides a TypedDict implementation but has not been updated since 2019.
* updated mypy to 0.910 which requires all types packages to be installed manually.
* exclude local-dev from type checking until core team can suggest an outcome for __init__.py duplicate packages
* re-add typing dependency which will be needed until Python 3.9
* ignore .mypy_cache
* add mypy stub for features module to replace inline definitions
* import annotations eager evaluation in billing.py as it was required to reference a class declared later in the module.
* remove the type definition of V1ProtocolSteps/V2ProtocolSteps to make tox happy
