* marketplace: change redhatsubscriptions model to return a list of customer ids (PROJQUAY-7129)
* Use list comprehension in model and expect to return None if no rows are
returned
* fix behavior for reconciler and fix unit test for reconciler
Reducing the number of DB calls in the repo list endpoint with quota enabled by:
- Adding the id to RepositoryBaseElement when the repositories are initially fetched, removing the need to fetch the repository ID's again
- Fetching the repository sizes with a single DB call using the IN operator
* db: use read replica for selected queries (PROJQUAY-6397)
We add a new param `can_use_read_replica` to the `select`
query. This allows us to choose which queries we want to
send to the read replica. This is useful in cases where
the read replica lags behind the primary and some queries
need the latest data
* feat: Add support for auto pruning at repository level (PROJQUAY-6354)
* Add repositoryautoprunepolicy table to alembic migration script
* Add repository auto-prune policy endpoints
* Add UI for repository auto-pruning policies
* case: apply repo auto-prune policy when no namespace policy given
* case: both namespace and repo pruning policy are given
* Add tests for repository autoprune endpoint
* Add cypress test for repository auto-prune
* Add repo auto-prune policy clean-up for repository deletion
* Add repository auto pruning tables to quay db snapshot for cypress tests
* Address review comments
* Add more tests + fix CI + reformat files
* Address review comments #2
---------
Signed-off-by: harishsurf <hgovinda@redhat.com>
* Adds handling for when a subscription returned from the subscription watch api has a quantity greater than 1. Number of private repos should be correctly calculated using the quantity.
* Updates ui so that subscriptions can only be added to an org as a group, i.e. a subscription with quantity = 2 cannot be split across organizations.
* update reconciliationworker to use webCustomerId instead of
ebsAccountNumber
* fix reconciler where it was incorrectly using the ebsAccountNumber to
create subscriptions
* add job to reconciler so that it reconciles different ids between the
database and the user api
* separate skus to be used by billing and skus to be used by reconciler
Currently if a new manifest is created that re-uses blobs already existing in the registry but not within the namespace it's possible for that manifest to be blocked if the newly referenced blobs exceed the quota limit. The issue being that the rejection only happens after the manifest has been created and tagged. This change blocks the manifest upload before the creation of the tag and prevents that tag from appearing in the UI and being pullable.
If the manifest upload get's rejected before the tag has been created for a new manifest, a temporary tag is created outside the time machine window so it is immediately available for GC.
* add Postgresql client certificate authentication option
* user(robots): disallow robot login and create PROJQUAY-5968
Add a config flag `ROBOTS_DISALLOW` to turn off Robot login and creation
in Quay for all accounts.
with the Flag set existing Robots cannot login anymore
```
$ podman login -u milang+test2 -p W7B...HQA quay.example.com
Error: logging into "quay.example.com": invalid username/password
```
This behavior will be reported in the logs accordingly
```
gunicorn-registry stdout | 2023-08-24 19:08:01,907 [253] [WARNING] [auth.credentials] Failed to validate credentials for robot milang+test2: Robot account has been disabled. Please contact your administrator.
```
as well as in the UI when creating a Robot Account which will display
the `Robot account has been disabled. Please contact your
administrator.` message.
* Revert "add Postgresql client certificate authentication option"
need to branch for PR (stupid me)
This reverts commit f8f54eead2.
* fixed error message to proper english and added schema definition for the option
* add migration for orgrhskus table
* add endpoints for managing and listing skus bound to an org
* create checks in billing flow to look for org-bound skus
* refactor RH marketplace api objects to be more usable in tests
* update cypress test db data and exclude it from pre-commit hook formatting
* Pagination: Fixing paginate for /api/v1/superuser/logs API
* using offset to fetch next page items
* adding datetime parsing
* using black to format code
* removing use_offset
* Removing redundant declaration
Garbage collect manifests no longer referenced in Quay from the
security scanner service.
Also moved quota related code from data/registry_model/ to data/model/
to avoid circular dependencies.
* api: allow robot token creation with a pre-defined token (PROJQUAY-5414)
This is for usecases where we want to explicitly create a robot token
with a pre-defined token in case of migration events when we migrate
from one quay to another, we can re-use the same robot token to
avoid resetting it in all places it is used
Adds the following changes:
- Allows remove_tag_from_timemachine to expire tags even if the time machine window is set to 0, immediately marking them for deletion. This allows the quota proxy pruner to expire tags with the same method call. This wasn't required for normal push/pulls as the user would just call the DELETE /tag endpoint.
- Remove hidden = true when expiring tags. For proxy Quay will attempt to lookup the tag referenced by the manifest in order to extend it's lifetime_end_ms. Hiding this tag prevents that logic from running correctly.
Add new parameter `allow_hidden` to `lookup_manifest_by_digest` method and set this to true on the manifest v2 endpoint.
Enables manifests to be pulled by digest, and fixes issues with recent versions of conftest being unable to push to quay.
Moves the resetting of child manifest temporary tags to happen on deletion instead of on push/pull. Resetting child manifest temporary tags caused issues in other portions of the code like proxy cache where temporary tags were deleted too early.
Allows for only unique blobs are counted at the namespace and repository level. Calculation includes manifest list sizes.
Add's the following internal configurations that default to true:
QUOTA_INVALIDATE_TOTALS: Invalidates calculated totals when FEATURE_QUOTA_MANAGEMENT is set to false
RESET_CHILD_MANIFEST_EXPIRATION: Resets the expiry for child manifests on push of the manifest list for immediate GC eligibility
PERMANENTLY_DELETE_TAGS: Enables features related to the permanent deletion of tags outside the configured time machine window
* API/UI: Filtering of tags API through query parameter (PROJQUAY-5362)
* Changing syntax of query param to add operation + added propagation of filtering from new UI
* added exception to return 400 on incorrect syntax
* Added tests to test filtering of /tags endpoint
* Minor fixes
* api: feature to limit org creation to superusers
Introduces the following configuration flags:
FEATURE_SUPERUSERS_ORG_CREATION_ONLY:
Limit org creation to superusers only
FEATURE_SUPERUSERS_FULL_ACCESS:
Grant superusers read/write access to registry content in all namespaces
FEATURE_RESTRICTED_USERS:
Users considered as restricted are not anle to create organization
RESTRICTED_USERS_WHITELIST:
Whitelist for FEATURE_RESTRICTED_USERS
RESTRICTED_USER_INCLUDE_ROBOTS:
Whether or not to include the user namespace's robots
RESTRICTED_USER_READ_ONLY:
Only allow read-only operations for restricted users
* Revert superuser repositorylist endpoint
- The database health check is currently not using the db_kwargs and not using ssl settings
- This is causing health check failures for MySQL behind SSL
Previous logic for claiming mirror ("locking") relied on the value
returned from updating the database row. Since this was always being
updated with a new expiration time, it would always succeed, even when
another process had already claimed the same mirror.
Method for calculating size of repo joined on repo id instead of manifest id. This causes deleted manifests to be counted. Change updates call to join on manifest id.
introduces a check of the OrganizationMemberPermission for pulls
against a proxy org.
if the proxy cache feature is disabled, or the org is not a proxy org
the check is not performed and Quay will behave normally.
this check does not mean pulls will work transparently though -
non-admin users need to be added to a team in the proxy org with the
member role, and default read and write permissions need to be given to
that team so that non-admin users can pull and update the cache for
images they do not own (the user who first pulls an image ends up
owning the repository since that is when the repo gets created).
