* storage: Disable pushes on registry (PROJQUAY-6870)
The current read-only option for Quay is not sometimes feasible, since it requires an insert of the service key and other manual config changes. For instance, if you want to just recalculate quota on the registry, but would like to allow all registry operations (including UI) without the possibility of pushes until recalculation is done, setting the whole registry `read-only` cannot be done since it makes the database read only as well.
This PR introduces a new flag called `DISABLE_PUSHES` which allows all registry operations to continue (changing tags, repo editing, robot account creation/deletion, user creation etc.) but will disable pushes of new images to the registry (i.e. backend storage will not change). If a registry already contains the image and a new tag is simply being added, that operation should succeed.
The following message would appear in the logs:
~~~
gunicorn-registry stdout | 2024-03-13 20:19:49,414 [369] [DEBUG] [endpoints.v2] sending response: b'{"errors":[{"code":"METHOD NOT ALLOWED","detail":{},"message":"Pushes to the registry are currently disabled. Please contact the administrator for more information."}]}\n'
gunicorn-registry stdout | 2024-03-13 20:19:49,414 [369] [INFO] [gunicorn.access] 172.17.0.1 - - [13/Mar/2024:20:19:49 +0000] "PUT /v2/ibazulic/mariadb/manifests/sha256:c4694ba424e0259694a5117bbb510d67340051f0bdb7f9fa8033941a2d66e53e HTTP/1.1" 405 169 "-" "skopeo/1.9.3"
nginx stdout | 172.17.0.1 (-) - - [13/Mar/2024:20:19:49 +0000] "PUT /v2/ibazulic/mariadb/manifests/sha256:c4694ba424e0259694a5117bbb510d67340051f0bdb7f9fa8033941a2d66e53e HTTP/1.1" 405 169 "-" "skopeo/1.9.3" (0.002 3813 0.002)
~~~
The flag defaults to `False` (pushes enabled), unless set otherwise.
* Removed constraint on storage replication when pushes are disabled
* Rebase
* Fix isort sorting
* Fix isort sorting #2
* Removed constraint on storage replication when pushes are disabled
* Rebase
* Remove constraint on storage replication worker
* Fix linting on config.py
* feat: Add support for auto pruning at repository level (PROJQUAY-6354)
* Add repositoryautoprunepolicy table to alembic migration script
* Add repository auto-prune policy endpoints
* Add UI for repository auto-pruning policies
* case: apply repo auto-prune policy when no namespace policy given
* case: both namespace and repo pruning policy are given
* Add tests for repository autoprune endpoint
* Add cypress test for repository auto-prune
* Add repo auto-prune policy clean-up for repository deletion
* Add repository auto pruning tables to quay db snapshot for cypress tests
* Address review comments
* Add more tests + fix CI + reformat files
* Address review comments #2
---------
Signed-off-by: harishsurf <hgovinda@redhat.com>
* add migration for orgrhskus table
* add endpoints for managing and listing skus bound to an org
* create checks in billing flow to look for org-bound skus
* refactor RH marketplace api objects to be more usable in tests
* update cypress test db data and exclude it from pre-commit hook formatting
Previous logic for claiming mirror ("locking") relied on the value
returned from updating the database row. Since this was always being
updated with a new expiration time, it would always succeed, even when
another process had already claimed the same mirror.
* api: update the quota api so that it's more consistent with the other apis (PROJQUAY-2936)
- Uodate the quota api to be more consistent with the rest of the
endpoints
- Handles some uncaught exceptions, such as division by zero
- Update some of the quota data models used by the api to take object
references instead of names to make it easier to use
- Update table model naming conventions
- swagger operationid multiple nicknames
- Added more test cases for api
- Remove unused functions
- Update the UI for better UX, based on the api changes made
* quota: fix ui input form value
* quota: join quota type query
* Remove unused functions
introduces the possibility to pull images from external registries
through Quay, storing them locally for faster subsequent pulls.
Closes PROJQUAY-3030 and PROJQUAY-3033
* initial commit
* fixing some bugs
* create quota management
Fix json request json type
Creation of quota is working
All quota crud operations
crud for quota limits
repository size reporting
adding registry model
error levels
namespacequota
remove holdover from user file
finalizing refactor to namespace over organization
finalization of functionality
fixing formatting to match with black style
missed some files in formatting
fixing access to attribute
add single test to verify its working
fix some bugs and add defensive catching
bug fixes and code resiliency
Bug fixes and making quota limits detect properly where necessary
remove transitive delete and other bug fixes
fix formatting and trasnitive deletion issues
fix repositorysize does not exist error
fix not nul constraint and add security tests
fix security tests and bug
more security test fixes
reorder security tests
put docker file back and adjust security testing
security tests reduced
Missed changes for status 200
missed additional 201 responses getting 200
security bypass for now
Another tweak to security testing
forgot 1 endpoint
bug fix for parsing dictionary
remove unnecessary check at blob head
add initdb for quota
Incorrect syntax repair
mysql only supports decimal
adding quota specific notifications
optimization
add permission checks
adjust security and add configuration parameter
fix security test for new security levels
Fix logic errors and improve caching
fix logic issue and error reporting
adjust things according to PR comments
fix refactor left overs
miscapitilazation
missed refactor location
refactor code to remove quota limit groupings
fix refactor errors
remove transitive deletion
fix transitive deletes
Transitive deletion work
Transitive deletion work
refactor registry model and remove it
place api behind feature flag
patch feature enabledment for tests
patch feature enabledment for tests
testing to see if the config is the problem
remove patch
fix new org bug
fixing notifications
mismatched parameters
fix org not exists
fixed paramter mismatch
fix nonetype access
fix nonetype access
new tables created user deletion issues
new tables created user deletion issues
parameter mismatch
fix transitive delete
fix model access error
record does not exist missing catch
fix quota deletion to always delete limits
quotalimits deletion on quota deletion
mistake
fix quota limits deletion
patch tests and disable feature
typo
switch to toggle feature
add feature patch to top of file
change testconfigpy
* change permissions
* adjust permissions
* change config access
* fix formatting
* gether feature information differently
* duplicate function name
* fix config name
* type conversion
* config adjustments
* incorrect keyword
* Update security api tests
* duplicate naming
* fix config schema
* revert files and fix error
* QuotaManagement: UI (PROJQUAY-2936) (#1)
* [WIP]: Quota Reporting on Quay UI
* Integrating quota reporting UI with backend
* Humanizing bytes on UI
* Quota Reporting UI on repo table view
* Taking pull and updating code
* Adding quota management view
* Added support for CRUD operations for org quota
* create quota management
Fix json request json type
Creation of quota is working
All quota crud operations
crud for quota limits
repository size reporting
adding registry model
error levels
namespacequota
remove holdover from user file
finalizing refactor to namespace over organization
finalization of functionality
fixing formatting to match with black style
missed some files in formatting
fixing access to attribute
add single test to verify its working
fix some bugs and add defensive catching
bug fixes and code resiliency
Bug fixes and making quota limits detect properly where necessary
remove transitive delete and other bug fixes
fix formatting and trasnitive deletion issues
fix repositorysize does not exist error
fix not nul constraint and add security tests
fix security tests and bug
more security test fixes
reorder security tests
put docker file back and adjust security testing
security tests reduced
Missed changes for status 200
missed additional 201 responses getting 200
security bypass for now
Another tweak to security testing
forgot 1 endpoint
bug fix for parsing dictionary
remove unnecessary check at blob head
add initdb for quota
Incorrect syntax repair
mysql only supports decimal
adding quota specific notifications
optimization
add permission checks
adjust security and add configuration parameter
fix security test for new security levels
Fix logic errors and improve caching
fix logic issue and error reporting
adjust things according to PR comments
fix refactor left overs
miscapitilazation
missed refactor location
refactor code to remove quota limit groupings
fix refactor errors
remove transitive deletion
fix transitive deletes
Transitive deletion work
Transitive deletion work
refactor registry model and remove it
place api behind feature flag
patch feature enabledment for tests
patch feature enabledment for tests
testing to see if the config is the problem
remove patch
fix new org bug
fixing notifications
mismatched parameters
fix org not exists
fixed paramter mismatch
fix nonetype access
fix nonetype access
new tables created user deletion issues
new tables created user deletion issues
parameter mismatch
fix transitive delete
fix model access error
record does not exist missing catch
fix quota deletion to always delete limits
quotalimits deletion on quota deletion
mistake
fix quota limits deletion
patch tests and disable feature
typo
switch to toggle feature
add feature patch to top of file
change testconfigpy
* Removing quota and state conf from repo-list and user-view
* Removing quota and state conf form app list page
* Removing quota conf from repo-list.html
* minor fixes
* Added Quota Repoting and configuring quota from UI
* Making quota configuration component reusable + added support to read bytes via KB, MB, etc + Added reporting for total org consumption + Added org consumption for super user panel + Added quota configurable support on super user panel
* Adding older quota management component
* Removing not reusable quota management component
* Adding % consumption for repo quotas
* Adding % consumption for organization level quota
* Adding check to verify request.args
* Removing todo
* Adding default 0 to quota
* Formatting with black
* Fixing params for tests
* Formatting test file
Co-authored-by: Keith Westphal <kwestpha@redhat.com>
* remove migration
* add migration back
* repair formatting
* QuotaManagement: Moving the logic for bytes conversion to human friendly units to the frontend (PROJQUAY-2936) (#3)
* Moving the logic for bytes conversion to human friendly units to the frontend
* Reading updates from quota_limit_id
* Formatting using black
* remote unused function
* Adding quota configuring on super user panel (#4)
* Converting quota bytes to human friendly format (#5)
* PR refactors
* invalid reference
* bad return value
* fix bad reference
* bad reference
* fix tests
* Quota Config: UI improvements (#6)
* Quota UI Improvements
* Rendering table for quota limit config
* Removing proxy cache files
* Disabling quota config for org view
* Removing redundant get
* Fixing PR requests
* repair formatting
Co-authored-by: Sunandadadi <Sunandadadi@users.noreply.github.com>
* Remove V3_UPGRADE_MODE
* Remove tag backfill worker and all callers to tag backfill and upgrades
* Change initdb to create all test data via the manifest builder, rather than manually via legacy images
* Convert various code paths to use the registry_model where we previously did not do so
* Convert the GC test suite to build via the manifest builder
* Delete an old, unused tool
* Delete the Pre OCI registry model
* Add additional error handling to the manifest creation code path
* Add additional error handling to the OCI tag creation code path
* Change how we respond to invalid manifest content types to better handle unknowns
* Change legacy secscan test suite to use the registry model
* Change the repo build badge to use the registry model (also fixes a bug)
* Delete now-unused data model code
* Remove old model adjustment code from OCI model
* Mark older data models as deprecated which will prevent new rows from being inserted
* Remove references to old registry test suite from various testing files
* Remove tag backfill worker (again; got re-added during rebase)
* Move all deprecated model checks into a central function
* Make data_migration more Pythonic
* Small requested fixes to Tag module styling
* Have tag backfill worker fail to migrate if there are TagManifest's
Since this backfill should only be called in future releases for empty models, this should catch someone attempting to upgrade from a too-old version
* Remove labelbackfillworker as it is no longer needed
* Remove unused invalid import
* Reimplement the tag test for the remaining method used
