Garbage collect manifests no longer referenced in Quay from the
security scanner service.
Also moved quota related code from data/registry_model/ to data/model/
to avoid circular dependencies.
* config: Set feature flag default for new vulnerability notifications to True
(PROJQUAY-4659)
Change the default from `False` to `True` for
`FEATURE_SECURITY_SCANNING_NOTIFY_ON_NEW_INDEX`.
Since this flag addresses a bug, it should be enabled by default.
* add mock return values for unit tests
* ui: Add custom TOS support
Enable adding of a custom TOS in the Quay footer for on-premise installations via the optional `TERMS_OF_SERVICE` config flag.
If the flag is not defined, the TOS field will not be visible on Quay rendering. Also changes the behaviour of the discovery endpoint to include custom terms of service instead of pointing to Quay.io terms of service for all installations.
* Fix formatting errors
* Fix TOS page for Quay.io deployments
* Change TOS variable name to TERMS_OF_SERVICE_URL for better clarity
* Removed reference to Quay.io from the discovery endpoint description
Updating the options introduced by quota.
QUOTA_TOTAL_DELAY_SECONDS: increasing to accommodate most deployments
QUOTA_BACKFILL: Should run when quota is enabled to prevent having to do two deployments
RESET_CHILD_MANIFEST_EXPIRATION: New option should be opt in
PERMANENTLY_DELETE_TAGS: New option should be opt-in
Allows for only unique blobs are counted at the namespace and repository level. Calculation includes manifest list sizes.
Add's the following internal configurations that default to true:
QUOTA_INVALIDATE_TOTALS: Invalidates calculated totals when FEATURE_QUOTA_MANAGEMENT is set to false
RESET_CHILD_MANIFEST_EXPIRATION: Resets the expiry for child manifests on push of the manifest list for immediate GC eligibility
PERMANENTLY_DELETE_TAGS: Enables features related to the permanent deletion of tags outside the configured time machine window
If quay doesn't clean up temporary files under `uploads` folder by default, that means it is very possible for customer to pay more money for storage usage. A lot of important customers are money-sensitive, so it doesn't make sense to set CLEAN_BLOB_UPLOAD_FOLDER as false by default.
- Similar to LDAP_SUPERUSER_FILTER, add a specific filter to define
restricted users, based on the LDAP_USER_FILTER
- restrict writes on restricted users' own namespace. Normal
permissions applies on organization membership
- add global readonly superuser GLOBAL_READONLY_SUPER_USERS (PROJQUAY-2604)
- Removes RESTRICTED_USER_INCLUDE_ROBOTS, FEATURE_RESTRICTED_READ_ONLY_USERS
* Config: Updating Cosign SBOM Media Types on Quay
* Added Layer types for SBOM and mime types for hacbs
* Minor ordering
* Adding syft mime type
* Removing mimetypes for HACBS
* api: feature to limit org creation to superusers
Introduces the following configuration flags:
FEATURE_SUPERUSERS_ORG_CREATION_ONLY:
Limit org creation to superusers only
FEATURE_SUPERUSERS_FULL_ACCESS:
Grant superusers read/write access to registry content in all namespaces
FEATURE_RESTRICTED_USERS:
Users considered as restricted are not anle to create organization
RESTRICTED_USERS_WHITELIST:
Whitelist for FEATURE_RESTRICTED_USERS
RESTRICTED_USER_INCLUDE_ROBOTS:
Whether or not to include the user namespace's robots
RESTRICTED_USER_READ_ONLY:
Only allow read-only operations for restricted users
* Revert superuser repositorylist endpoint
Removes read support for Clair V2, along with the need to package
jwtproxy with Quay.
TODO: Drop deprecate image api + image table, remove image data model.
introduces the possibility to pull images from external registries
through Quay, storing them locally for faster subsequent pulls.
Closes PROJQUAY-3030 and PROJQUAY-3033
* initial commit
* fixing some bugs
* create quota management
Fix json request json type
Creation of quota is working
All quota crud operations
crud for quota limits
repository size reporting
adding registry model
error levels
namespacequota
remove holdover from user file
finalizing refactor to namespace over organization
finalization of functionality
fixing formatting to match with black style
missed some files in formatting
fixing access to attribute
add single test to verify its working
fix some bugs and add defensive catching
bug fixes and code resiliency
Bug fixes and making quota limits detect properly where necessary
remove transitive delete and other bug fixes
fix formatting and trasnitive deletion issues
fix repositorysize does not exist error
fix not nul constraint and add security tests
fix security tests and bug
more security test fixes
reorder security tests
put docker file back and adjust security testing
security tests reduced
Missed changes for status 200
missed additional 201 responses getting 200
security bypass for now
Another tweak to security testing
forgot 1 endpoint
bug fix for parsing dictionary
remove unnecessary check at blob head
add initdb for quota
Incorrect syntax repair
mysql only supports decimal
adding quota specific notifications
optimization
add permission checks
adjust security and add configuration parameter
fix security test for new security levels
Fix logic errors and improve caching
fix logic issue and error reporting
adjust things according to PR comments
fix refactor left overs
miscapitilazation
missed refactor location
refactor code to remove quota limit groupings
fix refactor errors
remove transitive deletion
fix transitive deletes
Transitive deletion work
Transitive deletion work
refactor registry model and remove it
place api behind feature flag
patch feature enabledment for tests
patch feature enabledment for tests
testing to see if the config is the problem
remove patch
fix new org bug
fixing notifications
mismatched parameters
fix org not exists
fixed paramter mismatch
fix nonetype access
fix nonetype access
new tables created user deletion issues
new tables created user deletion issues
parameter mismatch
fix transitive delete
fix model access error
record does not exist missing catch
fix quota deletion to always delete limits
quotalimits deletion on quota deletion
mistake
fix quota limits deletion
patch tests and disable feature
typo
switch to toggle feature
add feature patch to top of file
change testconfigpy
* change permissions
* adjust permissions
* change config access
* fix formatting
* gether feature information differently
* duplicate function name
* fix config name
* type conversion
* config adjustments
* incorrect keyword
* Update security api tests
* duplicate naming
* fix config schema
* revert files and fix error
* QuotaManagement: UI (PROJQUAY-2936) (#1)
* [WIP]: Quota Reporting on Quay UI
* Integrating quota reporting UI with backend
* Humanizing bytes on UI
* Quota Reporting UI on repo table view
* Taking pull and updating code
* Adding quota management view
* Added support for CRUD operations for org quota
* create quota management
Fix json request json type
Creation of quota is working
All quota crud operations
crud for quota limits
repository size reporting
adding registry model
error levels
namespacequota
remove holdover from user file
finalizing refactor to namespace over organization
finalization of functionality
fixing formatting to match with black style
missed some files in formatting
fixing access to attribute
add single test to verify its working
fix some bugs and add defensive catching
bug fixes and code resiliency
Bug fixes and making quota limits detect properly where necessary
remove transitive delete and other bug fixes
fix formatting and trasnitive deletion issues
fix repositorysize does not exist error
fix not nul constraint and add security tests
fix security tests and bug
more security test fixes
reorder security tests
put docker file back and adjust security testing
security tests reduced
Missed changes for status 200
missed additional 201 responses getting 200
security bypass for now
Another tweak to security testing
forgot 1 endpoint
bug fix for parsing dictionary
remove unnecessary check at blob head
add initdb for quota
Incorrect syntax repair
mysql only supports decimal
adding quota specific notifications
optimization
add permission checks
adjust security and add configuration parameter
fix security test for new security levels
Fix logic errors and improve caching
fix logic issue and error reporting
adjust things according to PR comments
fix refactor left overs
miscapitilazation
missed refactor location
refactor code to remove quota limit groupings
fix refactor errors
remove transitive deletion
fix transitive deletes
Transitive deletion work
Transitive deletion work
refactor registry model and remove it
place api behind feature flag
patch feature enabledment for tests
patch feature enabledment for tests
testing to see if the config is the problem
remove patch
fix new org bug
fixing notifications
mismatched parameters
fix org not exists
fixed paramter mismatch
fix nonetype access
fix nonetype access
new tables created user deletion issues
new tables created user deletion issues
parameter mismatch
fix transitive delete
fix model access error
record does not exist missing catch
fix quota deletion to always delete limits
quotalimits deletion on quota deletion
mistake
fix quota limits deletion
patch tests and disable feature
typo
switch to toggle feature
add feature patch to top of file
change testconfigpy
* Removing quota and state conf from repo-list and user-view
* Removing quota and state conf form app list page
* Removing quota conf from repo-list.html
* minor fixes
* Added Quota Repoting and configuring quota from UI
* Making quota configuration component reusable + added support to read bytes via KB, MB, etc + Added reporting for total org consumption + Added org consumption for super user panel + Added quota configurable support on super user panel
* Adding older quota management component
* Removing not reusable quota management component
* Adding % consumption for repo quotas
* Adding % consumption for organization level quota
* Adding check to verify request.args
* Removing todo
* Adding default 0 to quota
* Formatting with black
* Fixing params for tests
* Formatting test file
Co-authored-by: Keith Westphal <kwestpha@redhat.com>
* remove migration
* add migration back
* repair formatting
* QuotaManagement: Moving the logic for bytes conversion to human friendly units to the frontend (PROJQUAY-2936) (#3)
* Moving the logic for bytes conversion to human friendly units to the frontend
* Reading updates from quota_limit_id
* Formatting using black
* remote unused function
* Adding quota configuring on super user panel (#4)
* Converting quota bytes to human friendly format (#5)
* PR refactors
* invalid reference
* bad return value
* fix bad reference
* bad reference
* fix tests
* Quota Config: UI improvements (#6)
* Quota UI Improvements
* Rendering table for quota limit config
* Removing proxy cache files
* Disabling quota config for org view
* Removing redundant get
* Fixing PR requests
* repair formatting
Co-authored-by: Sunandadadi <Sunandadadi@users.noreply.github.com>
* Revert "database: handle nested transaaction when trying to close before transaction (PROJQUAY-3303) (#1157)"
This reverts commit 24b3c15334.
* Revert "database: force close existing non-pooled connections before transaction (PROJQUAY-3303) (#1153)"
This reverts commit 7cdb88b578.
* Revert "database: force close existing non-pooled connections before transaction (PROJQUAY-3303) (#1153)"
This reverts commit 7cdb88b578.
* database: handle nested transaaction when trying to close before transaction (PROJQUAY-3303)
Handle nested transaction when trying to catch InterfaceError from
pymsql/mysql's idle connections. If the connection is already in a
transaction, then we'll assume it doesn't need to be closed
beforehand, as it's probably in-use, and unlikely to be cleaned up.
This hack is for handling possible MySQL closing idle
connections. Peewee/pymysql's will try to reuse the existing
connection after the MySQL server has already closed it, which will
return an InterfaceError. AFAIK, there isn't a way to actually test
for a a stale connection without actually running a query. Closing the
connection before the transaction forces peewee/pymysql to reopen a
new session to MySQL. This should only applies to non-registry
workers, as the registry workers use a pool by default, and shouldn't
have this issue. Follow-up to 58b06572.
Currently blobs leftover in the uploads directory during cancelled uploads do not get cleaned up since they are no longer tracked. This change cleans up the uploads storage directory directly.
