* api: Removing parsing of datetime (PROJQUAY-5360)
* added ignoretz flag to date util parser
* added removed extra line
---------
Co-authored-by: Sunandadadi <sunanda.3094@gmail.com>
* Pagination: Fixing paginate for /api/v1/superuser/logs API
* using offset to fetch next page items
* adding datetime parsing
* using black to format code
* removing use_offset
* Removing redundant declaration
---------
Co-authored-by: Sunandadadi <sunanda.3094@gmail.com>
Adds the following changes:
- Allows remove_tag_from_timemachine to expire tags even if the time machine window is set to 0, immediately marking them for deletion. This allows the quota proxy pruner to expire tags with the same method call. This wasn't required for normal push/pulls as the user would just call the DELETE /tag endpoint.
- Remove hidden = true when expiring tags. For proxy Quay will attempt to lookup the tag referenced by the manifest in order to extend it's lifetime_end_ms. Hiding this tag prevents that logic from running correctly.
Add new parameter `allow_hidden` to `lookup_manifest_by_digest` method and set this to true on the manifest v2 endpoint.
Enables manifests to be pulled by digest, and fixes issues with recent versions of conftest being unable to push to quay.
Moves the resetting of child manifest temporary tags to happen on deletion instead of on push/pull. Resetting child manifest temporary tags caused issues in other portions of the code like proxy cache where temporary tags were deleted too early.
Allows for only unique blobs are counted at the namespace and repository level. Calculation includes manifest list sizes.
Add's the following internal configurations that default to true:
QUOTA_INVALIDATE_TOTALS: Invalidates calculated totals when FEATURE_QUOTA_MANAGEMENT is set to false
RESET_CHILD_MANIFEST_EXPIRATION: Resets the expiry for child manifests on push of the manifest list for immediate GC eligibility
PERMANENTLY_DELETE_TAGS: Enables features related to the permanent deletion of tags outside the configured time machine window
* API/UI: Filtering of tags API through query parameter (PROJQUAY-5362)
* Changing syntax of query param to add operation + added propagation of filtering from new UI
* added exception to return 400 on incorrect syntax
* Added tests to test filtering of /tags endpoint
* Minor fixes
* api: feature to limit org creation to superusers
Introduces the following configuration flags:
FEATURE_SUPERUSERS_ORG_CREATION_ONLY:
Limit org creation to superusers only
FEATURE_SUPERUSERS_FULL_ACCESS:
Grant superusers read/write access to registry content in all namespaces
FEATURE_RESTRICTED_USERS:
Users considered as restricted are not anle to create organization
RESTRICTED_USERS_WHITELIST:
Whitelist for FEATURE_RESTRICTED_USERS
RESTRICTED_USER_INCLUDE_ROBOTS:
Whether or not to include the user namespace's robots
RESTRICTED_USER_READ_ONLY:
Only allow read-only operations for restricted users
* Revert superuser repositorylist endpoint
- The database health check is currently not using the db_kwargs and not using ssl settings
- This is causing health check failures for MySQL behind SSL
Previous logic for claiming mirror ("locking") relied on the value
returned from updating the database row. Since this was always being
updated with a new expiration time, it would always succeed, even when
another process had already claimed the same mirror.
Method for calculating size of repo joined on repo id instead of manifest id. This causes deleted manifests to be counted. Change updates call to join on manifest id.
introduces a check of the OrganizationMemberPermission for pulls
against a proxy org.
if the proxy cache feature is disabled, or the org is not a proxy org
the check is not performed and Quay will behave normally.
this check does not mean pulls will work transparently though -
non-admin users need to be added to a team in the proxy org with the
member role, and default read and write permissions need to be given to
that team so that non-admin users can pull and update the cache for
images they do not own (the user who first pulls an image ends up
owning the repository since that is when the repo gets created).
* Quota: Show system default on UI when quota configuration for the org is not set (PROJQUAY-3518)
* Fixing formatting
* Added function to reduce redundancy and shortened warning message
* added missing parameter to function call
* Fixed organization quota consumption view
* Fixing formatting
* Checking for None before calling function
* Tracking aws ip ranges.json
* Reverting change
* Added default quota limit + added error display div + fixed indentation + minor restructuring of html
* Add check for non-negative or zero quota
* Added Limit percent check. numbers between 1-100 only acceptable
* Show warning note if no quota limit is set
* Show warning if no Reject type limit is selected
* Adding Remove button to delete quota configuration
* If Reject does not exist add default reject quota limit + css
* Throw error if more than one Reject Limit Type
* Throw error on identical limits
* Fixing showing default limits
* Added Organization view settings
* Show Remove btn only if quota exists
* Fixing Remove modal pop up
* Quota Reporting if quota is not set
* Fixing Removing Quota for org
* Fixing super user and org admin permissions
* fix to checking default quota
* Added super user perms check on put, delete endpoints
* Fixing formatting
* Fixing multiple rejects checks + returing empty list instead of None
* Using super user API calls + hiding policy div till quota is set
* Added require_scpe and show_if decorators for super user permissions
* fixing configured quota check after return type changed from None to list
for non-cache orgs, get_namespace_size will almost never be called for
an org with no manifests.
for cache orgs, get_namespace_size is garanteed to be called once when
the org doesn't have any manifests: a first pull from a cache org will
ensure that.
this change does not negatively affect callers, as no callers rely on
get_namespace_size to return None.
fixes PROJQUAY-3538
* api: update the quota api so that it's more consistent with the other apis (PROJQUAY-2936)
- Uodate the quota api to be more consistent with the rest of the
endpoints
- Handles some uncaught exceptions, such as division by zero
- Update some of the quota data models used by the api to take object
references instead of names to make it easier to use
- Update table model naming conventions
- swagger operationid multiple nicknames
- Added more test cases for api
- Remove unused functions
- Update the UI for better UX, based on the api changes made
* quota: fix ui input form value
* quota: join quota type query
* Remove unused functions
Currently the CI breaks due to a dependency of black, `click`, breaking with it's latest release with `ImportError: cannot import name '_unicodefun' from 'click'`. Since black does not pin it's version of click it pulls in the latest version containing the breaking change and fails the CI check. This updates black with the patch. [See the original issue here.](https://github.com/psf/black/issues/2964) The rest of the changes are format updates introduced with the latest version of black.
introduces the possibility to pull images from external registries
through Quay, storing them locally for faster subsequent pulls.
Closes PROJQUAY-3030 and PROJQUAY-3033
* initial commit
* fixing some bugs
* create quota management
Fix json request json type
Creation of quota is working
All quota crud operations
crud for quota limits
repository size reporting
adding registry model
error levels
namespacequota
remove holdover from user file
finalizing refactor to namespace over organization
finalization of functionality
fixing formatting to match with black style
missed some files in formatting
fixing access to attribute
add single test to verify its working
fix some bugs and add defensive catching
bug fixes and code resiliency
Bug fixes and making quota limits detect properly where necessary
remove transitive delete and other bug fixes
fix formatting and trasnitive deletion issues
fix repositorysize does not exist error
fix not nul constraint and add security tests
fix security tests and bug
more security test fixes
reorder security tests
put docker file back and adjust security testing
security tests reduced
Missed changes for status 200
missed additional 201 responses getting 200
security bypass for now
Another tweak to security testing
forgot 1 endpoint
bug fix for parsing dictionary
remove unnecessary check at blob head
add initdb for quota
Incorrect syntax repair
mysql only supports decimal
adding quota specific notifications
optimization
add permission checks
adjust security and add configuration parameter
fix security test for new security levels
Fix logic errors and improve caching
fix logic issue and error reporting
adjust things according to PR comments
fix refactor left overs
miscapitilazation
missed refactor location
refactor code to remove quota limit groupings
fix refactor errors
remove transitive deletion
fix transitive deletes
Transitive deletion work
Transitive deletion work
refactor registry model and remove it
place api behind feature flag
patch feature enabledment for tests
patch feature enabledment for tests
testing to see if the config is the problem
remove patch
fix new org bug
fixing notifications
mismatched parameters
fix org not exists
fixed paramter mismatch
fix nonetype access
fix nonetype access
new tables created user deletion issues
new tables created user deletion issues
parameter mismatch
fix transitive delete
fix model access error
record does not exist missing catch
fix quota deletion to always delete limits
quotalimits deletion on quota deletion
mistake
fix quota limits deletion
patch tests and disable feature
typo
switch to toggle feature
add feature patch to top of file
change testconfigpy
* change permissions
* adjust permissions
* change config access
* fix formatting
* gether feature information differently
* duplicate function name
* fix config name
* type conversion
* config adjustments
* incorrect keyword
* Update security api tests
* duplicate naming
* fix config schema
* revert files and fix error
* QuotaManagement: UI (PROJQUAY-2936) (#1)
* [WIP]: Quota Reporting on Quay UI
* Integrating quota reporting UI with backend
* Humanizing bytes on UI
* Quota Reporting UI on repo table view
* Taking pull and updating code
* Adding quota management view
* Added support for CRUD operations for org quota
* create quota management
Fix json request json type
Creation of quota is working
All quota crud operations
crud for quota limits
repository size reporting
adding registry model
error levels
namespacequota
remove holdover from user file
finalizing refactor to namespace over organization
finalization of functionality
fixing formatting to match with black style
missed some files in formatting
fixing access to attribute
add single test to verify its working
fix some bugs and add defensive catching
bug fixes and code resiliency
Bug fixes and making quota limits detect properly where necessary
remove transitive delete and other bug fixes
fix formatting and trasnitive deletion issues
fix repositorysize does not exist error
fix not nul constraint and add security tests
fix security tests and bug
more security test fixes
reorder security tests
put docker file back and adjust security testing
security tests reduced
Missed changes for status 200
missed additional 201 responses getting 200
security bypass for now
Another tweak to security testing
forgot 1 endpoint
bug fix for parsing dictionary
remove unnecessary check at blob head
add initdb for quota
Incorrect syntax repair
mysql only supports decimal
adding quota specific notifications
optimization
add permission checks
adjust security and add configuration parameter
fix security test for new security levels
Fix logic errors and improve caching
fix logic issue and error reporting
adjust things according to PR comments
fix refactor left overs
miscapitilazation
missed refactor location
refactor code to remove quota limit groupings
fix refactor errors
remove transitive deletion
fix transitive deletes
Transitive deletion work
Transitive deletion work
refactor registry model and remove it
place api behind feature flag
patch feature enabledment for tests
patch feature enabledment for tests
testing to see if the config is the problem
remove patch
fix new org bug
fixing notifications
mismatched parameters
fix org not exists
fixed paramter mismatch
fix nonetype access
fix nonetype access
new tables created user deletion issues
new tables created user deletion issues
parameter mismatch
fix transitive delete
fix model access error
record does not exist missing catch
fix quota deletion to always delete limits
quotalimits deletion on quota deletion
mistake
fix quota limits deletion
patch tests and disable feature
typo
switch to toggle feature
add feature patch to top of file
change testconfigpy
* Removing quota and state conf from repo-list and user-view
* Removing quota and state conf form app list page
* Removing quota conf from repo-list.html
* minor fixes
* Added Quota Repoting and configuring quota from UI
* Making quota configuration component reusable + added support to read bytes via KB, MB, etc + Added reporting for total org consumption + Added org consumption for super user panel + Added quota configurable support on super user panel
* Adding older quota management component
* Removing not reusable quota management component
* Adding % consumption for repo quotas
* Adding % consumption for organization level quota
* Adding check to verify request.args
* Removing todo
* Adding default 0 to quota
* Formatting with black
* Fixing params for tests
* Formatting test file
Co-authored-by: Keith Westphal <kwestpha@redhat.com>
* remove migration
* add migration back
* repair formatting
* QuotaManagement: Moving the logic for bytes conversion to human friendly units to the frontend (PROJQUAY-2936) (#3)
* Moving the logic for bytes conversion to human friendly units to the frontend
* Reading updates from quota_limit_id
* Formatting using black
* remote unused function
* Adding quota configuring on super user panel (#4)
* Converting quota bytes to human friendly format (#5)
* PR refactors
* invalid reference
* bad return value
* fix bad reference
* bad reference
* fix tests
* Quota Config: UI improvements (#6)
* Quota UI Improvements
* Rendering table for quota limit config
* Removing proxy cache files
* Disabling quota config for org view
* Removing redundant get
* Fixing PR requests
* repair formatting
Co-authored-by: Sunandadadi <Sunandadadi@users.noreply.github.com>
Currently when attempting to mirror a registry containing unsigned images the mirror will fail due to not finding the source signature. This is caused by the updated version of Skopeo blocking unsigned images by default. This allows users to specify the ability to pull unsigned images per-repository. The Skopeo version is also now pinned.
OAuth Authorization Code Flow is broken in quay. Code
Flow is more secure than implicit flow and is used
by server side applications to get the access token
AuthorizationProvider uses explicit parameter names when it calls its
methods. Therefore methods of derivative classes, i.e. the methods of
the class DatabaseAuthorizationProvider, should have the same parameter
names.
The "uploading" column is an artifact from depending on writing to the
Image table (see BlobUpload table instead). As of 3.4, Quay no longer
writes to that table, and is only needed until quayio moves away from
Clair v2, after which work to remove "glue" code and fully deprecate
the Image table (amongst other tables) can start.
This is done as a separate commit from the actual migration so that it
can be cherrypicked.
Since NamespaceGCWorker does a superset of RepositoryGCWorker's
operations, make sure that quay_gc_repos_purged is incremented if
either workers deletes a repository.
Prevents the queueworker from setting the event to stop the poll_queue
job when a WorkerSleepException is raised. On WorkerSleepException,
the worker should instead skip this iteration (go to sleep). e.g when
the NamespaceGCWorker can't acquire a lock because it is already taken
by some other worker.
Reverts the gcworkers job timeout from 24h to 3h. In case of a
deadlock between processes (for example, redeploying the app will not
clear the existing Redis keys), 24h is too long waiting for the locks to
expires so that the workers can resume work.
Add missing Counter increment for on row deletion on the Manifest table.
Apply a manifest's "quay.expires-after" label expiry value to new tags pointing
to that existing manifest. Before, that label would be only applied at when the
manifest was created, and new tags targeting that manifest would not
have the manifest's corresponding expiry value set.
