Method for calculating size of repo joined on repo id instead of manifest id. This causes deleted manifests to be counted. Change updates call to join on manifest id.
introduces a check of the OrganizationMemberPermission for pulls
against a proxy org.
if the proxy cache feature is disabled, or the org is not a proxy org
the check is not performed and Quay will behave normally.
this check does not mean pulls will work transparently though -
non-admin users need to be added to a team in the proxy org with the
member role, and default read and write permissions need to be given to
that team so that non-admin users can pull and update the cache for
images they do not own (the user who first pulls an image ends up
owning the repository since that is when the repo gets created).
Removes read support for Clair V2, along with the need to package
jwtproxy with Quay.
TODO: Drop deprecate image api + image table, remove image data model.
The distribution spec does not require the docker-content-digest header
to be set in response to a manifest GET/HEAD request.
This changes both the proxy client and the registry proxy model to
correctly check whether a manifest is up-to-date with the upstream
registry or not when no digest header is received.
NOTE: when checking staleness against registries that do not return the
docker-content-digest header, Quay will make a GET request to the
registry and calculate the digest from the manifest itself. GET requests
usually count towards rate-limiting.
This change also sets the accept-encoding header to 'identity'. The python
requests library seems to automatically set the accept-encoding header to
'gzip'. Dockerhub ignores that header when serving blobs, but some
registries don't (namely registry.access.redhat.com). When Quay receives a
gzipped config blob (have not tested non-config blobs) for some reason it
doesn't know how to handle it. I suspect it has to do wit the fact that in
this case the content-length header will differ from the actual size of
the response body, so when Quay tries to upload the blob it cannot
correctly calculate the actual blob size, so it does a partial upload to
its object storage, which then results in a digest mismatch error
(BlobDigestMismatchException).
Split the work of indexing the recent manifests iterator into multiple
batches. This can reduce how often duplicate work happens when
allowing multiple workers to work on the same chunk of the table.
Index recent manifests in a separate background process, allowing the
main process to correctly select random slabs from the entire table
set and marking them completed in the allocator (rbtree). This avoids
the worker having to start iterating from the beginning of the table
whenever it is restarted.
* Quota: Show system default on UI when quota configuration for the org is not set (PROJQUAY-3518)
* Fixing formatting
* Added function to reduce redundancy and shortened warning message
* added missing parameter to function call
* Fixed organization quota consumption view
* Fixing formatting
* Checking for None before calling function
* Tracking aws ip ranges.json
* Reverting change
* Added default quota limit + added error display div + fixed indentation + minor restructuring of html
* Add check for non-negative or zero quota
* Added Limit percent check. numbers between 1-100 only acceptable
* Show warning note if no quota limit is set
* Show warning if no Reject type limit is selected
* Adding Remove button to delete quota configuration
* If Reject does not exist add default reject quota limit + css
* Throw error if more than one Reject Limit Type
* Throw error on identical limits
* Fixing showing default limits
* Added Organization view settings
* Show Remove btn only if quota exists
* Fixing Remove modal pop up
* Quota Reporting if quota is not set
* Fixing Removing Quota for org
* Fixing super user and org admin permissions
* fix to checking default quota
* Added super user perms check on put, delete endpoints
* Fixing formatting
* Fixing multiple rejects checks + returing empty list instead of None
* Using super user API calls + hiding policy div till quota is set
* Added require_scpe and show_if decorators for super user permissions
* fixing configured quota check after return type changed from None to list
Add histogram for request duration on indexer service. Add random
batch to manifest iterator to reduce duplicate work while manifests
are being backfilled.
Increases the maximum length of external_registry_username and
external_registry_password fields from 2048 to 4096. Some registries,
e.g ECR, issue a token to be used as password, whose length is longer
than 2048 after being encrypted and base64 encoded. Since JWT's specs
do not specify a maximum length, we set the combined max length of
these fields to approximately 8K, the default header size on most
servers (including nginx), since these credentials will be passed in a
auth header during the auth flow.
for non-cache orgs, get_namespace_size will almost never be called for
an org with no manifests.
for cache orgs, get_namespace_size is garanteed to be called once when
the org doesn't have any manifests: a first pull from a cache org will
ensure that.
this change does not negatively affect callers, as no callers rely on
get_namespace_size to return None.
fixes PROJQUAY-3538
Have the securityworker send an abort signal in the case where the
conditions for indexing a manifest are not met after running the
candidate query. This could be when:
- a manifest has a reference to a manifestsecuritystatus instance
- the manifestsecuritystatus reindex threshold is no longer
valid (because it was recently updated)
This signals that another worker overlapped, and allows the current
worker to move on to the next set.
* api: update the quota api so that it's more consistent with the other apis (PROJQUAY-2936)
- Uodate the quota api to be more consistent with the rest of the
endpoints
- Handles some uncaught exceptions, such as division by zero
- Update some of the quota data models used by the api to take object
references instead of names to make it easier to use
- Update table model naming conventions
- swagger operationid multiple nicknames
- Added more test cases for api
- Remove unused functions
- Update the UI for better UX, based on the api changes made
* quota: fix ui input form value
* quota: join quota type query
* Remove unused functions
Add a global lock on security worker iterations, based on the value of
the current pagination token. This is to avoid multilple worker
processes possibly doing work on the same set of manifests.
Currently the CI breaks due to a dependency of black, `click`, breaking with it's latest release with `ImportError: cannot import name '_unicodefun' from 'click'`. Since black does not pin it's version of click it pulls in the latest version containing the breaking change and fails the CI check. This updates black with the patch. [See the original issue here.](https://github.com/psf/black/issues/2964) The rest of the changes are format updates introduced with the latest version of black.
introduces the possibility to pull images from external registries
through Quay, storing them locally for faster subsequent pulls.
Closes PROJQUAY-3030 and PROJQUAY-3033
* initial commit
* fixing some bugs
* create quota management
Fix json request json type
Creation of quota is working
All quota crud operations
crud for quota limits
repository size reporting
adding registry model
error levels
namespacequota
remove holdover from user file
finalizing refactor to namespace over organization
finalization of functionality
fixing formatting to match with black style
missed some files in formatting
fixing access to attribute
add single test to verify its working
fix some bugs and add defensive catching
bug fixes and code resiliency
Bug fixes and making quota limits detect properly where necessary
remove transitive delete and other bug fixes
fix formatting and trasnitive deletion issues
fix repositorysize does not exist error
fix not nul constraint and add security tests
fix security tests and bug
more security test fixes
reorder security tests
put docker file back and adjust security testing
security tests reduced
Missed changes for status 200
missed additional 201 responses getting 200
security bypass for now
Another tweak to security testing
forgot 1 endpoint
bug fix for parsing dictionary
remove unnecessary check at blob head
add initdb for quota
Incorrect syntax repair
mysql only supports decimal
adding quota specific notifications
optimization
add permission checks
adjust security and add configuration parameter
fix security test for new security levels
Fix logic errors and improve caching
fix logic issue and error reporting
adjust things according to PR comments
fix refactor left overs
miscapitilazation
missed refactor location
refactor code to remove quota limit groupings
fix refactor errors
remove transitive deletion
fix transitive deletes
Transitive deletion work
Transitive deletion work
refactor registry model and remove it
place api behind feature flag
patch feature enabledment for tests
patch feature enabledment for tests
testing to see if the config is the problem
remove patch
fix new org bug
fixing notifications
mismatched parameters
fix org not exists
fixed paramter mismatch
fix nonetype access
fix nonetype access
new tables created user deletion issues
new tables created user deletion issues
parameter mismatch
fix transitive delete
fix model access error
record does not exist missing catch
fix quota deletion to always delete limits
quotalimits deletion on quota deletion
mistake
fix quota limits deletion
patch tests and disable feature
typo
switch to toggle feature
add feature patch to top of file
change testconfigpy
* change permissions
* adjust permissions
* change config access
* fix formatting
* gether feature information differently
* duplicate function name
* fix config name
* type conversion
* config adjustments
* incorrect keyword
* Update security api tests
* duplicate naming
* fix config schema
* revert files and fix error
* QuotaManagement: UI (PROJQUAY-2936) (#1)
* [WIP]: Quota Reporting on Quay UI
* Integrating quota reporting UI with backend
* Humanizing bytes on UI
* Quota Reporting UI on repo table view
* Taking pull and updating code
* Adding quota management view
* Added support for CRUD operations for org quota
* create quota management
Fix json request json type
Creation of quota is working
All quota crud operations
crud for quota limits
repository size reporting
adding registry model
error levels
namespacequota
remove holdover from user file
finalizing refactor to namespace over organization
finalization of functionality
fixing formatting to match with black style
missed some files in formatting
fixing access to attribute
add single test to verify its working
fix some bugs and add defensive catching
bug fixes and code resiliency
Bug fixes and making quota limits detect properly where necessary
remove transitive delete and other bug fixes
fix formatting and trasnitive deletion issues
fix repositorysize does not exist error
fix not nul constraint and add security tests
fix security tests and bug
more security test fixes
reorder security tests
put docker file back and adjust security testing
security tests reduced
Missed changes for status 200
missed additional 201 responses getting 200
security bypass for now
Another tweak to security testing
forgot 1 endpoint
bug fix for parsing dictionary
remove unnecessary check at blob head
add initdb for quota
Incorrect syntax repair
mysql only supports decimal
adding quota specific notifications
optimization
add permission checks
adjust security and add configuration parameter
fix security test for new security levels
Fix logic errors and improve caching
fix logic issue and error reporting
adjust things according to PR comments
fix refactor left overs
miscapitilazation
missed refactor location
refactor code to remove quota limit groupings
fix refactor errors
remove transitive deletion
fix transitive deletes
Transitive deletion work
Transitive deletion work
refactor registry model and remove it
place api behind feature flag
patch feature enabledment for tests
patch feature enabledment for tests
testing to see if the config is the problem
remove patch
fix new org bug
fixing notifications
mismatched parameters
fix org not exists
fixed paramter mismatch
fix nonetype access
fix nonetype access
new tables created user deletion issues
new tables created user deletion issues
parameter mismatch
fix transitive delete
fix model access error
record does not exist missing catch
fix quota deletion to always delete limits
quotalimits deletion on quota deletion
mistake
fix quota limits deletion
patch tests and disable feature
typo
switch to toggle feature
add feature patch to top of file
change testconfigpy
* Removing quota and state conf from repo-list and user-view
* Removing quota and state conf form app list page
* Removing quota conf from repo-list.html
* minor fixes
* Added Quota Repoting and configuring quota from UI
* Making quota configuration component reusable + added support to read bytes via KB, MB, etc + Added reporting for total org consumption + Added org consumption for super user panel + Added quota configurable support on super user panel
* Adding older quota management component
* Removing not reusable quota management component
* Adding % consumption for repo quotas
* Adding % consumption for organization level quota
* Adding check to verify request.args
* Removing todo
* Adding default 0 to quota
* Formatting with black
* Fixing params for tests
* Formatting test file
Co-authored-by: Keith Westphal <kwestpha@redhat.com>
* remove migration
* add migration back
* repair formatting
* QuotaManagement: Moving the logic for bytes conversion to human friendly units to the frontend (PROJQUAY-2936) (#3)
* Moving the logic for bytes conversion to human friendly units to the frontend
* Reading updates from quota_limit_id
* Formatting using black
* remote unused function
* Adding quota configuring on super user panel (#4)
* Converting quota bytes to human friendly format (#5)
* PR refactors
* invalid reference
* bad return value
* fix bad reference
* bad reference
* fix tests
* Quota Config: UI improvements (#6)
* Quota UI Improvements
* Rendering table for quota limit config
* Removing proxy cache files
* Disabling quota config for org view
* Removing redundant get
* Fixing PR requests
* repair formatting
Co-authored-by: Sunandadadi <Sunandadadi@users.noreply.github.com>
Display individual manifest size in manifest list view.
Also fixes a non-cluster Redis cache bug.
TODO: This requires making extra individiual requests to get the
children manifests. They should probably be cached.
For larger databases like quay.io the default batch
size is too high to complete the job in a resonable
time. Make the batch_size configurable so we can
adjust how many scans can be in-flight for a worker
An InterfaceError is raised by PyMySQL after a connection has
been idle for longer than the MySQL database's interactive_timeout
setting. This can happen on some workers without work to be done for a
long time. This causes Peewee to be unable to reuse its stale
connection. Instead, the peewee connection needs to be closed, and a
new session opened with MySQL.
Currently we randomly scan for entries to be indexed. This could take a
long time if we have a lot of unindexed images. Ideally, we want to
priortize scanning of newer images before we go to the old ones.
This PR creates a smaller batch of size `batch_size` from
`max_id - batch_size` to `max_id` to priortize scanning
those
Currently it is possible that Clair returns a state that Quay doesn't
know about and the previous logic will save this as a completed
manifest.
Signed-off-by: crozzy <joseph.crosland@gmail.com>
