feat: Add image pull statistics API endpoints and UI integration
- Add new API endpoints for tag and manifest pull statistics
- Integrate pull metrics into web UI with new table columns
- Add FEATURE_IMAGE_PULL_STATS feature flag and PULL_METRICS_REDIS config
- Add pullstatsredisflushworker to supervisord configuration
- Add comprehensive test coverage for pull statistics functionality
Co-authored-by: shudeshp <shudeshp@redhat.com>
* mirror: Add job timeout to mirror configurations (PROJQUAY-7249)
Previous global job timeout of 5 minutes was inadequate for big images. The timeout should now be configurable in much the same way as sync is. Minimum job length is 300 seconds/5 minutes.
The PR is still work in progress.
* Fix init db, remove reference to user data in logs
* Fix tests, change repo mirror configuration
* Fix tests, make mirroring cancellable through UI
* Add cancel mirror test, change HTML document to reflect mirror timeout
* Flake8 doesn't like when '==' is used with 'None'
* Fix mirror registry tests
* Add new cypress data to fix cypress tests
* Added ability to define upload chunk size to RADOS driver, small changes to repo mirror HTML page
* Fix database migration to follow HEAD
* Upload new database data for Cypress tests
* Make skopeo_timeout_interval mandatory on API calls
---------
Co-authored-by: Ivan Bazulic <ibazulic@redhat.com>
* marketplace: splittable sku for MW02702 (PROJQUAY-8151)
* Alembic migration to drop unique constraint on the orgrhsubscriptions
table
* Can split sub quantities of MW02702 across multiple orgs
* Can specify quantity for the MW02702 SKU across orgs on react UI
* Update angular UI to allow user to specify quantities for MW02702
Old manifest vulnerability status page, still in use, would cause artifacts on long URLs and huge vertical scroll bars. This fix limits the size of the displayed link thus eliminating vertical scrolls. Tested on up to 150% zoom on 4k resolution.
* Adds handling for when a subscription returned from the subscription watch api has a quantity greater than 1. Number of private repos should be correctly calculated using the quantity.
* Updates ui so that subscriptions can only be added to an org as a group, i.e. a subscription with quantity = 2 cannot be split across organizations.
* billing: marketplace UI
adds UI in billing section for managing user and org-bound skus
add more unit tests for org binding
changed endpoint for bulk attaching skus to orgs
Allows for only unique blobs are counted at the namespace and repository level. Calculation includes manifest list sizes.
Add's the following internal configurations that default to true:
QUOTA_INVALIDATE_TOTALS: Invalidates calculated totals when FEATURE_QUOTA_MANAGEMENT is set to false
RESET_CHILD_MANIFEST_EXPIRATION: Resets the expiry for child manifests on push of the manifest list for immediate GC eligibility
PERMANENTLY_DELETE_TAGS: Enables features related to the permanent deletion of tags outside the configured time machine window
ui: Show UI toggle on quay.io only to redhat users (PROJQUAY-4804)
New UI preview on quay.io is being released as a preview only
to RedHat users before being GA
* ui: basic support for cosign in the UI (PROJQUAY-3965)
* sharpen regex for cosign manifest tag name convention
Co-authored-by: Oleg Bulatov <oleg@bulatov.me>
* fix broken signature match
* addressing review comments
* fix tool tip width issues
Co-authored-by: Oleg Bulatov <oleg@bulatov.me>
* Quota: Configuring Quota for user panel
* Added Quota Consumed column on Super users panel
* Fixing tests
* Fixing tests
* Adding tests for user quota operations
* Reverting org api changes + new endpoint for super user get method
* Reverting changes
* Added tests
* Fetching user namespace or organization
* Quota: Show system default on UI when quota configuration for the org is not set (PROJQUAY-3518)
* Fixing formatting
* Added function to reduce redundancy and shortened warning message
* added missing parameter to function call
* Fixed organization quota consumption view
* Fixing formatting
* Checking for None before calling function
* Quota Management: Quota settings on Organization view needs to be read only (PROJQUAY-3622)
* Adding superuser permissions check on put and delete methods
* Reverting changes
* Reverting changes
* Tracking aws ip ranges.json
* Reverting change
* Added default quota limit + added error display div + fixed indentation + minor restructuring of html
* Add check for non-negative or zero quota
* Added Limit percent check. numbers between 1-100 only acceptable
* Show warning note if no quota limit is set
* Show warning if no Reject type limit is selected
* Adding Remove button to delete quota configuration
* If Reject does not exist add default reject quota limit + css
* Throw error if more than one Reject Limit Type
* Throw error on identical limits
* Fixing showing default limits
* Added Organization view settings
* Show Remove btn only if quota exists
* Fixing Remove modal pop up
* Quota Reporting if quota is not set
* Fixing Removing Quota for org
* Fixing super user and org admin permissions
* fix to checking default quota
* Added super user perms check on put, delete endpoints
* Fixing formatting
* Fixing multiple rejects checks + returing empty list instead of None
* Using super user API calls + hiding policy div till quota is set
* Added require_scpe and show_if decorators for super user permissions
* fixing configured quota check after return type changed from None to list
* api: update the quota api so that it's more consistent with the other apis (PROJQUAY-2936)
- Uodate the quota api to be more consistent with the rest of the
endpoints
- Handles some uncaught exceptions, such as division by zero
- Update some of the quota data models used by the api to take object
references instead of names to make it easier to use
- Update table model naming conventions
- swagger operationid multiple nicknames
- Added more test cases for api
- Remove unused functions
- Update the UI for better UX, based on the api changes made
* quota: fix ui input form value
* quota: join quota type query
* Remove unused functions
Currently when viewing the build logs and selecting `Copy Logs` the copy to clipboard will fail. This is because the content of the logs was being passed to the copy-clipboard function when it requires the ID of the element containing the text to copy. This passes the selector of the ID of the element containing the build logs.
* initial commit
* fixing some bugs
* create quota management
Fix json request json type
Creation of quota is working
All quota crud operations
crud for quota limits
repository size reporting
adding registry model
error levels
namespacequota
remove holdover from user file
finalizing refactor to namespace over organization
finalization of functionality
fixing formatting to match with black style
missed some files in formatting
fixing access to attribute
add single test to verify its working
fix some bugs and add defensive catching
bug fixes and code resiliency
Bug fixes and making quota limits detect properly where necessary
remove transitive delete and other bug fixes
fix formatting and trasnitive deletion issues
fix repositorysize does not exist error
fix not nul constraint and add security tests
fix security tests and bug
more security test fixes
reorder security tests
put docker file back and adjust security testing
security tests reduced
Missed changes for status 200
missed additional 201 responses getting 200
security bypass for now
Another tweak to security testing
forgot 1 endpoint
bug fix for parsing dictionary
remove unnecessary check at blob head
add initdb for quota
Incorrect syntax repair
mysql only supports decimal
adding quota specific notifications
optimization
add permission checks
adjust security and add configuration parameter
fix security test for new security levels
Fix logic errors and improve caching
fix logic issue and error reporting
adjust things according to PR comments
fix refactor left overs
miscapitilazation
missed refactor location
refactor code to remove quota limit groupings
fix refactor errors
remove transitive deletion
fix transitive deletes
Transitive deletion work
Transitive deletion work
refactor registry model and remove it
place api behind feature flag
patch feature enabledment for tests
patch feature enabledment for tests
testing to see if the config is the problem
remove patch
fix new org bug
fixing notifications
mismatched parameters
fix org not exists
fixed paramter mismatch
fix nonetype access
fix nonetype access
new tables created user deletion issues
new tables created user deletion issues
parameter mismatch
fix transitive delete
fix model access error
record does not exist missing catch
fix quota deletion to always delete limits
quotalimits deletion on quota deletion
mistake
fix quota limits deletion
patch tests and disable feature
typo
switch to toggle feature
add feature patch to top of file
change testconfigpy
* change permissions
* adjust permissions
* change config access
* fix formatting
* gether feature information differently
* duplicate function name
* fix config name
* type conversion
* config adjustments
* incorrect keyword
* Update security api tests
* duplicate naming
* fix config schema
* revert files and fix error
* QuotaManagement: UI (PROJQUAY-2936) (#1)
* [WIP]: Quota Reporting on Quay UI
* Integrating quota reporting UI with backend
* Humanizing bytes on UI
* Quota Reporting UI on repo table view
* Taking pull and updating code
* Adding quota management view
* Added support for CRUD operations for org quota
* create quota management
Fix json request json type
Creation of quota is working
All quota crud operations
crud for quota limits
repository size reporting
adding registry model
error levels
namespacequota
remove holdover from user file
finalizing refactor to namespace over organization
finalization of functionality
fixing formatting to match with black style
missed some files in formatting
fixing access to attribute
add single test to verify its working
fix some bugs and add defensive catching
bug fixes and code resiliency
Bug fixes and making quota limits detect properly where necessary
remove transitive delete and other bug fixes
fix formatting and trasnitive deletion issues
fix repositorysize does not exist error
fix not nul constraint and add security tests
fix security tests and bug
more security test fixes
reorder security tests
put docker file back and adjust security testing
security tests reduced
Missed changes for status 200
missed additional 201 responses getting 200
security bypass for now
Another tweak to security testing
forgot 1 endpoint
bug fix for parsing dictionary
remove unnecessary check at blob head
add initdb for quota
Incorrect syntax repair
mysql only supports decimal
adding quota specific notifications
optimization
add permission checks
adjust security and add configuration parameter
fix security test for new security levels
Fix logic errors and improve caching
fix logic issue and error reporting
adjust things according to PR comments
fix refactor left overs
miscapitilazation
missed refactor location
refactor code to remove quota limit groupings
fix refactor errors
remove transitive deletion
fix transitive deletes
Transitive deletion work
Transitive deletion work
refactor registry model and remove it
place api behind feature flag
patch feature enabledment for tests
patch feature enabledment for tests
testing to see if the config is the problem
remove patch
fix new org bug
fixing notifications
mismatched parameters
fix org not exists
fixed paramter mismatch
fix nonetype access
fix nonetype access
new tables created user deletion issues
new tables created user deletion issues
parameter mismatch
fix transitive delete
fix model access error
record does not exist missing catch
fix quota deletion to always delete limits
quotalimits deletion on quota deletion
mistake
fix quota limits deletion
patch tests and disable feature
typo
switch to toggle feature
add feature patch to top of file
change testconfigpy
* Removing quota and state conf from repo-list and user-view
* Removing quota and state conf form app list page
* Removing quota conf from repo-list.html
* minor fixes
* Added Quota Repoting and configuring quota from UI
* Making quota configuration component reusable + added support to read bytes via KB, MB, etc + Added reporting for total org consumption + Added org consumption for super user panel + Added quota configurable support on super user panel
* Adding older quota management component
* Removing not reusable quota management component
* Adding % consumption for repo quotas
* Adding % consumption for organization level quota
* Adding check to verify request.args
* Removing todo
* Adding default 0 to quota
* Formatting with black
* Fixing params for tests
* Formatting test file
Co-authored-by: Keith Westphal <kwestpha@redhat.com>
* remove migration
* add migration back
* repair formatting
* QuotaManagement: Moving the logic for bytes conversion to human friendly units to the frontend (PROJQUAY-2936) (#3)
* Moving the logic for bytes conversion to human friendly units to the frontend
* Reading updates from quota_limit_id
* Formatting using black
* remote unused function
* Adding quota configuring on super user panel (#4)
* Converting quota bytes to human friendly format (#5)
* PR refactors
* invalid reference
* bad return value
* fix bad reference
* bad reference
* fix tests
* Quota Config: UI improvements (#6)
* Quota UI Improvements
* Rendering table for quota limit config
* Removing proxy cache files
* Disabling quota config for org view
* Removing redundant get
* Fixing PR requests
* repair formatting
Co-authored-by: Sunandadadi <Sunandadadi@users.noreply.github.com>
Display individual manifest size in manifest list view.
Also fixes a non-cluster Redis cache bug.
TODO: This requires making extra individiual requests to get the
children manifests. They should probably be cached.
Currently when attempting to mirror a registry containing unsigned images the mirror will fail due to not finding the source signature. This is caused by the updated version of Skopeo blocking unsigned images by default. This allows users to specify the ability to pull unsigned images per-repository. The Skopeo version is also now pinned.
* Quay.io UI: Fetching severity from cvss score and removing visibility of unknown metrics (PROJQUAY-2541)
If vulnerability's severity is missing mapping severity from cvss score if present. Removed visibility if severity or distribition is Unknow
* Quay.io UI: Fetching severity from cvss score and removing visibility of unknown metrics (PROJQUAY-2541)
Fixing Typo
