When authenticating again a registry, if the www-authenticate header
doesn't specify a service, exclude it from the url
Signed-off-by: harishsurf <hgovinda@redhat.com>
- Similar to LDAP_SUPERUSER_FILTER, add a specific filter to define
restricted users, based on the LDAP_USER_FILTER
- restrict writes on restricted users' own namespace. Normal
permissions applies on organization membership
- add global readonly superuser GLOBAL_READONLY_SUPER_USERS (PROJQUAY-2604)
- Removes RESTRICTED_USER_INCLUDE_ROBOTS, FEATURE_RESTRICTED_READ_ONLY_USERS
* api: feature to limit org creation to superusers
Introduces the following configuration flags:
FEATURE_SUPERUSERS_ORG_CREATION_ONLY:
Limit org creation to superusers only
FEATURE_SUPERUSERS_FULL_ACCESS:
Grant superusers read/write access to registry content in all namespaces
FEATURE_RESTRICTED_USERS:
Users considered as restricted are not anle to create organization
RESTRICTED_USERS_WHITELIST:
Whitelist for FEATURE_RESTRICTED_USERS
RESTRICTED_USER_INCLUDE_ROBOTS:
Whether or not to include the user namespace's robots
RESTRICTED_USER_READ_ONLY:
Only allow read-only operations for restricted users
* Revert superuser repositorylist endpoint
* Update peewee types
Also remove tools/sharedimagestorage.py as it doesn't work anymore.
tools/sharedimagestorage.py:3: error: "ModelSelect[ImageStorage]" has no attribute "annotate"
* Remove endpoints/api/test/test_security.py from exclude list
* Format storage/test/test_azure.py
* Quota: Configuring Quota for user panel
* Added Quota Consumed column on Super users panel
* Fixing tests
* Fixing tests
* Adding tests for user quota operations
* Reverting org api changes + new endpoint for super user get method
* Reverting changes
* Added tests
* Fetching user namespace or organization
* Quota API: Add super user permissions on Organization endpoints (PROJQUAY-3742)
* Removing super user permissions form userquota endpoints
* Adding super user permission checks
* Moving super user scope decorator to class level
* Quota: Show system default on UI when quota configuration for the org is not set (PROJQUAY-3518)
* Fixing formatting
* Added function to reduce redundancy and shortened warning message
* added missing parameter to function call
* Fixed organization quota consumption view
* Fixing formatting
* Checking for None before calling function
* Quota Management: Quota settings on Organization view needs to be read only (PROJQUAY-3622)
* Adding superuser permissions check on put and delete methods
* Reverting changes
* Reverting changes
* Tracking aws ip ranges.json
* Reverting change
* Added default quota limit + added error display div + fixed indentation + minor restructuring of html
* Add check for non-negative or zero quota
* Added Limit percent check. numbers between 1-100 only acceptable
* Show warning note if no quota limit is set
* Show warning if no Reject type limit is selected
* Adding Remove button to delete quota configuration
* If Reject does not exist add default reject quota limit + css
* Throw error if more than one Reject Limit Type
* Throw error on identical limits
* Fixing showing default limits
* Added Organization view settings
* Show Remove btn only if quota exists
* Fixing Remove modal pop up
* Quota Reporting if quota is not set
* Fixing Removing Quota for org
* Fixing super user and org admin permissions
* fix to checking default quota
* Added super user perms check on put, delete endpoints
* Fixing formatting
* Fixing multiple rejects checks + returing empty list instead of None
* Using super user API calls + hiding policy div till quota is set
* Added require_scpe and show_if decorators for super user permissions
* fixing configured quota check after return type changed from None to list
* api: update the quota api so that it's more consistent with the other apis (PROJQUAY-2936)
- Uodate the quota api to be more consistent with the rest of the
endpoints
- Handles some uncaught exceptions, such as division by zero
- Update some of the quota data models used by the api to take object
references instead of names to make it easier to use
- Update table model naming conventions
- swagger operationid multiple nicknames
- Added more test cases for api
- Remove unused functions
- Update the UI for better UX, based on the api changes made
* quota: fix ui input form value
* quota: join quota type query
* Remove unused functions
introduces the possibility to pull images from external registries
through Quay, storing them locally for faster subsequent pulls.
Closes PROJQUAY-3030 and PROJQUAY-3033
* initial commit
* fixing some bugs
* create quota management
Fix json request json type
Creation of quota is working
All quota crud operations
crud for quota limits
repository size reporting
adding registry model
error levels
namespacequota
remove holdover from user file
finalizing refactor to namespace over organization
finalization of functionality
fixing formatting to match with black style
missed some files in formatting
fixing access to attribute
add single test to verify its working
fix some bugs and add defensive catching
bug fixes and code resiliency
Bug fixes and making quota limits detect properly where necessary
remove transitive delete and other bug fixes
fix formatting and trasnitive deletion issues
fix repositorysize does not exist error
fix not nul constraint and add security tests
fix security tests and bug
more security test fixes
reorder security tests
put docker file back and adjust security testing
security tests reduced
Missed changes for status 200
missed additional 201 responses getting 200
security bypass for now
Another tweak to security testing
forgot 1 endpoint
bug fix for parsing dictionary
remove unnecessary check at blob head
add initdb for quota
Incorrect syntax repair
mysql only supports decimal
adding quota specific notifications
optimization
add permission checks
adjust security and add configuration parameter
fix security test for new security levels
Fix logic errors and improve caching
fix logic issue and error reporting
adjust things according to PR comments
fix refactor left overs
miscapitilazation
missed refactor location
refactor code to remove quota limit groupings
fix refactor errors
remove transitive deletion
fix transitive deletes
Transitive deletion work
Transitive deletion work
refactor registry model and remove it
place api behind feature flag
patch feature enabledment for tests
patch feature enabledment for tests
testing to see if the config is the problem
remove patch
fix new org bug
fixing notifications
mismatched parameters
fix org not exists
fixed paramter mismatch
fix nonetype access
fix nonetype access
new tables created user deletion issues
new tables created user deletion issues
parameter mismatch
fix transitive delete
fix model access error
record does not exist missing catch
fix quota deletion to always delete limits
quotalimits deletion on quota deletion
mistake
fix quota limits deletion
patch tests and disable feature
typo
switch to toggle feature
add feature patch to top of file
change testconfigpy
* change permissions
* adjust permissions
* change config access
* fix formatting
* gether feature information differently
* duplicate function name
* fix config name
* type conversion
* config adjustments
* incorrect keyword
* Update security api tests
* duplicate naming
* fix config schema
* revert files and fix error
* QuotaManagement: UI (PROJQUAY-2936) (#1)
* [WIP]: Quota Reporting on Quay UI
* Integrating quota reporting UI with backend
* Humanizing bytes on UI
* Quota Reporting UI on repo table view
* Taking pull and updating code
* Adding quota management view
* Added support for CRUD operations for org quota
* create quota management
Fix json request json type
Creation of quota is working
All quota crud operations
crud for quota limits
repository size reporting
adding registry model
error levels
namespacequota
remove holdover from user file
finalizing refactor to namespace over organization
finalization of functionality
fixing formatting to match with black style
missed some files in formatting
fixing access to attribute
add single test to verify its working
fix some bugs and add defensive catching
bug fixes and code resiliency
Bug fixes and making quota limits detect properly where necessary
remove transitive delete and other bug fixes
fix formatting and trasnitive deletion issues
fix repositorysize does not exist error
fix not nul constraint and add security tests
fix security tests and bug
more security test fixes
reorder security tests
put docker file back and adjust security testing
security tests reduced
Missed changes for status 200
missed additional 201 responses getting 200
security bypass for now
Another tweak to security testing
forgot 1 endpoint
bug fix for parsing dictionary
remove unnecessary check at blob head
add initdb for quota
Incorrect syntax repair
mysql only supports decimal
adding quota specific notifications
optimization
add permission checks
adjust security and add configuration parameter
fix security test for new security levels
Fix logic errors and improve caching
fix logic issue and error reporting
adjust things according to PR comments
fix refactor left overs
miscapitilazation
missed refactor location
refactor code to remove quota limit groupings
fix refactor errors
remove transitive deletion
fix transitive deletes
Transitive deletion work
Transitive deletion work
refactor registry model and remove it
place api behind feature flag
patch feature enabledment for tests
patch feature enabledment for tests
testing to see if the config is the problem
remove patch
fix new org bug
fixing notifications
mismatched parameters
fix org not exists
fixed paramter mismatch
fix nonetype access
fix nonetype access
new tables created user deletion issues
new tables created user deletion issues
parameter mismatch
fix transitive delete
fix model access error
record does not exist missing catch
fix quota deletion to always delete limits
quotalimits deletion on quota deletion
mistake
fix quota limits deletion
patch tests and disable feature
typo
switch to toggle feature
add feature patch to top of file
change testconfigpy
* Removing quota and state conf from repo-list and user-view
* Removing quota and state conf form app list page
* Removing quota conf from repo-list.html
* minor fixes
* Added Quota Repoting and configuring quota from UI
* Making quota configuration component reusable + added support to read bytes via KB, MB, etc + Added reporting for total org consumption + Added org consumption for super user panel + Added quota configurable support on super user panel
* Adding older quota management component
* Removing not reusable quota management component
* Adding % consumption for repo quotas
* Adding % consumption for organization level quota
* Adding check to verify request.args
* Removing todo
* Adding default 0 to quota
* Formatting with black
* Fixing params for tests
* Formatting test file
Co-authored-by: Keith Westphal <kwestpha@redhat.com>
* remove migration
* add migration back
* repair formatting
* QuotaManagement: Moving the logic for bytes conversion to human friendly units to the frontend (PROJQUAY-2936) (#3)
* Moving the logic for bytes conversion to human friendly units to the frontend
* Reading updates from quota_limit_id
* Formatting using black
* remote unused function
* Adding quota configuring on super user panel (#4)
* Converting quota bytes to human friendly format (#5)
* PR refactors
* invalid reference
* bad return value
* fix bad reference
* bad reference
* fix tests
* Quota Config: UI improvements (#6)
* Quota UI Improvements
* Rendering table for quota limit config
* Removing proxy cache files
* Disabling quota config for org view
* Removing redundant get
* Fixing PR requests
* repair formatting
Co-authored-by: Sunandadadi <Sunandadadi@users.noreply.github.com>
Currently when attempting to mirror a registry containing unsigned images the mirror will fail due to not finding the source signature. This is caused by the updated version of Skopeo blocking unsigned images by default. This allows users to specify the ability to pull unsigned images per-repository. The Skopeo version is also now pinned.
* Add dev dependencies mypy and typing
* Add makefile target `types-test`, not yet included in `test` target.
* Generate stubs for imported modules to avoid mypy complaining about missing types.
* Remove generated stubs as there are way too many and they cause tons of mess in the repo. Switched to ignoring untyped modules for now, to concentrate on Quay-only type checking.
* mypy config changed to ignore missing imports
* ignore property decorator as it is not supported by mypy
* mypy annotations for many configuration variables
* re-generate mypy_stubs directory as its necessary in some classes for base classes to prevent mypy errors
* util/registry/queuefile referred to non existent definition of Empty class in multiprocessing.queues
* ignore type checking for things like monkey patching and exported/re-imported objects that
mypy does not allow.
* Adjust mypy config to warn us about unreachable return paths and useless expressions.
* Add the __annotations__ property to INTERNAL_ONLY_PROPERTIES so that it is not part of the config schema testing
* Remove redundant dependencies `typing` and `typing-extensions` which are NOOP after Python 3.5
* Remove mypy-extensions which only provides a TypedDict implementation but has not been updated since 2019.
* updated mypy to 0.910 which requires all types packages to be installed manually.
* exclude local-dev from type checking until core team can suggest an outcome for __init__.py duplicate packages
* re-add typing dependency which will be needed until Python 3.9
* ignore .mypy_cache
* add mypy stub for features module to replace inline definitions
* import annotations eager evaluation in billing.py as it was required to reference a class declared later in the module.
* remove the type definition of V1ProtocolSteps/V2ProtocolSteps to make tox happy
Allows forward slashes to be used in repository names according to
https://docs.docker.com/docker-hub/repos/.
NOTE: This change simply allows the use of "/" in repository
names needed for certain Openshift use cases. This does not implement
any new permission model for nested paths. i.e A repository with a
nested path is treated as a single repository under a _single_
namespace.
RepoMirrorConfig in the current database migration version has a
non-null constraint on the internal_robot field, but the model in
database.py does not.
Updates the model to match the current database revision, and handles
delete api calls when there are mirrors still using the robot.
Also set a default test DATABASE_SECRET_KEY when generating the test.db
* Change verbs to use a DerivedStorageForManifest table instead of DerivedStorageForImage
This allows us to deprecate the DerivedStorageForImage table.
Fixes https://issues.redhat.com/browse/PROJQUAY-519
* Change uploaded blob tracking to use its own table and deprecate
RepositoryTag
* Start recording the compressed layers size and config media type on the
manifest row in the database
NOTE: This change includes a database migration which will *lock* the
manifest table
* Change tag API to return the layers size from the manifest
* Remove unused code
* Add new config_media_type field to OCI types
* Fix secscan V2 test for us no longer writing temp images
* Remove unused uploading field
* Switch registry model to use synthetic legacy images
Legacy images are now (with exception of the V2 security model) read from the *manifest* and sythensized in memory. The legacy image IDs are generated realtime based on the hashids library. This change also further deprecates a bunch of our Image APIs, reducing them to only returning the image IDs, and emptying out the remaining metadata (to avoid the requirement of us loading the information for the manifest from storage).
This has been tested with our full clients test suite with success.
* Add a backfill worker for manifest layers compressed sizes
* Change image tracks into manifest tracks now that we no longer have
manifest-less tags
* Add back in the missing method
* Add missing joins to reduce extra queries
* Remove unnecessary join when looking up legacy images
* Remove extra hidden filter on tag queries
* Further DB improvements
* Delete all Verbs, as they were deprecated
* Add back missing parameter in manifest data type
* Fix join to return None for the robot if not defined on mirror config
* switch to using secscan_v4_model for all indexing and remove most of secscan_v2_model code
* Add a missing join
* Remove files accidentally re-added due to rebase
* Add back hashids lib
* Rebase fixes
* Fix broken test
* Remove unused GPG signer now that ACI conversion is removed
* Remove duplicated repomirrorworker
* Remove unused notification code for secscan. We'll re-add it once Clair
V4 security notifications are ready to go
* Fix formatting
* Stop writing Image rows when creating manifests
* Stop writing empty layer blobs for manifests
As these blobs are shared, we don't need to write ManifestBlob rows
for them
* Remove further unused code
* Add doc comment to _build_blob_map
* Add unit test for synthetic V1 IDs
* Remove unused import
* Add an invalid value test to synthetic ID decode tests
* Add manifest backfill worker back in
Seems to have been removed at some point
* Add a test for cached active tags
* Rename test_shared to not conflict with another same-named test file
Pytest doesn't like having two test modules with the same name
* Have manifestbackfillworker also copy over the config_media_type if present
Co-authored-by: alecmerdler <alecmerdler@gmail.com>
