1
0
mirror of https://github.com/quay/quay.git synced 2026-01-29 08:42:15 +03:00
Commit Graph

211 Commits

Author SHA1 Message Date
Marcus Kok
2ab7dc29f4 marketplace: add support for quantity from subscriptions api (PROJQUAY-6551) (#2633)
* Adds handling for when a subscription returned from the subscription watch api has a quantity greater than 1. Number of private repos should be correctly calculated using the quantity.

* Updates ui so that subscriptions can only be added to an org as a group, i.e. a subscription with quantity = 2 cannot be split across organizations.
2024-01-29 15:21:30 -05:00
Ivan Bazulic
46d1322ca7 api: Return layer compressed size with manifest request (PROJQUAY-6616) (#2627)
Previously, we weren't returning the total compressed layer size (in bytes) which is part of the manifest table. This makes it cumbersome to identify the real size of an image via `api/v1/repository/{LOCATION}/manifest/{manifestref}` endpoint since only individual layer sizes were returned and the total would then need to be manually summed over each individual layer. With this change, the API endpoint will return the compressed size of all layers if we're referencing a real image, if a reference points to an OCI index or a manifest list, the returned size will be 0.
2024-01-22 11:49:39 -05:00
Daniel Messer
e8ff33e728 logs: add failure logging for login, push, pull and delete events (PROJQUAY-5411) (#1903)
* add login failure logging

Signed-off-by: dmesser <dmesser@redhat.com>

* move failure logging into credential validation

Signed-off-by: dmesser <dmesser@redhat.com>

* more precise tracking of affected users

Signed-off-by: dmesser <dmesser@redhat.com>

* fix indent

Signed-off-by: dmesser <dmesser@redhat.com>

* differentiate robots with wrong credentials

Signed-off-by: dmesser <dmesser@redhat.com>

* don't audit failures by default

Signed-off-by: dmesser <dmesser@redhat.com>

* discrete failure tracking for logins, push, pulls and deletes

Signed-off-by: dmesser <dmesser@redhat.com>

* refine log metadata

Signed-off-by: dmesser <dmesser@redhat.com>

* login failure log visualization

Signed-off-by: dmesser <dmesser@redhat.com>

* properly use data model

Signed-off-by: dmesser <dmesser@redhat.com>

* fix unit test bug

Signed-off-by: dmesser <dmesser@redhat.com>

* track non-existing repos differently

Signed-off-by: dmesser <dmesser@redhat.com>

* log view visualization of failed pushes and pulls

Signed-off-by: dmesser <dmesser@redhat.com>

* ensure all tests are conducted with failure logging

Signed-off-by: dmesser <dmesser@redhat.com>

* additional unicode protection

Signed-off-by: dmesser <dmesser@redhat.com>

* python black formatting

Signed-off-by: dmesser <dmesser@redhat.com>

* add cypress test data

Signed-off-by: dmesser <dmesser@redhat.com>

* add safety checks for ascii conversion attempts

Signed-off-by: dmesser <dmesser@redhat.com>

* adjusting unit test with correct error message

Signed-off-by: dmesser <dmesser@redhat.com>

* update to alembic head

Signed-off-by: dmesser <dmesser@redhat.com>

* add standard oauth token metadata in audit

Signed-off-by: dmesser <dmesser@redhat.com>

* update alembic head

Signed-off-by: dmesser <dmesser@redhat.com>

* correct field name

Signed-off-by: dmesser <dmesser@redhat.com>

* formatting

Signed-off-by: dmesser <dmesser@redhat.com>

* bump alembic head

Signed-off-by: dmesser <dmesser@redhat.com>

* refactor auth logging imports

Signed-off-by: dmesser <dmesser@redhat.com>

* bump alembic head

Signed-off-by: dmesser <dmesser@redhat.com>

* formatting

Signed-off-by: dmesser <dmesser@redhat.com>

* restore module

Signed-off-by: dmesser <dmesser@redhat.com>

* pre-commit fixes

Signed-off-by: dmesser <dmesser@redhat.com>

* adding missing default

Signed-off-by: dmesser <dmesser@redhat.com>

* bump alembic head

Signed-off-by: dmesser <dmesser@redhat.com>

* update test data

Signed-off-by: dmesser <dmesser@redhat.com>

* refactoring to save db calls

Signed-off-by: dmesser <dmesser@redhat.com>

* fix unit tests

Signed-off-by: dmesser <dmesser@redhat.com>

* handle unicode conversion errors on email look up

Signed-off-by: dmesser <dmesser@redhat.com>

* bump alembic head

Signed-off-by: dmesser <dmesser@redhat.com>

* proper debug logging and conditional db calls

Signed-off-by: dmesser <dmesser@redhat.com>

* omit wildcard import

Signed-off-by: dmesser <dmesser@redhat.com>

* re-add import

Signed-off-by: dmesser <dmesser@redhat.com>

---------

Signed-off-by: dmesser <dmesser@redhat.com>
2024-01-16 16:46:20 +01:00
Sunandadadi
a7737722fc cache: adding caching on look_up repository (PROJQUAY-6472) (#2521)
* adding caching on look_up repository

* fixing query count in tests

* fixing tests

* isort fix
2024-01-16 09:58:56 -05:00
Marcus Kok
2a4ac09306 billing: marketplace UI (PROJQUAY-6551) (#2595)
* billing: marketplace UI

adds UI in billing section for managing user and org-bound skus

add more unit tests for org binding

changed endpoint for bulk attaching skus to orgs
2024-01-11 11:48:38 -05:00
Marcus Kok
1c893baba5 marketplace: update reconciliationworker to use webCustomerId instead of ebsAccountNumber (PROJQUAY-233) (#2582)
* update reconciliationworker to use webCustomerId instead of
ebsAccountNumber

* fix reconciler where it was incorrectly using the ebsAccountNumber to
  create subscriptions
* add job to reconciler so that it reconciles different ids between the
  database and the user api
* separate skus to be used by billing and skus to be used by reconciler
2024-01-05 16:15:37 -05:00
Brandon Caton
236e2fe4fd autoprune: adding audit logs to namespace autoprune policy API (PROJQUAY-6229) (#2431)
Add's audit logs to the CRUD endpoints for both user and organization namespace auto-prune policies.
2023-12-04 13:51:52 -05:00
Brandon Caton
aaba7539b9 api: adding nickname decorator to autoprune policy endpoints (PROJQUAY-6483) (#2524)
Adding nickname decorator to auto-prune policy endpoints
2023-12-04 09:14:41 -05:00
Sunandadadi
6fc77823da Revert "api: add caching for get_repository method (PROJQUAY-6472)" (#2517)
Revert "api: add caching for get_repository method (PROJQUAY-6472) (#2515)"

This reverts commit 9c89e843f3.
2023-12-01 09:07:19 -05:00
Sunandadadi
9c89e843f3 api: add caching for get_repository method (PROJQUAY-6472) (#2515)
* api: add caching for get_repository method (PROJQUAY-6472)

* fixing formatting

* using pickle to serialize and deserialize objects

* adding model_cache parameter to lookup_repository func for interface and registry_proxy_model

* adding test

* formatting fix
2023-11-30 12:46:28 -05:00
Kenny Lee Sin Cheong
0b6b1598de billing: apply free trial to stripe checkout sessions (PROJQUAY-6405) (#2491) 2023-11-15 16:38:55 -05:00
Sunandadadi
21e502f86e api: graceful error handling for robot acct already exists (PROJQUAY-6261) (#2436)
* Using try except to catch errors + added tests

* catching InvalidRobotException
2023-10-31 11:37:47 -04:00
Brandon Caton
e5a5e17814 api: accepting empty body for create robot endpoints (PROJQUAY-6224) (#2420)
accepting empty body for create robot endpoints
2023-10-19 09:39:32 +02:00
Brandon Caton
22c4bbfee5 autoprune: add auto-prune policy endpoints (PROJQUAY-6096) (#2393)
Add's the endpoints to create, update, delete, get, and list organization and user auto-prune policies.
2023-10-10 16:38:46 -04:00
Brandon Caton
2a672551fc quota: prevent tag creation on rejected manifest (PROJQUAY-3906) (#2247)
Currently if a new manifest is created that re-uses blobs already existing in the registry but not within the namespace it's possible for that manifest to be blocked if the newly referenced blobs exceed the quota limit. The issue being that the rejection only happens after the manifest has been created and tagged. This change blocks the manifest upload before the creation of the tag and prevents that tag from appearing in the UI and being pullable.

If the manifest upload get's rejected before the tag has been created for a new manifest, a temporary tag is created outside the time machine window so it is immediately available for GC.
2023-10-03 10:30:27 -04:00
Brandon Caton
dcdf132fce quota: removing extra calls to get namespace quotas (PROJQUAY-6048) (#2267)
Caches the result of retrieving the namespace quota limit.
2023-09-29 15:15:03 -04:00
Oleg Bulatov
84fa795ae7 chore: fix isort config and remove isort: skip_file (#2196)
* chore: pass config to isort as it doesn't always detect it

* chore: mark package "test" as local, not stdlib

* chore: remove "isort: skip_file"

* chore: fix app in test_load_security_information

* chore: fix app in test_notification

* chore: fix app in test_index_report
2023-09-21 11:46:03 -04:00
Syed Ahmed
54fcfd14f9 secscan: Cache clair vuln reports (PROJQUAY-6057) (#2245)
uses modelcache to for caching clair security report responses
2023-09-20 20:33:41 +00:00
Ivan Bazulic
abfde5b9d2 georeplication: Enqueue blobs for replication on manifest list pushes (PROJQUAY-5256) (#2212)
* georeplication: Enqueue blobs for replication on manifest list pushes (PROJQUAY-5256)
Previously, blob replication was only queued when normal manifests were pushed to the registry. This created problems with manifest lists who contain child manifests whose blobs were never queued. The end result was a huge discrepancy of storage usage between different defined storage engines.
With this fix, blobs will always be enqueued for replication regardless of the image type.

* Update `black` in `requirements-dev.txt` to match the version we use on Github
2023-09-12 13:58:08 -04:00
Kenny Lee Sin Cheong
72f7c64ed6 chore: update werkzeug and related package versions (PROJQUAY-5098) (#1982)
* chore: update werkzeug and related package versions (PROJQUAY-5098)

Path converter related change reference: https://github.com/pallets/werkzeug/issues/2506

* Update query count
2023-09-12 11:51:09 -04:00
Sunandadadi
c93b6d080d api: fix duplicate robot accounts (PROJQUAY-5931) (#2192)
* api: fix duplicate robot accounts (PROJQUAY-5931)

* fixing formating

* referecing named tuples by names
2023-09-01 12:03:33 -04:00
Marcus Kok
e44783fe19 billing: Assign SKU to org (PROJQUAY-5363) (#1989)
* add migration for orgrhskus table

* add endpoints for managing and listing skus bound to an org

* create checks in billing flow to look for org-bound skus

* refactor RH marketplace api objects to be more usable in tests

* update cypress test db data and exclude it from pre-commit hook formatting
2023-08-25 14:52:54 -04:00
Kenny Lee Sin Cheong
5f63b3a7bb chore: drop deprecated tables and remove unused code (PROJQUAY-522) (#2089)
* chore: drop deprecated tables and remove unused code

* isort imports

* migration: check for table existence before drop
2023-08-25 12:17:24 -04:00
Kenny Lee Sin Cheong
0ea4826956 secscan: garbage collect manifests (#1663)
Garbage collect manifests no longer referenced in Quay from the
security scanner service.

Also moved quota related code from data/registry_model/ to data/model/
to avoid circular dependencies.
2023-07-06 11:59:47 -04:00
Oleg Bulatov
ea49bb53a3 chore: Bump mypy (#1962) 2023-06-28 13:47:32 +02:00
Ivan Bazulic
d7864ed4ee ui: Add custom TOS support (PROJQUAY-5648) (#1973)
* ui: Add custom TOS support
Enable adding of a custom TOS in the Quay footer for on-premise installations via the optional `TERMS_OF_SERVICE` config flag.
If the flag is not defined, the TOS field will not be visible on Quay rendering. Also changes the behaviour of the discovery endpoint to include custom terms of service instead of pointing to Quay.io terms of service for all installations.

* Fix formatting errors

* Fix TOS page for Quay.io deployments

* Change TOS variable name to TERMS_OF_SERVICE_URL for better clarity

* Removed reference to Quay.io from the discovery endpoint description
2023-06-19 11:44:52 -04:00
Kenny Lee Sin Cheong
d48df9b251 chore: deprecate image table support (PROJQUAY-522) (#1841)
Remove code related to deprecated image table
2023-06-14 10:54:54 -04:00
Brandon Caton
e38d70f0cb api: add permanently delete tag usage log (PROJQUAY-5496) (#1887)
Now logging the permanently delete tag as a usage log.
2023-06-02 10:43:14 -04:00
Marcus Kok
b911d480cf api: allow manifests to be pulled by digest (PROJQUAY-5467) (#1877)
Add new parameter `allow_hidden` to `lookup_manifest_by_digest` method and set this to true on the manifest v2 endpoint.

Enables manifests to be pulled by digest, and fixes issues with recent versions of conftest being unable to push to quay.
2023-05-26 09:43:27 -04:00
Harish Govindarajulu
e349762d78 proxy: Allow anonymous pulls from registries (PROJQUAY-5273) (#1906)
This change allows validation of proxy cache config api to support anonymous pulls
when the registry returns a 401 since repo name is not known during initial proxy config

Signed-off-by: harishsurf <hgovinda@redhat.com>
2023-05-25 11:53:15 -04:00
Marcus Kok
de8c48fa28 marketplace: fixing allowed repo count (PROJQUAY-5513) (#1891)
* Aggregate stripe plans and subscriptions.

* Validate end date for subscriptions when fetching from marketplace.

* Check subscription returned from stripe api is non-null value when
finding stripe plan for sku.
2023-05-25 10:46:58 -04:00
Brandon Caton
6cf0a3531b quota: calculating registry size (PROJQUAY-5476) (#1879)
Allows superusers to trigger a calculation of the deduplicated registry size. A superuser can go to the organization panel of the superuser page and select Calculate to queue a calculation of the registry total. The total will only be calculated when requested. Includes warning to user of increase of database load when running calculation.
2023-05-24 17:10:40 -04:00
Brandon Caton
9adf2d8cf0 superuser: paginate user's list (PROJQUAY-4297) (#1881)
Allow loading of superuser user list for larger registries by paginating API call.
2023-05-18 11:20:30 -04:00
Brandon Caton
c505a6bae8 superuser: paginating superuser organization list (PROJQUAY-4297) (#1876)
Allows super user organization list to load for larger registries.
2023-05-17 14:49:09 -04:00
Daniel Messer
41cd8330d0 logs: add audit log events for login/logout to Quay (PROJQUAY-2344) (#1866)
* add auditing of login and logout events

Signed-off-by: dmesser <dmesser@redhat.com>

* python black formatting

Signed-off-by: dmesser <dmesser@redhat.com>

* correctly identify username during

Signed-off-by: dmesser <dmesser@redhat.com>

* revert change to existing code

Signed-off-by: dmesser <dmesser@redhat.com>

* remove superfluous ip tracking

Signed-off-by: dmesser <dmesser@redhat.com>

* beautify login messages

Signed-off-by: dmesser <dmesser@redhat.com>

* remove unused import

Signed-off-by: dmesser <dmesser@redhat.com>

* move alembic patch to head

Signed-off-by: dmesser <dmesser@redhat.com>

* correctly log target namespace for robots

Signed-off-by: dmesser <dmesser@redhat.com>

* python black formatting

Signed-off-by: dmesser <dmesser@redhat.com>

* login event detection refinement

Signed-off-by: dmesser <dmesser@redhat.com>

* add missing visualization

Signed-off-by: dmesser <dmesser@redhat.com>

* enrich login event data

Signed-off-by: dmesser <dmesser@redhat.com>

* make login auditing configurable

Signed-off-by: dmesser <dmesser@redhat.com>

* make update-testdata

Signed-off-by: dmesser <dmesser@redhat.com>

---------

Signed-off-by: dmesser <dmesser@redhat.com>
2023-05-17 11:22:35 +02:00
Marcus Kok
0a1c7fb22e marketplace: add reconciler (PROJQUAY-5320) (#1817)
marketplace: add reconciler (PROJQUAY-5320)
* check RH marketplace after stripe for private repo creation / changing visibility
* add reconciliation worker that creates RH subscriptions in marketplace for pre-existing stripe customers
2023-05-08 09:48:17 -04:00
Kenny Lee Sin Cheong
2d949b3b2e billing: allow changing subscription on empty stripe_id (PROJQUAY-5413) (#1857)
Fix to allow non-stripe user changing to free plan in order to convert
from user to organization.
2023-05-03 10:01:13 -04:00
Daniel Messer
37e4990b07 logs: Add audit logs for organization and user namespace activities (PROJQUAY-3482) (#1846)
* add auditing events for orgs

Signed-off-by: dmesser <dmesser@redhat.com>

* add database migrations

Signed-off-by: dmesser <dmesser@redhat.com>

* fix org delete race condition

Signed-off-by: dmesser <dmesser@redhat.com>

* action log visualization

Signed-off-by: dmesser <dmesser@redhat.com>

* audit superuser events

Signed-off-by: dmesser <dmesser@redhat.com>

* additional user auditing

Signed-off-by: dmesser <dmesser@redhat.com>

* audit log visualization refinement

Signed-off-by: dmesser <dmesser@redhat.com>

* consistent email change auditing

Signed-off-by: dmesser <dmesser@redhat.com>

* python black formatting

Signed-off-by: dmesser <dmesser@redhat.com>

* fix incorrect log kind

Signed-off-by: dmesser <dmesser@redhat.com>

* missing log kind in migration

Signed-off-by: dmesser <dmesser@redhat.com>

* python black formatting

Signed-off-by: dmesser <dmesser@redhat.com>

* even more python black formatting

Signed-off-by: dmesser <dmesser@redhat.com>

* remove unnecessary import

Signed-off-by: dmesser <dmesser@redhat.com>

* bump alembic revision to head

Signed-off-by: dmesser <dmesser@redhat.com>

* alembic metadata change

Signed-off-by: dmesser <dmesser@redhat.com>

---------

Signed-off-by: dmesser <dmesser@redhat.com>
2023-05-03 13:52:10 +02:00
Brandon Caton
a2c379d47c quota: Include blob deduplication in totals (PROJQUAY-3942) (#1751)
Allows for only unique blobs are counted at the namespace and repository level. Calculation includes manifest list sizes.
Add's the following internal configurations that default to true:
QUOTA_INVALIDATE_TOTALS: Invalidates calculated totals when FEATURE_QUOTA_MANAGEMENT is set to false
RESET_CHILD_MANIFEST_EXPIRATION: Resets the expiry for child manifests on push of the manifest list for immediate GC eligibility
PERMANENTLY_DELETE_TAGS: Enables features related to the permanent deletion of tags outside the configured time machine window
2023-05-01 16:40:01 -04:00
Sunandadadi
5c34296920 API/UI: Filtering of tags API through query parameter (PROJQUAY-5362) (#1839)
* API/UI: Filtering of tags API through query parameter (PROJQUAY-5362)

* Changing syntax of query param to add operation + added propagation of filtering from new UI

* added exception to return 400 on incorrect syntax

* Added tests to test filtering of /tags endpoint

* Minor fixes
2023-04-28 16:16:41 -04:00
Syed Ahmed
2db3b186f9 ui: add support for exposing quay UI as a dynamic plugin (PROJQUAY-3203) (#1799)
* ui: add support for exposing quay UI as a dynamic plugin (PROJQUAY-3203)

* Introduces a new SSO JWT based auth for client side Oauth
* Adds a new entrypoint component for the UI without topnav and sidenav for plugin
* Adds webpack config to build dynamic plugin
2023-04-20 19:05:07 -04:00
Kenny Lee Sin Cheong
e7a7b4a050 billing: fallback to cards api if paymentmethod is not set (PROJQUAY-5129) (#1826) 2023-04-13 12:36:19 -04:00
Kenny Lee Sin Cheong
89725309be billing: update Stripe checkout to support 3DS (PROJQUAY-5129) (#1818)
Update Stripe checkout in order to support auth requirements from
banks.
2023-04-11 14:41:37 -04:00
Harish Govindarajulu
be1bddfd6a feat: Allow action logs to be forwarded to Splunk (PROJQUAY-4993) (#1764)
Signed-off-by: harishsurf <hgovinda@redhat.com>
2023-03-30 13:33:55 -04:00
Kenny Lee Sin Cheong
d05c32b9d1 billing: update default subscription payment behavior (#1778)
Update stripe payment behavior to 'default_incomplete'.
2023-03-08 12:19:05 -05:00
Kenny Lee Sin Cheong
16e5321108 permissions: lazy-load superuser permissions (PROJQUAY-5117) (#1761) 2023-03-06 10:29:55 -05:00
Kenny Lee Sin Cheong
23b39720d1 registry: add option to ignore unknown mediatypes (PROJQUAY-5018) (#1750) 2023-02-09 14:10:05 -05:00
Kenny Lee Sin Cheong
80fdb92462 secscan: add config to limit manifests with layer size too large to index (PROJQUAY-4957) (#1733)
Also removes deprecated Clair v2 configs.
2023-01-31 15:50:16 -05:00
Syed Ahmed
68844dbf5e login: Re-raise the export compliance exception on RHSSO (prod) (PROJQUAY-4844) (#1726)
Pass the correct arg quay_username
2023-01-24 17:11:54 -05:00
Kenny Lee Sin Cheong
6e8e2d2fe7 chore: remove deprecated appr code (PROJQUAY-4992) (#1718) 2023-01-24 10:11:04 +01:00