1
0
mirror of https://github.com/quay/quay.git synced 2026-01-29 08:42:15 +03:00
Commit Graph

221 Commits

Author SHA1 Message Date
Marcus Kok
2ab7dc29f4 marketplace: add support for quantity from subscriptions api (PROJQUAY-6551) (#2633)
* Adds handling for when a subscription returned from the subscription watch api has a quantity greater than 1. Number of private repos should be correctly calculated using the quantity.

* Updates ui so that subscriptions can only be added to an org as a group, i.e. a subscription with quantity = 2 cannot be split across organizations.
2024-01-29 15:21:30 -05:00
Marcus Kok
54bc56d5be marketplace: return array of valid subscriptions when looking up subscription (PROJQUAY-6551) (#2613)
return array of valid subscriptions when looking up subscription
2024-01-17 10:07:29 -05:00
Daniel Messer
e8ff33e728 logs: add failure logging for login, push, pull and delete events (PROJQUAY-5411) (#1903)
* add login failure logging

Signed-off-by: dmesser <dmesser@redhat.com>

* move failure logging into credential validation

Signed-off-by: dmesser <dmesser@redhat.com>

* more precise tracking of affected users

Signed-off-by: dmesser <dmesser@redhat.com>

* fix indent

Signed-off-by: dmesser <dmesser@redhat.com>

* differentiate robots with wrong credentials

Signed-off-by: dmesser <dmesser@redhat.com>

* don't audit failures by default

Signed-off-by: dmesser <dmesser@redhat.com>

* discrete failure tracking for logins, push, pulls and deletes

Signed-off-by: dmesser <dmesser@redhat.com>

* refine log metadata

Signed-off-by: dmesser <dmesser@redhat.com>

* login failure log visualization

Signed-off-by: dmesser <dmesser@redhat.com>

* properly use data model

Signed-off-by: dmesser <dmesser@redhat.com>

* fix unit test bug

Signed-off-by: dmesser <dmesser@redhat.com>

* track non-existing repos differently

Signed-off-by: dmesser <dmesser@redhat.com>

* log view visualization of failed pushes and pulls

Signed-off-by: dmesser <dmesser@redhat.com>

* ensure all tests are conducted with failure logging

Signed-off-by: dmesser <dmesser@redhat.com>

* additional unicode protection

Signed-off-by: dmesser <dmesser@redhat.com>

* python black formatting

Signed-off-by: dmesser <dmesser@redhat.com>

* add cypress test data

Signed-off-by: dmesser <dmesser@redhat.com>

* add safety checks for ascii conversion attempts

Signed-off-by: dmesser <dmesser@redhat.com>

* adjusting unit test with correct error message

Signed-off-by: dmesser <dmesser@redhat.com>

* update to alembic head

Signed-off-by: dmesser <dmesser@redhat.com>

* add standard oauth token metadata in audit

Signed-off-by: dmesser <dmesser@redhat.com>

* update alembic head

Signed-off-by: dmesser <dmesser@redhat.com>

* correct field name

Signed-off-by: dmesser <dmesser@redhat.com>

* formatting

Signed-off-by: dmesser <dmesser@redhat.com>

* bump alembic head

Signed-off-by: dmesser <dmesser@redhat.com>

* refactor auth logging imports

Signed-off-by: dmesser <dmesser@redhat.com>

* bump alembic head

Signed-off-by: dmesser <dmesser@redhat.com>

* formatting

Signed-off-by: dmesser <dmesser@redhat.com>

* restore module

Signed-off-by: dmesser <dmesser@redhat.com>

* pre-commit fixes

Signed-off-by: dmesser <dmesser@redhat.com>

* adding missing default

Signed-off-by: dmesser <dmesser@redhat.com>

* bump alembic head

Signed-off-by: dmesser <dmesser@redhat.com>

* update test data

Signed-off-by: dmesser <dmesser@redhat.com>

* refactoring to save db calls

Signed-off-by: dmesser <dmesser@redhat.com>

* fix unit tests

Signed-off-by: dmesser <dmesser@redhat.com>

* handle unicode conversion errors on email look up

Signed-off-by: dmesser <dmesser@redhat.com>

* bump alembic head

Signed-off-by: dmesser <dmesser@redhat.com>

* proper debug logging and conditional db calls

Signed-off-by: dmesser <dmesser@redhat.com>

* omit wildcard import

Signed-off-by: dmesser <dmesser@redhat.com>

* re-add import

Signed-off-by: dmesser <dmesser@redhat.com>

---------

Signed-off-by: dmesser <dmesser@redhat.com>
2024-01-16 16:46:20 +01:00
Marcus Kok
26451766ad marketplace: make sure customer id from api is returned as an int (PROJQUAY-233) (#2590)
* make sure customer id from api is returned as an int
2024-01-08 15:41:34 -05:00
Marcus Kok
1c893baba5 marketplace: update reconciliationworker to use webCustomerId instead of ebsAccountNumber (PROJQUAY-233) (#2582)
* update reconciliationworker to use webCustomerId instead of
ebsAccountNumber

* fix reconciler where it was incorrectly using the ebsAccountNumber to
  create subscriptions
* add job to reconciler so that it reconciles different ids between the
  database and the user api
* separate skus to be used by billing and skus to be used by reconciler
2024-01-05 16:15:37 -05:00
Brandon Caton
ac221a601d ui: implementing build history page (PROJQUAY-6293) (#2523)
implementing build history page
2023-12-13 16:03:04 -05:00
Marcus Kok
1bf3f44820 marketplace: update user ebs number lookup to find personal account numbers (PROJQUAY-233) (#2545)
* update user ebs number lookup to find account numbers with type = person
2023-12-06 11:10:29 -05:00
Michaela Lang
0dfa72d036 user(robots): disallow robot login and create 2nd (PROJQUAY-5968) (#2483)
* Fixed typo, added ROBOTS_WHITELIST as without the ROBOTS_DISALLOW does not make sense

* added ROBOTS_WHITELIST default to config.py
2023-12-05 17:31:08 +01:00
Marcus Kok
4c2433413c marketplace: add logging for user api (PROJQUAY-233) (#2513)
* add logging for user api
2023-12-01 10:29:05 -05:00
Marcus Kok
3aa2121326 marketplace: return None if subscription api times out (PROJQUAY-5363) (#2437)
* handles `ReadTimeout` exception by returning `None` for subscription API
2023-10-27 09:27:16 -04:00
Marcus Kok
685cbef1a2 marketplace: use get method for ebsAccountNumber lookup (PROJQUAY-6219) (#2417)
use get method for ebsAccountNumber lookup
2023-10-18 14:33:51 -04:00
Syed Ahmed
27f1699b72 ui: Add delay after write operations (PROJQUAY-6146) (#2400)
This is useful for cases where reads and writes are
going to different DBs and there is a replication lag
between them
2023-10-12 12:45:16 -04:00
Brandon Caton
0e496b46a5 autoprune: add initial setup for the autoprune feature (PROJQUAY-6094) (#2277)
Adds the auto-prune worker, database models, and feature flag.
2023-10-05 13:37:36 -04:00
dependabot[bot]
d2e5a69b26 build(deps): bump pyjwt from 2.4.0 to 2.8.0 (#2166)
* build(deps): bump pyjwt from 2.4.0 to 2.8.0

Bumps [pyjwt](https://github.com/jpadilla/pyjwt) from 2.4.0 to 2.8.0.
- [Release notes](https://github.com/jpadilla/pyjwt/releases)
- [Changelog](https://github.com/jpadilla/pyjwt/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/jpadilla/pyjwt/compare/2.4.0...2.8.0)

---
updated-dependencies:
- dependency-name: pyjwt
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Expect new messages

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Oleg Bulatov <oleg@bulatov.me>
2023-10-03 14:08:00 +02:00
Oleg Bulatov
84fa795ae7 chore: fix isort config and remove isort: skip_file (#2196)
* chore: pass config to isort as it doesn't always detect it

* chore: mark package "test" as local, not stdlib

* chore: remove "isort: skip_file"

* chore: fix app in test_load_security_information

* chore: fix app in test_notification

* chore: fix app in test_index_report
2023-09-21 11:46:03 -04:00
Michaela Lang
67028af9e2 user(robots): disallow robot login and create (PROJQUAY-5968) (#2155)
* add Postgresql client certificate authentication option

* user(robots): disallow robot login and create PROJQUAY-5968

Add a config flag `ROBOTS_DISALLOW` to turn off Robot login and creation
in Quay for all accounts.

with the Flag set existing Robots cannot login anymore
```
$ podman login -u milang+test2 -p W7B...HQA quay.example.com
Error: logging into "quay.example.com": invalid username/password
```

This behavior will be reported in the logs accordingly
```
gunicorn-registry stdout | 2023-08-24 19:08:01,907 [253] [WARNING] [auth.credentials] Failed to validate credentials for robot milang+test2: Robot account has been disabled. Please contact your administrator.
```

as well as in the UI when creating a Robot Account which will display
the `Robot account has been disabled. Please contact your
administrator.` message.

* Revert "add Postgresql client certificate authentication option"
  need to branch for PR (stupid me)

This reverts commit f8f54eead2.

* fixed error message to proper english and added schema definition for the option
2023-09-20 13:07:41 +02:00
Kenny Lee Sin Cheong
72f7c64ed6 chore: update werkzeug and related package versions (PROJQUAY-5098) (#1982)
* chore: update werkzeug and related package versions (PROJQUAY-5098)

Path converter related change reference: https://github.com/pallets/werkzeug/issues/2506

* Update query count
2023-09-12 11:51:09 -04:00
Brandon Caton
eede634af9 config: updating GeoLite2-Country (PROJQUAY-6023) (#2218)
updating GeoLite2-Country.mmdb
2023-09-11 11:42:09 -04:00
Marcus Kok
e44783fe19 billing: Assign SKU to org (PROJQUAY-5363) (#1989)
* add migration for orgrhskus table

* add endpoints for managing and listing skus bound to an org

* create checks in billing flow to look for org-bound skus

* refactor RH marketplace api objects to be more usable in tests

* update cypress test db data and exclude it from pre-commit hook formatting
2023-08-25 14:52:54 -04:00
Kenny Lee Sin Cheong
5f63b3a7bb chore: drop deprecated tables and remove unused code (PROJQUAY-522) (#2089)
* chore: drop deprecated tables and remove unused code

* isort imports

* migration: check for table existence before drop
2023-08-25 12:17:24 -04:00
Kenny Lee Sin Cheong
8d0ee3865e secscan: fix metrics to track all request types to secscan service (PROJQUAY-3839) (#2108) 2023-08-11 11:33:02 -04:00
Ivan Bazulic
20845a136f chore: Remove reference to Image table from the backfill replication script (#2110)
The `image` table is deprecated and also not used in the backfill replication script. This PR removes the reference left from the previous PR.
2023-08-10 12:09:10 -04:00
Jonathan King
faa0315a04 georep: Add error handling for remove location script (PROJQUAY-5878) (#2071)
- Gracefully handle errors that are thrown during remove location script
- This should handle scenarios where the location is not found or not provided, as well as other error cases
2023-08-02 12:14:26 -04:00
Kenny Lee Sin Cheong
0ea4826956 secscan: garbage collect manifests (#1663)
Garbage collect manifests no longer referenced in Quay from the
security scanner service.

Also moved quota related code from data/registry_model/ to data/model/
to avoid circular dependencies.
2023-07-06 11:59:47 -04:00
Oleg Bulatov
ea49bb53a3 chore: Bump mypy (#1962) 2023-06-28 13:47:32 +02:00
Marcus Kok
e622414280 config: Enable notifications for new indexes by default (PROJQUAY-5682) (#1993)
* config: Set feature flag default for new vulnerability notifications to True
(PROJQUAY-4659)

Change the default from `False` to `True` for
`FEATURE_SECURITY_SCANNING_NOTIFY_ON_NEW_INDEX`.

Since this flag addresses a bug, it should be enabled by default.

* add mock return values for unit tests
2023-06-27 16:01:43 +02:00
Syed Ahmed
2371c4d605 api: allow robot token creation with a pre-defined token (PROJQUAY-5414) (#1972)
* api: allow robot token creation with a pre-defined token (PROJQUAY-5414)

This is for usecases where we want to explicitly create a robot token
with a pre-defined token in case of migration events when we migrate
from one quay to another, we can re-use the same robot token to
avoid resetting it in all places it is used
2023-06-23 15:53:30 -04:00
Ivan Bazulic
d7864ed4ee ui: Add custom TOS support (PROJQUAY-5648) (#1973)
* ui: Add custom TOS support
Enable adding of a custom TOS in the Quay footer for on-premise installations via the optional `TERMS_OF_SERVICE` config flag.
If the flag is not defined, the TOS field will not be visible on Quay rendering. Also changes the behaviour of the discovery endpoint to include custom terms of service instead of pointing to Quay.io terms of service for all installations.

* Fix formatting errors

* Fix TOS page for Quay.io deployments

* Change TOS variable name to TERMS_OF_SERVICE_URL for better clarity

* Removed reference to Quay.io from the discovery endpoint description
2023-06-19 11:44:52 -04:00
Daniel Messer
e1e8dc8efb audit: ignore errors due to read-only mode in audit logs (PROJQUAY-5598) (#1928)
* ignore errors due to read-only mode in audit logs

Signed-off-by: dmesser <dmesser@redhat.com>

* remove superfluous ReadOnlyMode handling

Signed-off-by: dmesser <dmesser@redhat.com>

* add debug log messae

Signed-off-by: dmesser <dmesser@redhat.com>

---------

Signed-off-by: dmesser <dmesser@redhat.com>
2023-06-16 12:08:27 +02:00
Kenny Lee Sin Cheong
d48df9b251 chore: deprecate image table support (PROJQUAY-522) (#1841)
Remove code related to deprecated image table
2023-06-14 10:54:54 -04:00
Jonathan King
88fd1bae9a geo-rep: Initialize features from config in util script (PROJQUAY-5627) (#1966)
- Features must be initialized in order to check if storage replication is enabled, so we need to import and intialize the config from inside the util script
2023-06-12 15:39:21 -04:00
Jonathan King
2d93fc7cd0 geo-rep: Add util script to remove geo-rep location and corresponding image locations (PROJQUAY-4995) (#1892)
- This removes first the ImageStoragePlacements that are associated with the location
- Once those are cleaned up, the ImageStorageLocation itself is removed from the database
2023-06-06 17:25:56 -04:00
Marcus Kok
de8c48fa28 marketplace: fixing allowed repo count (PROJQUAY-5513) (#1891)
* Aggregate stripe plans and subscriptions.

* Validate end date for subscriptions when fetching from marketplace.

* Check subscription returned from stripe api is non-null value when
finding stripe plan for sku.
2023-05-25 10:46:58 -04:00
Brandon Caton
6cf0a3531b quota: calculating registry size (PROJQUAY-5476) (#1879)
Allows superusers to trigger a calculation of the deduplicated registry size. A superuser can go to the organization panel of the superuser page and select Calculate to queue a calculation of the registry total. The total will only be calculated when requested. Includes warning to user of increase of database load when running calculation.
2023-05-24 17:10:40 -04:00
Sunandadadi
230fd24f3d Util: Reading new UI feedback form url from config parameter (PROJQUAY-5463) (#1902)
* Util: Reading feedback form url from config parameter (PROJQUAY-5463)

* updating form link for quay plugin
2023-05-24 10:43:25 -04:00
Harish Govindarajulu
490a6b2ce8 logs: Add SSL cert support and test coverage for splunk logging (PROJQUAY-4993) (#1878)
Signed-off-by: harishsurf <hgovinda@redhat.com>
2023-05-19 16:24:13 -04:00
Daniel Messer
41cd8330d0 logs: add audit log events for login/logout to Quay (PROJQUAY-2344) (#1866)
* add auditing of login and logout events

Signed-off-by: dmesser <dmesser@redhat.com>

* python black formatting

Signed-off-by: dmesser <dmesser@redhat.com>

* correctly identify username during

Signed-off-by: dmesser <dmesser@redhat.com>

* revert change to existing code

Signed-off-by: dmesser <dmesser@redhat.com>

* remove superfluous ip tracking

Signed-off-by: dmesser <dmesser@redhat.com>

* beautify login messages

Signed-off-by: dmesser <dmesser@redhat.com>

* remove unused import

Signed-off-by: dmesser <dmesser@redhat.com>

* move alembic patch to head

Signed-off-by: dmesser <dmesser@redhat.com>

* correctly log target namespace for robots

Signed-off-by: dmesser <dmesser@redhat.com>

* python black formatting

Signed-off-by: dmesser <dmesser@redhat.com>

* login event detection refinement

Signed-off-by: dmesser <dmesser@redhat.com>

* add missing visualization

Signed-off-by: dmesser <dmesser@redhat.com>

* enrich login event data

Signed-off-by: dmesser <dmesser@redhat.com>

* make login auditing configurable

Signed-off-by: dmesser <dmesser@redhat.com>

* make update-testdata

Signed-off-by: dmesser <dmesser@redhat.com>

---------

Signed-off-by: dmesser <dmesser@redhat.com>
2023-05-17 11:22:35 +02:00
Marcus Kok
c353946910 marketplace: fix path to api cert (PROJQUAY-5409) (#1870)
remove /auth from path to marketplace cert

secrets are mounted to /conf/stack by default
2023-05-10 07:37:30 -04:00
Marcus Kok
0a1c7fb22e marketplace: add reconciler (PROJQUAY-5320) (#1817)
marketplace: add reconciler (PROJQUAY-5320)
* check RH marketplace after stripe for private repo creation / changing visibility
* add reconciliation worker that creates RH subscriptions in marketplace for pre-existing stripe customers
2023-05-08 09:48:17 -04:00
Brandon Caton
a2c379d47c quota: Include blob deduplication in totals (PROJQUAY-3942) (#1751)
Allows for only unique blobs are counted at the namespace and repository level. Calculation includes manifest list sizes.
Add's the following internal configurations that default to true:
QUOTA_INVALIDATE_TOTALS: Invalidates calculated totals when FEATURE_QUOTA_MANAGEMENT is set to false
RESET_CHILD_MANIFEST_EXPIRATION: Resets the expiry for child manifests on push of the manifest list for immediate GC eligibility
PERMANENTLY_DELETE_TAGS: Enables features related to the permanent deletion of tags outside the configured time machine window
2023-05-01 16:40:01 -04:00
Marcus Kok
780bca5eeb locking: change log severity (PROJQUAY-5221) (#1820)
When worker fails to acquire lock make sure a log message is sent as a warning for better visibility when troubleshooting.
2023-04-10 09:56:48 -04:00
Harish Govindarajulu
be1bddfd6a feat: Allow action logs to be forwarded to Splunk (PROJQUAY-4993) (#1764)
Signed-off-by: harishsurf <hgovinda@redhat.com>
2023-03-30 13:33:55 -04:00
Marcus Kok
c3fc3a82ab secscan: send notifications for new indexes (PROJQUAY-4659) (#1813)
On new pushes, allow notifications being sent to user if vulnerabilities are found.
2023-03-29 13:17:52 -04:00
Syed Ahmed
98d89a1fce cors: check for request_origin being set (PROJQUAY-5213) (#1811)
Some browsers might not set the Origin header
in the request. Ignore the origin check in such
cases
2023-03-28 06:13:34 -04:00
Ivan Bazulic
b2a5b3abb7 ldap: Don't convert dashes to underscores in usernames (PROJQUAY-5253) (#1808)
* ldap: Don't convert dashes to underscores in usernames (PROJQUAY-5253)
When `FEATURE_USER_CONFIRMATION` is set to true and an LDAP username that has dashes inside is used, Quay will automatically change dashes to underscores. This breaks `LDAP_RESTRICTED_USER_FILTER`  since namespaces in Quay do not have to be the same as LDAP namespaces.
The change can only impact ancient Docker versions prior to version 1.5 where support for dashes in namespaces has been introduced.

* Fix validation test
2023-03-27 14:54:34 -04:00
Syed Ahmed
84abdba076 api: Allow setting multiple CORS_ORIGIN (PROJQUAY-5213) (#1791)
api: Allow setting multiple CORS_ORIGIN (PROJQUAY-5213)

multiple CORS Origin are required for integration with console.redhat
2023-03-22 11:15:59 -04:00
Modassar Rana
618945b607 Added support for IBMZ (#1737)
Updated support for Z
2023-02-17 15:04:04 +01:00
Syed Ahmed
0ae31c6ebc storage: Add MultiCDN storage provider (PROJQUAY-5048) (#1747)
* storage: Add MultiCDN storage provider (PROJQUAY-5048)

This storage provider can route to different underlying sub-providers
based on a critiera. Currently supported filters are source_ip and
namespace.

Example Config:

- MultiCDNStorage
- providers:
    TargetName1:
        - ProviderName1
        - porviderConfig1
    Targetname2:
        - ProviderName2
        - ProviderConfig2
  default_provider: TargetName1
  rules:
  - namespace: test
    continent: APAC
    target: TargetName2
2023-02-13 09:59:30 -05:00
Kenny Lee Sin Cheong
4aa84a528c secscan: fix string to int conversion (PROJQUAY-4395) (#1736) 2023-02-02 16:13:07 -05:00
Kenny Lee Sin Cheong
e1985942a6 secscan: handle non backfilled layers_compressed_size (PROJQUAY-4395) (#1734) 2023-02-02 15:04:01 -05:00