1
0
mirror of https://github.com/quay/quay.git synced 2026-01-26 06:21:37 +03:00
Commit Graph

71 Commits

Author SHA1 Message Date
Joseph Schorr
59bc23313c Add additional detection for transitive deletes and fix those found (#281)
We were not testing for transitive deletes when performing *User*
deletion, which led us to have a few
2020-03-25 15:58:12 -04:00
Joseph Schorr
3d2aadf91c Update the manifest builder for V1 API to not construct Image rows (#280)
Instead, we now simply save all the Image information into the
in-memory session and then construct the manifest directly at the end.

Fixes https://issues.redhat.com/browse/PROJQUAY-513
2020-03-25 15:05:17 -04:00
Joseph Schorr
dce80ab26d Fix support for empty manifests in V2_2 (#278)
Fixes https://issues.redhat.com/browse/PROJQUAY-510
2020-03-24 11:26:08 -04:00
Joseph Schorr
49862a9eda Move shared image modules into a shared package in prep for adding OCI (#271) 2020-03-20 12:33:43 -04:00
Joseph Schorr
79ca6f28cf Remove the shared registry model impl now that we have a single (#275)
registry model
2020-03-19 16:57:43 -04:00
Joseph Schorr
8701577cff Have the RepositoryActionCount worker cleanup old rows in RAC (#274)
We don't make use of any action counts older than a year, so this
change will have the worker remove old rows, one month (roughly) at
a time
2020-03-18 17:00:14 -04:00
Alec Merdler
7b827d953c add new 'ManifestSecurityStatus' database model for Clair v4 (#216) 2020-03-17 15:31:15 -07:00
Joseph Schorr
a32a1d58d9 Small changes missing from the previous cleanup PR (#270) 2020-03-17 16:40:57 -04:00
Joseph Schorr
0844256a71 Major post-OCI cleanup (#225)
* Remove V3_UPGRADE_MODE

* Remove tag backfill worker and all callers to tag backfill and upgrades

* Change initdb to create all test data via the manifest builder, rather than manually via legacy images

* Convert various code paths to use the registry_model where we previously did not do so

* Convert the GC test suite to build via the manifest builder

* Delete an old, unused tool

* Delete the Pre OCI registry model

* Add additional error handling to the manifest creation code path

* Add additional error handling to the OCI tag creation code path

* Change how we respond to invalid manifest content types to better handle unknowns

* Change legacy secscan test suite to use the registry model

* Change the repo build badge to use the registry model (also fixes a bug)

* Delete now-unused data model code

* Remove old model adjustment code from OCI model

* Mark older data models as deprecated which will prevent new rows from being inserted

* Remove references to old registry test suite from various testing files

* Remove tag backfill worker (again; got re-added during rebase)

* Move all deprecated model checks into a central function

* Make data_migration more Pythonic

* Small requested fixes to Tag module styling

* Have tag backfill worker fail to migrate if there are TagManifest's

Since this backfill should only be called in future releases for empty models, this should catch someone attempting to upgrade from a too-old version

* Remove labelbackfillworker as it is no longer needed

* Remove unused invalid import

* Reimplement the tag test for the remaining method used
2020-03-17 13:27:29 -04:00
Joseph Schorr
5619452776 Add ability to filter user lookups in LDAP user system (#252)
This allows customers to specify additional restrictions on the users
that are allowed to be used

Fixes https://issues.redhat.com/browse/PROJQUAY-226
2020-03-16 11:38:14 -04:00
Joseph Schorr
7b8d96a250 Ensure that temporary tags cannot be written for repositories marked (#258)
for deletion

The purge code assumes that once it has deleted tags, no new ones will
be added, so we need to enforce that state
2020-03-12 15:41:54 -04:00
Jake Moshenko
7f56b221e8 Gen migrations (#240)
* Move truthy_bool to a neutral location.

* Add dba operator migration version generation code.
Change existing migrations to use an injected and wrapped op.

* Generate databasemigration hints files for all existing migrations.

* Reformat with black.
2020-03-11 13:54:08 -04:00
Joseph Schorr
04cf7cc2c3 Catch infinite loops in GC purge code and raise an exception instead (#257)
If we have a bug that causes the GC's assumptions to be wrong (such
as an Image added after a purge began), then the purge loop can enter a
state where it will never halt. This change detects that condition and
raises an exception so we can properly investigate the bug.

Fixes https://issues.redhat.com/browse/PROJQUAY-452
2020-03-10 17:30:12 -04:00
Joseph Schorr
5b50258897 Change the lookup of active orgs to skip the unnecessary heavy filter (#249) 2020-03-04 16:50:17 -05:00
Joseph Schorr
45e6573401 Small worker fixes (#248)
* Fix a copy-paste error within the global prom stats worker

Also adds a test

* Fix queueitem deletion metric reporting
2020-03-04 16:28:10 -05:00
Joseph Schorr
424c1a19d7 Have the BlobUpload cleanup worker run on a single instance only (#239)
Instead of running simultaneously across multiple nodes, we change the
worker to run only from a single instance. This is better for the DB and
the previous behavior was not necessary given the size of the BlobUpload
table.

Fixes https://issues.redhat.com/browse/PROJQUAY-365
2020-02-27 13:16:40 -05:00
Joseph Schorr
1b023b8c2f Fix performance of the global prom stats worker queries (#238)
- Change the counts to use the MySQL information schema for estimates
- Change the lookup of active users to skip the unnecessary heavy filter
2020-02-27 13:16:35 -05:00
Joseph Schorr
d0f2deb6db Merge pull request #227 from josephschorr/joseph.schorr/PROJQUAY-220/tag-history-future
Hide all future tag expirations in the timeline, by default
2020-02-19 14:49:43 -05:00
Joseph Schorr
fa35777511 Hide all future tag expirations in the timeline, by default
Fixes https://issues.redhat.com/browse/PROJQUAY-220
2020-02-17 13:40:36 -05:00
Joseph Schorr
ca2c2c67ea Add pre-deletion of RepositoryAuthorizedEmail rows 2020-02-14 16:26:50 -05:00
Joseph Schorr
11e4d96130 Handle integrity errors being raised in v2 auth
Instead of raising an unhandled exception, we now check for the repository during the create process. If it was created since we last checked, we simply use the reference. Otherwise, we return an error to the user.

Fixes https://issues.redhat.com/projects/PROJQUAY/issues/PROJQUAY-289
2020-02-13 13:48:05 -05:00
Alec Merdler
c688f3a05e Merge pull request #180 from alecmerdler/fix-docstrings
Fix All the Docstrings
2020-02-06 21:06:43 -08:00
alecmerdler
3dd5f045ff fix all the docstrings 2020-02-05 19:55:07 -08:00
Joseph Schorr
7753b5be54 Pre-flight the cleanup of all big tables referencing a purged repo
This should prevent the final deletion transaction from timing out
2020-02-04 13:23:11 -05:00
Joseph Schorr
2d15282129 Merge pull request #212 from josephschorr/extra-storage-verification
Add extra verification on storage GC to ensure we don't end up with dangling ImageStorage rows
2020-01-31 16:43:52 -05:00
Joseph Schorr
e71ce8c811 Add extra verification on storage GC to ensure we don't end up with dangling ImageStorage rows
Also adds an optimization to avoid GCing specialized blobs
2020-01-31 14:49:26 -05:00
Joseph Schorr
51e35e01cb Add additional tests for the new V2 secscan wrapper
Should help us catch typos in the future
2020-01-31 14:30:35 -05:00
Joseph Schorr
21b149325a Additional fixes for config load failure in manifest creation
- Add additional retries and slightly increase the delay time
- Add a proper exception returned on failure
- Add a test to verify that we raise the proper exception
2020-01-31 11:46:48 -05:00
Joseph Schorr
43d2d3e5fe Fix broken enum value in secscan model 2020-01-31 11:25:57 -05:00
Joseph Schorr
36729c7672 Add a retry and proper error handling to the retriever
Some storage engines are eventually consistent, so we add a small number of retries, and we return `None` properly if the file could not be loaded
2020-01-30 16:06:48 -05:00
Joseph Schorr
59deab00ec Catch IntegrityError's in a better way and log
Currently, if we see an IntegrityError, we assume its because the manifest already exists. However, there appears to be a corner case where this is not the case. This change allows us to catch that additional exception, log the original issue, and return a better error to the user

Fixes https://issues.redhat.com/browse/PROJQUAY-219
2020-01-30 15:36:05 -05:00
Joseph Schorr
89ec88c3c1 Merge pull request #175 from josephschorr/joseph.schorr/PROJQUAY-177/abstract-sec-api
Place all security scanner access behind a new data model interface
2020-01-29 16:29:21 -05:00
Joseph Schorr
d9d6083ad6 Place all security scanner access behind a new data model interface
This is in preparation for the beginning of support for the new API for ClairCore (V4)

Fixes https://issues.redhat.com/browse/PROJQUAY-177
2020-01-29 13:02:14 -05:00
Joseph Schorr
51f6afff6f Merge pull request #119 from josephschorr/joseph.schorr/PROJQUAY-124/async-repo-delete
Change repository deletion to occur in the background
2020-01-28 13:21:47 -05:00
Joseph Schorr
55a214417f Fix the robot backfill for encrypted tokens if a federated login is missing 2020-01-27 16:11:30 -05:00
Joseph Schorr
6db1910a97 Change repository deletion to occur in the background
Fixes https://issues.redhat.com/browse/PROJQUAY-124
2020-01-27 14:16:01 -05:00
Kurtis Mullins
1b2d29d1e8 Revert "Revert "Remove the active migration for encrypted tokens now that it is complete"" 2020-01-27 11:22:22 -05:00
Joseph Schorr
e129fbac32 Merge pull request #168 from josephschorr/joseph.schorr/PROJQUAY-178/label-fixes
Fix label handling for empty labels
2020-01-22 11:20:49 -05:00
Kurtis Mullins
90d0dbf3fb Revert "Remove the active migration for encrypted tokens now that it is complete" 2020-01-21 11:53:54 -05:00
Joseph Schorr
883a415dc4 Remove the active migration for encrypted tokens now that it is complete
Fixes https://issues.redhat.com/browse/PROJQUAY-185
2020-01-20 14:51:51 -05:00
Joseph Schorr
06e1d12332 Fix bitbucket trigger creation around access tokens
Fixes https://issues.redhat.com/browse/PROJQUAY-186
2020-01-20 10:42:04 -05:00
Joseph Schorr
b87374e296 Merge pull request #163 from josephschorr/joseph.schorr/PROJQUAY-170/appr-limits
Add a configurable limit for app registry responses
2020-01-16 16:52:15 -05:00
Joseph Schorr
d97a1f6d90 Skip empty labels in label creation
This can happen if a Dockerfile contains a LABEL statement without a key value. In this case, we simply ignore the label and don't add it to our table, as it has no defined key.
2020-01-16 16:33:57 -05:00
Joseph Schorr
345bc6a155 Add missing descriptions to create manifest exceptions around labels
This ensures the error is raised to the user
2020-01-16 16:09:05 -05:00
Joseph Schorr
945a18052a Add a configurable limit for app registry responses
https://issues.redhat.com/browse/PROJQUAY-170
2020-01-16 10:48:53 -05:00
Joseph Schorr
095c5dcf64 Merge pull request #158 from josephschorr/joseph.schjorr/PROJQUAY-168/fix-migration
Small fixes for the encrypted token migration
2020-01-16 10:44:06 -05:00
alecmerdler
d2787bb010 add test for FieldEncrypter.decrypt_field 2020-01-15 14:13:51 -08:00
Joseph Schorr
52594e0a17 Small fixes for the encrypted token migration
Fixes the migration ID in one of the phases, and adds an extra assertion and checks in the robot migration

Fixes https://issues.redhat.com/browse/PROJQUAY-168
2020-01-15 13:30:33 -05:00
Jimmy Zelinskie
28a5851025 Merge pull request #110 from jzelinskie/consistent-metrics
metrics: push/pull bytes metrics named consistent
2020-01-10 15:06:59 -05:00
Jimmy Zelinskie
2e3424d623 metrics: push/pull bytes metrics named consistent 2020-01-10 13:44:22 -05:00