docker/quay
1
0
Fork 0
mirror of https://github.com/quay/quay.git synced 2025-07-12 17:41:46 +03:00
Code Activity

Deprecate Image rows and move to in-memory synthesized legacy images [Python 3] (#442)

Browse Source 
* Change verbs to use a DerivedStorageForManifest table instead of DerivedStorageForImage

This allows us to deprecate the DerivedStorageForImage table.

Fixes https://issues.redhat.com/browse/PROJQUAY-519

* Change uploaded blob tracking to use its own table and deprecate
RepositoryTag

* Start recording the compressed layers size and config media type on the
manifest row in the database

NOTE: This change includes a database migration which will *lock* the
manifest table

* Change tag API to return the layers size from the manifest

* Remove unused code

* Add new config_media_type field to OCI types

* Fix secscan V2 test for us no longer writing temp images

* Remove unused uploading field

* Switch registry model to use synthetic legacy images

Legacy images are now (with exception of the V2 security model) read from the *manifest* and sythensized in memory. The legacy image IDs are generated realtime based on the hashids library. This change also further deprecates a bunch of our Image APIs, reducing them to only returning the image IDs, and emptying out the remaining metadata (to avoid the requirement of us loading the information for the manifest from storage).

This has been tested with our full clients test suite with success.

* Add a backfill worker for manifest layers compressed sizes

* Change image tracks into manifest tracks now that we no longer have
manifest-less tags

* Add back in the missing method

* Add missing joins to reduce extra queries

* Remove unnecessary join when looking up legacy images

* Remove extra hidden filter on tag queries

* Further DB improvements

* Delete all Verbs, as they were deprecated

* Add back missing parameter in manifest data type

* Fix join to return None for the robot if not defined on mirror config

* switch to using secscan_v4_model for all indexing and remove most of secscan_v2_model code

* Add a missing join

* Remove files accidentally re-added due to rebase

* Add back hashids lib

* Rebase fixes

* Fix broken test

* Remove unused GPG signer now that ACI conversion is removed

* Remove duplicated repomirrorworker

* Remove unused notification code for secscan. We'll re-add it once Clair
V4 security notifications are ready to go

* Fix formatting

* Stop writing Image rows when creating manifests

* Stop writing empty layer blobs for manifests

As these blobs are shared, we don't need to write ManifestBlob rows
for them

* Remove further unused code

* Add doc comment to _build_blob_map

* Add unit test for synthetic V1 IDs

* Remove unused import

* Add an invalid value test to synthetic ID decode tests

* Add manifest backfill worker back in

Seems to have been removed at some point

* Add a test for cached active tags

* Rename test_shared to not conflict with another same-named test file

Pytest doesn't like having two test modules with the same name

* Have manifestbackfillworker also copy over the config_media_type if present

Co-authored-by: alecmerdler <alecmerdler@gmail.com>
This commit is contained in:
Joseph Schorr
2020-07-28 13:03:10 -04:00
committed by GitHub
parent 8d57e769fa
commit 0e628b1569
127 changed files with 1881 additions and 10002 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
test/test_api_usage.py
View File

@ -2444,7 +2444,6 @@ class TestDeleteRepository(ApiTestCase):
# Make sure the repository has some images and tags.
repo_ref = registry_model.lookup_repository(ADMIN_ACCESS_USER, "complex")
self.assertTrue(len(list(registry_model.get_legacy_images(repo_ref))) > 0)
self.assertTrue(len(list(registry_model.list_all_active_repository_tags(repo_ref))) > 0)
# Add some data for the repository, in addition to is already existing images and tags.
@ -2525,11 +2524,11 @@ class TestGetRepository(ApiTestCase):
self.login(ADMIN_ACCESS_USER)
# base + repo + is_starred + tags
with assert_query_count(BASE_LOGGEDIN_QUERY_COUNT + 4 + 1):
with assert_query_count(BASE_LOGGEDIN_QUERY_COUNT + 4):
self.getJsonResponse(Repository, params=dict(repository=ADMIN_ACCESS_USER + "/simple"))
# base + repo + is_starred + tags
with assert_query_count(BASE_LOGGEDIN_QUERY_COUNT + 4 + 1):
with assert_query_count(BASE_LOGGEDIN_QUERY_COUNT + 4):
json = self.getJsonResponse(
Repository, params=dict(repository=ADMIN_ACCESS_USER + "/gargantuan")
)
@ -3326,8 +3325,7 @@ class TestListAndDeleteTag(ApiTestCase):
params=dict(repository=ADMIN_ACCESS_USER + "/complex", tag="sometag"),
)
sometag_images = json["images"]
self.assertEqual(sometag_images, staging_images)
assert json["images"]
# Move the tag.
self.putResponse(
@ -3344,8 +3342,7 @@ class TestListAndDeleteTag(ApiTestCase):
)
sometag_new_images = json["images"]
self.assertEqual(1, len(sometag_new_images))
self.assertEqual(staging_images[-1], sometag_new_images[0])
assert sometag_new_images
def test_deletesubtag(self):
self.login(ADMIN_ACCESS_USER)
@ -3384,7 +3381,7 @@ class TestListAndDeleteTag(ApiTestCase):
self.login(ADMIN_ACCESS_USER)
repo_ref = registry_model.lookup_repository(ADMIN_ACCESS_USER, "simple")
latest_tag = registry_model.get_repo_tag(repo_ref, "latest", include_legacy_image=True)
latest_tag = registry_model.get_repo_tag(repo_ref, "latest")
# Create 8 tags in the simple repo.
remaining_tags = {"latest", "prod"}
@ -3392,7 +3389,7 @@ class TestListAndDeleteTag(ApiTestCase):
tag_name = "tag" + str(i)
remaining_tags.add(tag_name)
assert registry_model.retarget_tag(
repo_ref, tag_name, latest_tag.legacy_image, storage, docker_v2_signing_key
repo_ref, tag_name, latest_tag.manifest, storage, docker_v2_signing_key
)
# Make sure we can iterate over all of them.