Adds composite index manifestblob_repository_id_blob_id. From our
testing on the quay.io production DB, This increased the number of rows
filtered for the manifestblob & imagestorage join slow query from 1.8%
to 100%
Co-authored-by: syed <syed@apache.org>
The "uploading" column is an artifact from depending on writing to the
Image table (see BlobUpload table instead). As of 3.4, Quay no longer
writes to that table, and is only needed until quayio moves away from
Clair v2, after which work to remove "glue" code and fully deprecate
the Image table (amongst other tables) can start.
This is done as a separate commit from the actual migration so that it
can be cherrypicked.
Remove GlobalLock from Redis model cache implementation in
favor of 'nx=True' when setting the key.
Signed-off-by: Alec Merdler <alecmerdler@gmail.com>
Since NamespaceGCWorker does a superset of RepositoryGCWorker's
operations, make sure that quay_gc_repos_purged is incremented if
either workers deletes a repository.
Adds implementation of DataModelCache interface backed by Redis.
All containers in a Quay cluster deployment will share a single
model cache, rather than each container using its own cache.
Signed-off-by: Alec Merdler <alecmerdler@gmail.com>
Prevents the queueworker from setting the event to stop the poll_queue
job when a WorkerSleepException is raised. On WorkerSleepException,
the worker should instead skip this iteration (go to sleep). e.g when
the NamespaceGCWorker can't acquire a lock because it is already taken
by some other worker.
Reverts the gcworkers job timeout from 24h to 3h. In case of a
deadlock between processes (for example, redeploying the app will not
clear the existing Redis keys), 24h is too long waiting for the locks to
expires so that the workers can resume work.
Add missing Counter increment for on row deletion on the Manifest table.
Apply a manifest's "quay.expires-after" label expiry value to new tags pointing
to that existing manifest. Before, that label would be only applied at when the
manifest was created, and new tags targeting that manifest would not
have the manifest's corresponding expiry value set.
* clairv4: Add package and vulnerability metadata into secscan response
* Update field name
* Add missing mapping
* Fix code format
* Piggyback Metadata field to expose updater, repository and distro details
* Fix format
* Use distro and repo details from vulnerability
* Trigger PR check
* fix: Add missing model mapping
* Add response with Metadata field
* Modify response according to new Metadata format
RepoMirrorConfig in the current database migration version has a
non-null constraint on the internal_robot field, but the model in
database.py does not.
Updates the model to match the current database revision, and handles
delete api calls when there are mirrors still using the robot.
Also set a default test DATABASE_SECRET_KEY when generating the test.db
We now break out of the transaction if the repository already exists,
so that Postgres doesn't raise an error when we try to retrieve the
existing repository
Fixes https://issues.redhat.com/browse/PROJQUAY-973
Due to the requirement for the shared empty layer for manifest schema 1,
we need to make sure it is written to the ImageStorage table, even if
the only schemas pushed are version 2
Fixes https://issues.redhat.com/browse/PROJQUAY-948
* Add an extra "metadata" field to Elasticsearch logentries
The current "metadata_json" field is stored as a text field, and
cannot be searched easily. "metadata" will store the given dict as a
dynamically mapped Object type.
* Remove duplicate "metadata_json" field in Elasticsearch
Instead, json dumps the "metadata" field for the Log datatype.
This changes the lookup to occur outside of the transaction as otherwise
we won't find the manifest that was inserted during the transaction's
view
Fixes https://issues.redhat.com/browse/PROJQUAY-915
* Change verbs to use a DerivedStorageForManifest table instead of DerivedStorageForImage
This allows us to deprecate the DerivedStorageForImage table.
Fixes https://issues.redhat.com/browse/PROJQUAY-519
* Change uploaded blob tracking to use its own table and deprecate
RepositoryTag
* Start recording the compressed layers size and config media type on the
manifest row in the database
NOTE: This change includes a database migration which will *lock* the
manifest table
* Change tag API to return the layers size from the manifest
* Remove unused code
* Add new config_media_type field to OCI types
* Fix secscan V2 test for us no longer writing temp images
* Remove unused uploading field
* Switch registry model to use synthetic legacy images
Legacy images are now (with exception of the V2 security model) read from the *manifest* and sythensized in memory. The legacy image IDs are generated realtime based on the hashids library. This change also further deprecates a bunch of our Image APIs, reducing them to only returning the image IDs, and emptying out the remaining metadata (to avoid the requirement of us loading the information for the manifest from storage).
This has been tested with our full clients test suite with success.
* Add a backfill worker for manifest layers compressed sizes
* Change image tracks into manifest tracks now that we no longer have
manifest-less tags
* Add back in the missing method
* Add missing joins to reduce extra queries
* Remove unnecessary join when looking up legacy images
* Remove extra hidden filter on tag queries
* Further DB improvements
* Delete all Verbs, as they were deprecated
* Add back missing parameter in manifest data type
* Fix join to return None for the robot if not defined on mirror config
* switch to using secscan_v4_model for all indexing and remove most of secscan_v2_model code
* Add a missing join
* Remove files accidentally re-added due to rebase
* Add back hashids lib
* Rebase fixes
* Fix broken test
* Remove unused GPG signer now that ACI conversion is removed
* Remove duplicated repomirrorworker
* Remove unused notification code for secscan. We'll re-add it once Clair
V4 security notifications are ready to go
* Fix formatting
* Stop writing Image rows when creating manifests
* Stop writing empty layer blobs for manifests
As these blobs are shared, we don't need to write ManifestBlob rows
for them
* Remove further unused code
* Add doc comment to _build_blob_map
* Add unit test for synthetic V1 IDs
* Remove unused import
* Add an invalid value test to synthetic ID decode tests
* Add manifest backfill worker back in
Seems to have been removed at some point
* Add a test for cached active tags
* Rename test_shared to not conflict with another same-named test file
Pytest doesn't like having two test modules with the same name
* Have manifestbackfillworker also copy over the config_media_type if present
Co-authored-by: alecmerdler <alecmerdler@gmail.com>
The PooledCache implementation incorrectly returns `None` instead of
the default if an exception occurs, but `None` is a valid cached value,
so we instead catch the exceptions ourselves now and skip the caching
code if we encounter an exception
The behavior between keystonev2 and keystonev3 is different today.
In keystonev3 implementation, the username comes from [user.name][1],
but keystonev2 just uses the incoming `username_or_email`.
Also, by reading `user.name`, we can use keystone APIKeys to do
authentication which is necessary due to potential 2FA requirements.
[1]: f4179e5e71/data/users/keystone.py (L316)
