In Clair 4.7, an API field documented to have version information in no
particular format may URL encode it. This change attempts to decode and
read a known field that contains the "fixed" version. If that fails, the
value will be passed through, as it is now.
Signed-off-by: Hank Donnay <hdonnay@redhat.com>
Co-authored-by: Hank Donnay <hdonnay@redhat.com>
(PROJQUAY-4659)
Change the default from `False` to `True` for
`FEATURE_SECURITY_SCANNING_NOTIFY_ON_NEW_INDEX`.
Since this flag addresses a bug, it should be enabled by default.
Currently Quay is displaying the Clair response with no
interpretation meaning when Clair reports on vulns per repo
they appear to be duplicated (RHEL based images with multiple
repos and packages existing in more than one). The correct way
to fix this is via an interpretation layer in Quay, this change
is a stop-gap to improve readability for the user.
Signed-off-by: crozzy <joseph.crosland@gmail.com>
Signed-off-by: crozzy <joseph.crosland@gmail.com>
Currently if Clair returns errors Quay will delete the security
status for that manifest, meaning it will revert to a "Queued"
status and needs to be reindexed. This in-turn means that the
functionality is not immediately recovered when Clair becomes
healthy. It is also generally a bad idea (except in specific
use-cases to have a data modification in a read path).
Signed-off-by: crozzy <joseph.crosland@gmail.com>
Removes read support for Clair V2, along with the need to package
jwtproxy with Quay.
TODO: Drop deprecate image api + image table, remove image data model.
Split the work of indexing the recent manifests iterator into multiple
batches. This can reduce how often duplicate work happens when
allowing multiple workers to work on the same chunk of the table.
Index recent manifests in a separate background process, allowing the
main process to correctly select random slabs from the entire table
set and marking them completed in the allocator (rbtree). This avoids
the worker having to start iterating from the beginning of the table
whenever it is restarted.
Add histogram for request duration on indexer service. Add random
batch to manifest iterator to reduce duplicate work while manifests
are being backfilled.
Have the securityworker send an abort signal in the case where the
conditions for indexing a manifest are not met after running the
candidate query. This could be when:
- a manifest has a reference to a manifestsecuritystatus instance
- the manifestsecuritystatus reindex threshold is no longer
valid (because it was recently updated)
This signals that another worker overlapped, and allows the current
worker to move on to the next set.
Add a global lock on security worker iterations, based on the value of
the current pagination token. This is to avoid multilple worker
processes possibly doing work on the same set of manifests.
For larger databases like quay.io the default batch
size is too high to complete the job in a resonable
time. Make the batch_size configurable so we can
adjust how many scans can be in-flight for a worker
Currently we randomly scan for entries to be indexed. This could take a
long time if we have a lot of unindexed images. Ideally, we want to
priortize scanning of newer images before we go to the old ones.
This PR creates a smaller batch of size `batch_size` from
`max_id - batch_size` to `max_id` to priortize scanning
those
Currently it is possible that Clair returns a state that Quay doesn't
know about and the previous logic will save this as a completed
manifest.
Signed-off-by: crozzy <joseph.crosland@gmail.com>
* Add dev dependencies mypy and typing
* Add makefile target `types-test`, not yet included in `test` target.
* Generate stubs for imported modules to avoid mypy complaining about missing types.
* Remove generated stubs as there are way too many and they cause tons of mess in the repo. Switched to ignoring untyped modules for now, to concentrate on Quay-only type checking.
* mypy config changed to ignore missing imports
* ignore property decorator as it is not supported by mypy
* mypy annotations for many configuration variables
* re-generate mypy_stubs directory as its necessary in some classes for base classes to prevent mypy errors
* util/registry/queuefile referred to non existent definition of Empty class in multiprocessing.queues
* ignore type checking for things like monkey patching and exported/re-imported objects that
mypy does not allow.
* Adjust mypy config to warn us about unreachable return paths and useless expressions.
* Add the __annotations__ property to INTERNAL_ONLY_PROPERTIES so that it is not part of the config schema testing
* Remove redundant dependencies `typing` and `typing-extensions` which are NOOP after Python 3.5
* Remove mypy-extensions which only provides a TypedDict implementation but has not been updated since 2019.
* updated mypy to 0.910 which requires all types packages to be installed manually.
* exclude local-dev from type checking until core team can suggest an outcome for __init__.py duplicate packages
* re-add typing dependency which will be needed until Python 3.9
* ignore .mypy_cache
* add mypy stub for features module to replace inline definitions
* import annotations eager evaluation in billing.py as it was required to reference a class declared later in the module.
* remove the type definition of V1ProtocolSteps/V2ProtocolSteps to make tox happy
* Quay.io UI: Fetching severity from cvss score and removing visibility of unknown metrics (PROJQUAY-2541)
If vulnerability's severity is missing mapping severity from cvss score if present. Removed visibility if severity or distribition is Unknow
* Quay.io UI: Fetching severity from cvss score and removing visibility of unknown metrics (PROJQUAY-2541)
Fixing Typo
If Clair returns an error the current behaviour is to
error out, thus not indexing any subsequent manifests.
This change allows the worker to continue indexing
subsequent manifests after one failure.
Signed-off-by: crozzy <joseph.crosland@gmail.com>
Handles the case where a V2 scanner is not configured (noop
implementation) and the proxy model tries to load security information
that has not been indexed in V4 yet. In such case, the noop
implementation of the V2 scanner should return a COULD_NOT_LOAD
status (instead of None, to comply to the secscan model interface),
and ignore the fallback to V2, and finally just return the V4
status (NOT_YET_INDEXED)
* clairv4: Add package and vulnerability metadata into secscan response
* Update field name
* Add missing mapping
* Fix code format
* Piggyback Metadata field to expose updater, repository and distro details
* Fix format
* Use distro and repo details from vulnerability
* Trigger PR check
* fix: Add missing model mapping
* Add response with Metadata field
* Modify response according to new Metadata format
* Change verbs to use a DerivedStorageForManifest table instead of DerivedStorageForImage
This allows us to deprecate the DerivedStorageForImage table.
Fixes https://issues.redhat.com/browse/PROJQUAY-519
* Change uploaded blob tracking to use its own table and deprecate
RepositoryTag
* Start recording the compressed layers size and config media type on the
manifest row in the database
NOTE: This change includes a database migration which will *lock* the
manifest table
* Change tag API to return the layers size from the manifest
* Remove unused code
* Add new config_media_type field to OCI types
* Fix secscan V2 test for us no longer writing temp images
* Remove unused uploading field
* Switch registry model to use synthetic legacy images
Legacy images are now (with exception of the V2 security model) read from the *manifest* and sythensized in memory. The legacy image IDs are generated realtime based on the hashids library. This change also further deprecates a bunch of our Image APIs, reducing them to only returning the image IDs, and emptying out the remaining metadata (to avoid the requirement of us loading the information for the manifest from storage).
This has been tested with our full clients test suite with success.
* Add a backfill worker for manifest layers compressed sizes
* Change image tracks into manifest tracks now that we no longer have
manifest-less tags
* Add back in the missing method
* Add missing joins to reduce extra queries
* Remove unnecessary join when looking up legacy images
* Remove extra hidden filter on tag queries
* Further DB improvements
* Delete all Verbs, as they were deprecated
* Add back missing parameter in manifest data type
* Fix join to return None for the robot if not defined on mirror config
* switch to using secscan_v4_model for all indexing and remove most of secscan_v2_model code
* Add a missing join
* Remove files accidentally re-added due to rebase
* Add back hashids lib
* Rebase fixes
* Fix broken test
* Remove unused GPG signer now that ACI conversion is removed
* Remove duplicated repomirrorworker
* Remove unused notification code for secscan. We'll re-add it once Clair
V4 security notifications are ready to go
* Fix formatting
* Stop writing Image rows when creating manifests
* Stop writing empty layer blobs for manifests
As these blobs are shared, we don't need to write ManifestBlob rows
for them
* Remove further unused code
* Add doc comment to _build_blob_map
* Add unit test for synthetic V1 IDs
* Remove unused import
* Add an invalid value test to synthetic ID decode tests
* Add manifest backfill worker back in
Seems to have been removed at some point
* Add a test for cached active tags
* Rename test_shared to not conflict with another same-named test file
Pytest doesn't like having two test modules with the same name
* Have manifestbackfillworker also copy over the config_media_type if present
Co-authored-by: alecmerdler <alecmerdler@gmail.com>
* Convert all Python2 to Python3 syntax.
* Removes oauth2lib dependency
* Replace mockredis with fakeredis
* byte/str conversions
* Removes nonexisting __nonzero__ in Python3
* Python3 Dockerfile and related
* [PROJQUAY-98] Replace resumablehashlib with rehash
* PROJQUAY-123 - replace gpgme with python3-gpg
* [PROJQUAY-135] Fix unhashable class error
* Update external dependencies for Python 3
- Move github.com/app-registry/appr to github.com/quay/appr
- github.com/coderanger/supervisor-stdout
- github.com/DevTable/container-cloud-config
- Update to latest mockldap with changes applied from coreos/mockldap
- Update dependencies in requirements.txt and requirements-dev.txt
* Default FLOAT_REPR function to str in json encoder and removes keyword assignment
True, False, and str were not keywords in Python2...
* [PROJQUAY-165] Replace package `bencode` with `bencode.py`
- Bencode is not compatible with Python 3.x and is no longer
maintained. Bencode.py appears to be a drop-in replacement/fork
that is compatible with Python 3.
* Make sure monkey.patch is called before anything else (
* Removes anunidecode dependency and replaces it with text_unidecode
* Base64 encode/decode pickle dumps/loads when storing value in DB
Base64 encodes/decodes the serialized values when storing them in the
DB. Also make sure to return a Python3 string instead of a Bytes when
coercing for db, otherwise, Postgres' TEXT field will convert it into
a hex representation when storing the value.
* Implement __hash__ on Digest class
In Python 3, if a class defines __eq__() but not __hash__(), its
instances will not be usable as items in hashable collections (e.g sets).
* Remove basestring check
* Fix expected message in credentials tests
* Fix usage of Cryptography.Fernet for Python3 (#219)
- Specifically, this addresses the issue where Byte<->String
conversions weren't being applied correctly.
* Fix utils
- tar+stream layer format utils
- filelike util
* Fix storage tests
* Fix endpoint tests
* Fix workers tests
* Fix docker's empty layer bytes
* Fix registry tests
* Appr
* Enable CI for Python 3.6
* Skip buildman tests
Skip buildman tests while it's being rewritten to allow ci to pass.
* Install swig for CI
* Update expected exception type in redis validation test
* Fix gpg signing calls
Fix gpg calls for updated gpg wrapper, and add signing tests.
* Convert / to // for Python3 integer division
* WIP: Update buildman to use asyncio instead of trollius.
This dependency is considered deprecated/abandoned and was only
used as an implementation/backport of asyncio on Python 2.x
This is a work in progress, and is included in the PR just to get the
rest of the tests passing. The builder is actually being rewritten.
* Target Python 3.8
* Removes unused files
- Removes unused files that were added accidentally while rebasing
- Small fixes/cleanup
- TODO tasks comments
* Add TODO to verify rehash backward compat with resumablehashlib
* Revert "[PROJQUAY-135] Fix unhashable class error" and implements __hash__ instead.
This reverts commit 735e38e3c1d072bf50ea864bc7e119a55d3a8976.
Instead, defines __hash__ for encryped fields class, using the parent
field's implementation.
* Remove some unused files ad imports
Co-authored-by: Kenny Lee Sin Cheong <kenny.lee@redhat.com>
Co-authored-by: Tom McKay <thomasmckay@redhat.com>
