* secscan: Fix Slack notification creation on initial index (PROJQUAY-7037)
This fixes the Slack and e-mail notifications when images are indexed on initial push, where `tags` information was missing. If an image is a manifest child, instead of the tag, we provide a full SHA digest to the client.
Also adds the ability to filter security vulnerabilites depending on their severity. If the `NOTIFICATION_MIN_SEVERITY_ON_NEW_INDEX` is not set in the `config.yaml` file, we will automatically create notifications only for vulnerabilities marked "high" or "critical". This variable can take values defined here:
3248a72da6/util/secscan/__init__.py (L1)
Example:
```
NOTIFICATION_MIN_SEVERITY_ON_NEW_INDEX: Medium
```
* Removed superfulous debug statements
* Add default to config.yaml, remove extra comment
* Fix isort sorting
* Update vulnerability_found.html
* Update vulnerability_found.html
* Isort formatting fix
* Update vulnerability_found.html
* feat: Add support for auto pruning at repository level (PROJQUAY-6354)
* Add repositoryautoprunepolicy table to alembic migration script
* Add repository auto-prune policy endpoints
* Add UI for repository auto-pruning policies
* case: apply repo auto-prune policy when no namespace policy given
* case: both namespace and repo pruning policy are given
* Add tests for repository autoprune endpoint
* Add cypress test for repository auto-prune
* Add repo auto-prune policy clean-up for repository deletion
* Add repository auto pruning tables to quay db snapshot for cypress tests
* Address review comments
* Add more tests + fix CI + reformat files
* Address review comments #2
---------
Signed-off-by: harishsurf <hgovinda@redhat.com>
* chore: pass config to isort as it doesn't always detect it
* chore: mark package "test" as local, not stdlib
* chore: remove "isort: skip_file"
* chore: fix app in test_load_security_information
* chore: fix app in test_notification
* chore: fix app in test_index_report
In Clair 4.7, an API field documented to have version information in no
particular format may URL encode it. This change attempts to decode and
read a known field that contains the "fixed" version. If that fails, the
value will be passed through, as it is now.
Signed-off-by: Hank Donnay <hdonnay@redhat.com>
Garbage collect manifests no longer referenced in Quay from the
security scanner service.
Also moved quota related code from data/registry_model/ to data/model/
to avoid circular dependencies.
* config: Set feature flag default for new vulnerability notifications to True
(PROJQUAY-4659)
Change the default from `False` to `True` for
`FEATURE_SECURITY_SCANNING_NOTIFY_ON_NEW_INDEX`.
Since this flag addresses a bug, it should be enabled by default.
* add mock return values for unit tests
(PROJQUAY-5600)
Catch exception thrown when looking up tag `lifetime_start_ms` and set
`created_at` property for a manifest to None.
If `created_at` is None, don't calculate indexing SLI for manifest.
Currently Quay is displaying the Clair response with no
interpretation meaning when Clair reports on vulns per repo
they appear to be duplicated (RHEL based images with multiple
repos and packages existing in more than one). The correct way
to fix this is via an interpretation layer in Quay, this change
is a stop-gap to improve readability for the user.
Signed-off-by: crozzy <joseph.crosland@gmail.com>
Signed-off-by: crozzy <joseph.crosland@gmail.com>
Currently if Clair returns errors Quay will delete the security
status for that manifest, meaning it will revert to a "Queued"
status and needs to be reindexed. This in-turn means that the
functionality is not immediately recovered when Clair becomes
healthy. It is also generally a bad idea (except in specific
use-cases to have a data modification in a read path).
Signed-off-by: crozzy <joseph.crosland@gmail.com>
Removes read support for Clair V2, along with the need to package
jwtproxy with Quay.
TODO: Drop deprecate image api + image table, remove image data model.
Split the work of indexing the recent manifests iterator into multiple
batches. This can reduce how often duplicate work happens when
allowing multiple workers to work on the same chunk of the table.
Index recent manifests in a separate background process, allowing the
main process to correctly select random slabs from the entire table
set and marking them completed in the allocator (rbtree). This avoids
the worker having to start iterating from the beginning of the table
whenever it is restarted.
Add histogram for request duration on indexer service. Add random
batch to manifest iterator to reduce duplicate work while manifests
are being backfilled.
Have the securityworker send an abort signal in the case where the
conditions for indexing a manifest are not met after running the
candidate query. This could be when:
- a manifest has a reference to a manifestsecuritystatus instance
- the manifestsecuritystatus reindex threshold is no longer
valid (because it was recently updated)
This signals that another worker overlapped, and allows the current
worker to move on to the next set.
Add a global lock on security worker iterations, based on the value of
the current pagination token. This is to avoid multilple worker
processes possibly doing work on the same set of manifests.
For larger databases like quay.io the default batch
size is too high to complete the job in a resonable
time. Make the batch_size configurable so we can
adjust how many scans can be in-flight for a worker
Currently we randomly scan for entries to be indexed. This could take a
long time if we have a lot of unindexed images. Ideally, we want to
priortize scanning of newer images before we go to the old ones.
This PR creates a smaller batch of size `batch_size` from
`max_id - batch_size` to `max_id` to priortize scanning
those
Currently it is possible that Clair returns a state that Quay doesn't
know about and the previous logic will save this as a completed
manifest.
Signed-off-by: crozzy <joseph.crosland@gmail.com>
* Add dev dependencies mypy and typing
* Add makefile target `types-test`, not yet included in `test` target.
* Generate stubs for imported modules to avoid mypy complaining about missing types.
* Remove generated stubs as there are way too many and they cause tons of mess in the repo. Switched to ignoring untyped modules for now, to concentrate on Quay-only type checking.
* mypy config changed to ignore missing imports
* ignore property decorator as it is not supported by mypy
* mypy annotations for many configuration variables
* re-generate mypy_stubs directory as its necessary in some classes for base classes to prevent mypy errors
* util/registry/queuefile referred to non existent definition of Empty class in multiprocessing.queues
* ignore type checking for things like monkey patching and exported/re-imported objects that
mypy does not allow.
* Adjust mypy config to warn us about unreachable return paths and useless expressions.
* Add the __annotations__ property to INTERNAL_ONLY_PROPERTIES so that it is not part of the config schema testing
* Remove redundant dependencies `typing` and `typing-extensions` which are NOOP after Python 3.5
* Remove mypy-extensions which only provides a TypedDict implementation but has not been updated since 2019.
* updated mypy to 0.910 which requires all types packages to be installed manually.
* exclude local-dev from type checking until core team can suggest an outcome for __init__.py duplicate packages
* re-add typing dependency which will be needed until Python 3.9
* ignore .mypy_cache
* add mypy stub for features module to replace inline definitions
* import annotations eager evaluation in billing.py as it was required to reference a class declared later in the module.
* remove the type definition of V1ProtocolSteps/V2ProtocolSteps to make tox happy
* Quay.io UI: Fetching severity from cvss score and removing visibility of unknown metrics (PROJQUAY-2541)
If vulnerability's severity is missing mapping severity from cvss score if present. Removed visibility if severity or distribition is Unknow
* Quay.io UI: Fetching severity from cvss score and removing visibility of unknown metrics (PROJQUAY-2541)
Fixing Typo
If Clair returns an error the current behaviour is to
error out, thus not indexing any subsequent manifests.
This change allows the worker to continue indexing
subsequent manifests after one failure.
Signed-off-by: crozzy <joseph.crosland@gmail.com>
