* chore: update ci to use new large ubuntu 24.04 runner
Signed-off-by: Brady Pratt <bpratt@redhat.com>
Co-Authored-By: Dave O'Connor <doconnor@redhat.com>
* fix: add libfreetype6-dev for Ubuntu 24.04 compatibility
The reportlab package requires FreeType development headers to build.
On Ubuntu 24.04, this dependency is not pulled in transitively and
must be explicitly installed. This fixes the "cannot find ft2build.h"
build error.
Added libfreetype6-dev to all jobs that install system dependencies
in CI.yaml and CI-nightly.yaml workflows.
Signed-off-by: Brady Pratt <bpratt@redhat.com>
Co-Authored-By: Dave O'Connor <doconnor@redhat.com>
* chore: set the TEST_DATETIME to a static value
this caused an issue in xdist when generating test names
Signed-off-by: Brady Pratt <bpratt@redhat.com>
* chore: cache pip packages in CI
Signed-off-by: Brady Pratt <bpratt@redhat.com>
* chore: run registry tests with -n auto
Signed-off-by: Brady Pratt <bpratt@redhat.com>
* chore: run psql with -n auto
Signed-off-by: Brady Pratt <bpratt@redhat.com>
* chore: add file locking to prevent parallel test db init race condition
When running pytest -n auto with multiple workers, both workers would
simultaneously execute populate_database(), causing duplicate key
violations on shared tables like imagestoragelocation:
Worker 1: Check if User "devtable" exists → No → Start populating
Worker 2: Check if User "devtable" exists → No → Start populating
Both: INSERT INTO imagestoragelocation (name) VALUES ('local_eu')
Result: IntegrityError - duplicate key violation
Solution: Wrap init_db_path fixture with FileLock to ensure only one
worker initializes the database at a time. The lock file is created
in pytest's shared temp directory, coordinating across all workers.
- First worker acquires lock and populates database
- Subsequent workers wait at lock, then see database is already
populated (via User.get() check in populate_database())
- Works for both PostgreSQL and MySQL
- 300-second timeout prevents deadlocks
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* chore: run mysql with -n auto
Signed-off-by: Brady Pratt <bpratt@redhat.com>
---------
Signed-off-by: Brady Pratt <bpratt@redhat.com>
Co-authored-by: Dave O'Connor <doconnor@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
* Add ngnix routing logic to default to react UI for downstream
* Remove defaulting from env, update Makefile to build react by default for local
* Add cypress test for signin and create account workflow
* Add missing routes + fallback to backend server
* Hide UI toggle when defaulting to new UI
* Adds forgot password + recovery email, recaptcha, missing login checks
* Add external login screen + support for other login types for new UI
* Add new screen for update user after external login
* Add authorized apps section under external logins tab
* Implement updateuser react component + fix cypress test
* Fix external login OAuth flow for react
* switch logic to default to new ui
* Add DEFAULT_UI: angular to config for cypress CI
* Fix cypress tests for oauth-callback
* Rebase and fix merge conflicts
---------
Signed-off-by: harishsurf <hgovinda@redhat.com>
* nginx: Increase the number and size of proxy buffers (PROJQUAY-6950)
From [nginx documentation](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering):
> When buffering is enabled, nginx receives a response from the proxied server as soon as possible, saving it into the buffers set by the proxy_buffer_size and proxy_buffers directives.
> If the whole response does not fit into memory, a part of it can be saved to a temporary file on the disk.
> Writing to temporary files is controlled by the proxy_max_temp_file_size and proxy_temp_file_write_size directives.
>
> When buffering is disabled, the response is passed to a client synchronously, immediately as it is received. nginx will not try to read the whole response from the proxied server.
> The maximum size of the data that nginx can receive from the server at a time is set by the proxy_buffer_size directive.
By default, the value of `proxy_buffer_size` directive, if not set in the nginx configuration, is equal to one memory page which on most platforms equals 4 KiB of memory. When `FEATURE_PROXY_STORAGE` is turned on and STS driver is used, the size of headers reaches that limit of 4 KiB and, if it surpasses it, nginx will error out and will not process the request. With this PR we set the buffers to an adequate size so that proxy requests are properly processed by nginx.
* Add additional nginx directive to make buffers work
* chore: make local-dev-up should restart quay container
make local-dev-up deletes generated files that are needed by Quay. To
generate them again, the container should be restarted.
As docker-compose up doesn't have an option to restart the container, we
use docker-compose stop.
* chore: pass DOCKER_USER to docker-compose up clair
clair container depends on quay, so `docker-compose up clair` should
have correct environment variables for quay.
* chore: increase max_connections for clair-db
clair indexer can use up to 100 connections in our configuration, i.e.
all available postgres connections. In this case other workers are not
able to connect to postgres and cannot do their job.
* dev: Add pre-commit script to run black (PROJQUAY-4039)
Automatically formats python files before commit
* Add makefile target to install hook
* Update instructions for installing pre-commit hook
introduces the possibility to pull images from external registries
through Quay, storing them locally for faster subsequent pulls.
Closes PROJQUAY-3030 and PROJQUAY-3033
* dockerfile: update upstream image to use ubi8 as base (PROJQUAY-3148)
Update the upstream base image to use `ubi:8`, since we're already
maintaining an essentially identical ubi:8 image for production.
CentOS Steam's Python differ from ubi8's which would require us
maintaining 2 sets of python requirements for each.
* Revert "build(deps): bump pillow from 8.3.2 to 9.0.0 (#1059)"
This reverts commit 102705c946. Ubi:8's Python
doens't have Pillow 9.0.0.
* deps: bump Pillow to 8.4.0
* Removes sudo requirement from Makefile for local-dev-down
* Forces restart of quay and clair containers on local-docker-rebuild. This allows changes to volume mounted files to be picked up.
* Removes legacy Dockerfile
* Add dev dependencies mypy and typing
* Add makefile target `types-test`, not yet included in `test` target.
* Generate stubs for imported modules to avoid mypy complaining about missing types.
* Remove generated stubs as there are way too many and they cause tons of mess in the repo. Switched to ignoring untyped modules for now, to concentrate on Quay-only type checking.
* mypy config changed to ignore missing imports
* ignore property decorator as it is not supported by mypy
* mypy annotations for many configuration variables
* re-generate mypy_stubs directory as its necessary in some classes for base classes to prevent mypy errors
* util/registry/queuefile referred to non existent definition of Empty class in multiprocessing.queues
* ignore type checking for things like monkey patching and exported/re-imported objects that
mypy does not allow.
* Adjust mypy config to warn us about unreachable return paths and useless expressions.
* Add the __annotations__ property to INTERNAL_ONLY_PROPERTIES so that it is not part of the config schema testing
* Remove redundant dependencies `typing` and `typing-extensions` which are NOOP after Python 3.5
* Remove mypy-extensions which only provides a TypedDict implementation but has not been updated since 2019.
* updated mypy to 0.910 which requires all types packages to be installed manually.
* exclude local-dev from type checking until core team can suggest an outcome for __init__.py duplicate packages
* re-add typing dependency which will be needed until Python 3.9
* ignore .mypy_cache
* add mypy stub for features module to replace inline definitions
* import annotations eager evaluation in billing.py as it was required to reference a class declared later in the module.
* remove the type definition of V1ProtocolSteps/V2ProtocolSteps to make tox happy
* local-dev: implement local development environment
this commit copies the files in /init into /local-dev, edits those files
to support hot-reload features for local development, and introduces
docker-compose/makefile targets in order to support local dev.
Signed-off-by: ldelossa <ldelossa@redhat.com>
* local-dev: hop quay workers to gunicorn
this commit adds a uwsgi worker application delegate and a factory
function allowing each worker to be ran by gunicorn.
each worker now supports hot-reload and will reload itself when it's
code is updated.
this changes only affects the local dev env.
Signed-off-by: ldelossa <ldelossa@redhat.com>
* local-dev: add docs
Signed-off-by: ldelossa <ldelossa@redhat.com>
* Reenable builder in supervisord config
* Rewrites the buildmanager to use gRPC
Rewrite of the current buildmanager using gRPC.
This deprecates the enterprise type builder, as individual nodes will
no longer keep track of build states because of WAMP.
Also removes trollius, which was required by the WAMP servers.
Instead, gRPC uses a threaded model to serve its requests.
Deprecates etcd as state trakcing for build states in favor of Redis
only.
Defines a state interface to manage/transition build states, implemented by the
buildmanager.
* Fix incorrect reference to aws connection
* Truncate the "Token" tag in ec2 to 36 char.
Normalize the token tag to 36 char in EC2.
Add an expiration to the original redis key, in the event that the
expiry handler is not able to delete the key, the original should be
removed eventually.
* Orchestrator: add context to KeyError
* EXPOSE 50051 in Dockerfiles
* Add buildman/README
* Convert all Python2 to Python3 syntax.
* Removes oauth2lib dependency
* Replace mockredis with fakeredis
* byte/str conversions
* Removes nonexisting __nonzero__ in Python3
* Python3 Dockerfile and related
* [PROJQUAY-98] Replace resumablehashlib with rehash
* PROJQUAY-123 - replace gpgme with python3-gpg
* [PROJQUAY-135] Fix unhashable class error
* Update external dependencies for Python 3
- Move github.com/app-registry/appr to github.com/quay/appr
- github.com/coderanger/supervisor-stdout
- github.com/DevTable/container-cloud-config
- Update to latest mockldap with changes applied from coreos/mockldap
- Update dependencies in requirements.txt and requirements-dev.txt
* Default FLOAT_REPR function to str in json encoder and removes keyword assignment
True, False, and str were not keywords in Python2...
* [PROJQUAY-165] Replace package `bencode` with `bencode.py`
- Bencode is not compatible with Python 3.x and is no longer
maintained. Bencode.py appears to be a drop-in replacement/fork
that is compatible with Python 3.
* Make sure monkey.patch is called before anything else (
* Removes anunidecode dependency and replaces it with text_unidecode
* Base64 encode/decode pickle dumps/loads when storing value in DB
Base64 encodes/decodes the serialized values when storing them in the
DB. Also make sure to return a Python3 string instead of a Bytes when
coercing for db, otherwise, Postgres' TEXT field will convert it into
a hex representation when storing the value.
* Implement __hash__ on Digest class
In Python 3, if a class defines __eq__() but not __hash__(), its
instances will not be usable as items in hashable collections (e.g sets).
* Remove basestring check
* Fix expected message in credentials tests
* Fix usage of Cryptography.Fernet for Python3 (#219)
- Specifically, this addresses the issue where Byte<->String
conversions weren't being applied correctly.
* Fix utils
- tar+stream layer format utils
- filelike util
* Fix storage tests
* Fix endpoint tests
* Fix workers tests
* Fix docker's empty layer bytes
* Fix registry tests
* Appr
* Enable CI for Python 3.6
* Skip buildman tests
Skip buildman tests while it's being rewritten to allow ci to pass.
* Install swig for CI
* Update expected exception type in redis validation test
* Fix gpg signing calls
Fix gpg calls for updated gpg wrapper, and add signing tests.
* Convert / to // for Python3 integer division
* WIP: Update buildman to use asyncio instead of trollius.
This dependency is considered deprecated/abandoned and was only
used as an implementation/backport of asyncio on Python 2.x
This is a work in progress, and is included in the PR just to get the
rest of the tests passing. The builder is actually being rewritten.
* Target Python 3.8
* Removes unused files
- Removes unused files that were added accidentally while rebasing
- Small fixes/cleanup
- TODO tasks comments
* Add TODO to verify rehash backward compat with resumablehashlib
* Revert "[PROJQUAY-135] Fix unhashable class error" and implements __hash__ instead.
This reverts commit 735e38e3c1d072bf50ea864bc7e119a55d3a8976.
Instead, defines __hash__ for encryped fields class, using the parent
field's implementation.
* Remove some unused files ad imports
Co-authored-by: Kenny Lee Sin Cheong <kenny.lee@redhat.com>
Co-authored-by: Tom McKay <thomasmckay@redhat.com>
* Remove V3_UPGRADE_MODE
* Remove tag backfill worker and all callers to tag backfill and upgrades
* Change initdb to create all test data via the manifest builder, rather than manually via legacy images
* Convert various code paths to use the registry_model where we previously did not do so
* Convert the GC test suite to build via the manifest builder
* Delete an old, unused tool
* Delete the Pre OCI registry model
* Add additional error handling to the manifest creation code path
* Add additional error handling to the OCI tag creation code path
* Change how we respond to invalid manifest content types to better handle unknowns
* Change legacy secscan test suite to use the registry model
* Change the repo build badge to use the registry model (also fixes a bug)
* Delete now-unused data model code
* Remove old model adjustment code from OCI model
* Mark older data models as deprecated which will prevent new rows from being inserted
* Remove references to old registry test suite from various testing files
* Remove tag backfill worker (again; got re-added during rebase)
* Move all deprecated model checks into a central function
* Make data_migration more Pythonic
* Small requested fixes to Tag module styling
* Have tag backfill worker fail to migrate if there are TagManifest's
Since this backfill should only be called in future releases for empty models, this should catch someone attempting to upgrade from a too-old version
* Remove labelbackfillworker as it is no longer needed
* Remove unused invalid import
* Reimplement the tag test for the remaining method used
