docker/quay
docker/quay
1
0
Fork 0
mirror of https://github.com/quay/quay.git synced 2025-04-18 10:44:06 +03:00
Code

Python 3 (#153)

Browse Source 
* Convert all Python2 to Python3 syntax.

* Removes oauth2lib dependency

* Replace mockredis with fakeredis

* byte/str conversions

* Removes nonexisting __nonzero__ in Python3

* Python3 Dockerfile and related

* [PROJQUAY-98] Replace resumablehashlib with rehash

* PROJQUAY-123 - replace gpgme with python3-gpg

* [PROJQUAY-135] Fix unhashable class error

* Update external dependencies for Python 3

- Move github.com/app-registry/appr to github.com/quay/appr
- github.com/coderanger/supervisor-stdout
- github.com/DevTable/container-cloud-config
- Update to latest mockldap with changes applied from coreos/mockldap
- Update dependencies in requirements.txt and requirements-dev.txt

* Default FLOAT_REPR function to str in json encoder and removes keyword assignment

True, False, and str were not keywords in Python2...

* [PROJQUAY-165] Replace package `bencode` with `bencode.py`

- Bencode is not compatible with Python 3.x and is no longer
  maintained. Bencode.py appears to be a drop-in replacement/fork
  that is compatible with Python 3.

* Make sure monkey.patch is called before anything else (

* Removes anunidecode dependency and replaces it with text_unidecode

* Base64 encode/decode pickle dumps/loads when storing value in DB

Base64 encodes/decodes the serialized values when storing them in the
DB. Also make sure to return a Python3 string instead of a Bytes when
coercing for db, otherwise, Postgres' TEXT field will convert it into
a hex representation when storing the value.

* Implement __hash__ on Digest class

In Python 3, if a class defines __eq__() but not __hash__(), its
instances will not be usable as items in hashable collections (e.g sets).

* Remove basestring check

* Fix expected message in credentials tests

* Fix usage of Cryptography.Fernet for Python3 (#219)

- Specifically, this addresses the issue where Byte<->String
  conversions weren't being applied correctly.

* Fix utils

- tar+stream layer format utils
- filelike util

* Fix storage tests

* Fix endpoint tests

* Fix workers tests

* Fix docker's empty layer bytes

* Fix registry tests

* Appr

* Enable CI for Python 3.6

* Skip buildman tests

Skip buildman tests while it's being rewritten to allow ci to pass.

* Install swig for CI

* Update expected exception type in redis validation test

* Fix gpg signing calls

Fix gpg calls for updated gpg wrapper, and add signing tests.

* Convert / to // for Python3 integer division

* WIP: Update buildman to use asyncio instead of trollius.

This dependency is considered deprecated/abandoned and was only
used as an implementation/backport of asyncio on Python 2.x
This is a work in progress, and is included in the PR just to get the
rest of the tests passing. The builder is actually being rewritten.

* Target Python 3.8

* Removes unused files

- Removes unused files that were added accidentally while rebasing
- Small fixes/cleanup
- TODO tasks comments

* Add TODO to verify rehash backward compat with resumablehashlib

* Revert "[PROJQUAY-135] Fix unhashable class error" and implements __hash__ instead.

This reverts commit 735e38e3c1d072bf50ea864bc7e119a55d3a8976.
Instead, defines __hash__ for encryped fields class, using the parent
field's implementation.

* Remove some unused files ad imports

Co-authored-by: Kenny Lee Sin Cheong <kenny.lee@redhat.com>
Co-authored-by: Tom McKay <thomasmckay@redhat.com>
This commit is contained in:
Kurtis Mullins 2020-06-05 16:50:13 -04:00 committed by GitHub
parent 77c0d87341
commit 38be6d05d0
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
343 changed files with 4003 additions and 3085 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

47
.github/workflows/CI.yaml vendored
View File

@ -16,10 +16,10 @@ jobs:
steps:
- uses: actions/checkout@v2
- name: Set up Python 3.7
- name: Set up Python 3.8
uses: actions/setup-python@v1
with:
python-version: 3.7
python-version: 3.8
- name: Install dependencies
run: |
@ -31,7 +31,8 @@ jobs:
- name: Check Formatting
run: |
black --line-length=100 --target-version=py27 --check --diff .
# TODO(kleesc): Re-enable after buildman rewrite
black --line-length=100 --target-version=py38 --check --diff --exclude "/(\.eggs|\.git|\.hg|\.mypy_cache|\.nox|\.tox|\.venv|_build|buck-out|build|dist|buildman)/" .
unit:
name: Unit Test
@ -39,20 +40,20 @@ jobs:
steps:
- uses: actions/checkout@v2
- name: Set up Python 2.7
- name: Set up Python 3.8
uses: actions/setup-python@v1
with:
python-version: 2.7
python-version: 3.8
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install libgpgme-dev libldap2-dev libsasl2-dev
sudo apt-get install libgpgme-dev libldap2-dev libsasl2-dev swig
python -m pip install --upgrade pip
cat requirements-dev.txt | grep tox | xargs pip install
- name: tox
run: tox -e py27-unit
run: tox -e py38-unit
registry:
name: E2E Registry Tests
@ -60,20 +61,20 @@ jobs:
steps:
- uses: actions/checkout@v2
- name: Set up Python 2.7
- name: Set up Python 3.8
uses: actions/setup-python@v1
with:
python-version: 2.7
python-version: 3.8
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install libgpgme-dev libldap2-dev libsasl2-dev
sudo apt-get install libgpgme-dev libldap2-dev libsasl2-dev swig
python -m pip install --upgrade pip
cat requirements-dev.txt | grep tox | xargs pip install
- name: tox
run: tox -e py27-registry
run: tox -e py38-registry
docker:
name: Docker Build
@ -89,15 +90,15 @@ jobs:
steps:
- uses: actions/checkout@v2
- name: Set up Python 2.7
- name: Set up Python 3.8
uses: actions/setup-python@v1
with:
python-version: 2.7
python-version: 3.8
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install libgpgme-dev libldap2-dev libsasl2-dev docker.io
sudo apt-get install libgpgme-dev libldap2-dev libsasl2-dev swig docker.io
sudo systemctl unmask docker
sudo systemctl start docker
docker version
@ -105,7 +106,7 @@ jobs:
cat requirements-dev.txt | grep tox | xargs pip install
- name: tox
run: tox -e py27-mysql
run: tox -e py38-mysql
psql:
name: E2E Postgres Test
@ -113,15 +114,15 @@ jobs:
steps:
- uses: actions/checkout@v2
- name: Set up Python 2.7
- name: Set up Python 3.8
uses: actions/setup-python@v1
with:
python-version: 2.7
python-version: 3.8
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install libgpgme-dev libldap2-dev libsasl2-dev docker.io
sudo apt-get install libgpgme-dev libldap2-dev libsasl2-dev swig docker.io
sudo systemctl unmask docker
sudo systemctl start docker
docker version
@ -129,7 +130,7 @@ jobs:
cat requirements-dev.txt | grep tox | xargs pip install
- name: tox
run: tox -e py27-psql
run: tox -e py38-psql
oci:
name: OCI Conformance
@ -142,10 +143,10 @@ jobs:
repository: opencontainers/distribution-spec
path: dist-spec
- name: Set up Python 2.7
- name: Set up Python 3.8
uses: actions/setup-python@v1
with:
python-version: 2.7
python-version: 3.8
- name: Set up Go 1.14
uses: actions/setup-go@v1
@ -162,7 +163,7 @@ jobs:
run: |
# Quay
sudo apt-get update
sudo apt-get install libgpgme-dev libldap2-dev libsasl2-dev
sudo apt-get install libgpgme-dev libldap2-dev libsasl2-dev swig
python -m pip install --upgrade pip
pip install -r <(cat requirements.txt requirements-dev.txt)
@ -172,4 +173,4 @@ jobs:
CGO_ENABLED=0 go test -c -o conformance.test
- name: conformance
run: TEST=true PYTHONPATH=. pytest test/registry/conformance_tests.py -s -vv
run: TEST=true PYTHONPATH=. pytest test/registry/conformance_tests.py -s -vv --ignore=buildman # TODO(kleesc): Remove --ignore=buildman after rewrite

69
Dockerfile
View File

@ -1,14 +1,14 @@
FROM centos:7
FROM centos:8
LABEL maintainer "thomasmckay@redhat.com"
ENV OS=linux \
ARCH=amd64 \
PYTHON_VERSION=2.7 \
PYTHON_VERSION=3.6 \
PATH=$HOME/.local/bin/:$PATH \
PYTHONUNBUFFERED=1 \
PYTHONIOENCODING=UTF-8 \
LC_ALL=en_US.UTF-8 \
LANG=en_US.UTF-8 \
LC_ALL=C.UTF-8 \
LANG=C.UTF-8 \
PIP_NO_CACHE_DIR=off
ENV QUAYDIR /quay-registry
@ -19,54 +19,36 @@ RUN mkdir $QUAYDIR
WORKDIR $QUAYDIR
RUN INSTALL_PKGS="\
python27 \
python27-python-pip \
rh-nginx112 rh-nginx112-nginx \
python3 \
nginx \
openldap \
scl-utils \
gcc-c++ git \
openldap-devel \
gpgme-devel \
python3-devel \
python3-gpg \
dnsmasq \
memcached \
openssl \
skopeo \
" && \
yum install -y yum-utils && \
yum install -y epel-release centos-release-scl && \
yum -y --setopt=tsflags=nodocs --setopt=skip_missing_names_on_install=False install $INSTALL_PKGS && \
yum -y update && \
yum -y clean all
COPY . .
RUN scl enable python27 "\
pip install --upgrade setuptools==44 pip && \
pip install -r requirements.txt --no-cache && \
pip install -r requirements-dev.txt --no-cache && \
pip freeze && \
RUN alternatives --set python /usr/bin/python3 && \
python -m pip install --upgrade setuptools pip && \
python -m pip install -r requirements.txt --no-cache && \
python -m pip freeze && \
mkdir -p $QUAYDIR/static/webfonts && \
mkdir -p $QUAYDIR/static/fonts && \
mkdir -p $QUAYDIR/static/ldn && \
PYTHONPATH=$QUAYPATH python -m external_libraries \
"
RUN cp -r $QUAYDIR/static/ldn $QUAYDIR/config_app/static/ldn && \
PYTHONPATH=$QUAYPATH python -m external_libraries && \
cp -r $QUAYDIR/static/ldn $QUAYDIR/config_app/static/ldn && \
cp -r $QUAYDIR/static/fonts $QUAYDIR/config_app/static/fonts && \
cp -r $QUAYDIR/static/webfonts $QUAYDIR/config_app/static/webfonts
# Check python dependencies for GPL
# Due to the following bug, pip results must be piped to a file before grepping:
# https://github.com/pypa/pip/pull/3304
# 'docutils' is a setup dependency of botocore required by s3transfer. It's under
# GPLv3, and so is manually removed.
RUN rm -Rf /opt/rh/python27/root/usr/lib/python2.7/site-packages/docutils && \
scl enable python27 "pip freeze" | grep -v '^-e' | awk -F == '{print $1}' | grep -v docutils > piplist.txt && \
scl enable python27 "xargs -a piplist.txt pip --disable-pip-version-check show" > pipinfo.txt && \
test -z "$(cat pipinfo.txt | grep GPL | grep -v LGPL)" && \
rm -f piplist.txt pipinfo.txt
# # Front-end
RUN curl --silent --location https://rpm.nodesource.com/setup_12.x | bash - && \
yum install -y nodejs && \
curl --silent --location https://dl.yarnpkg.com/rpm/yarn.repo | tee /etc/yum.repos.d/yarn.repo && \
@ -76,14 +58,11 @@ RUN curl --silent --location https://rpm.nodesource.com/setup_12.x | bash - && \
yarn build && \
yarn build-config-app
# TODO: Build jwtproxy in dist-git
# https://jira.coreos.com/browse/QUAY-1315
ENV JWTPROXY_VERSION=0.0.3
RUN curl -fsSL -o /usr/local/bin/jwtproxy "https://github.com/coreos/jwtproxy/releases/download/v${JWTPROXY_VERSION}/jwtproxy-${OS}-${ARCH}" && \
chmod +x /usr/local/bin/jwtproxy
# TODO: Build pushgateway in dist-git
# https://jira.coreos.com/browse/QUAY-1324
ENV PUSHGATEWAY_VERSION=1.0.0
RUN curl -fsSL "https://github.com/prometheus/pushgateway/releases/download/v${PUSHGATEWAY_VERSION}/pushgateway-${PUSHGATEWAY_VERSION}.${OS}-${ARCH}.tar.gz" | \
tar xz "pushgateway-${PUSHGATEWAY_VERSION}.${OS}-${ARCH}/pushgateway" && \
@ -95,16 +74,16 @@ RUN curl -fsSL "https://github.com/prometheus/pushgateway/releases/download/v${P
RUN curl -fsSL https://ip-ranges.amazonaws.com/ip-ranges.json -o util/ipresolver/aws-ip-ranges.json
RUN ln -s $QUAYCONF /conf && \
mkdir /var/log/nginx && \
ln -sf /dev/stdout /var/log/nginx/access.log && \
ln -sf /dev/stdout /var/log/nginx/error.log && \
chmod -R a+rwx /var/log/nginx
# Cleanup
RUN UNINSTALL_PKGS="\
gcc-c++ \
gcc-c++ git \
openldap-devel \
gpgme-devel \
python3-devel \
optipng \
kernel-headers \
" && \
@ -118,24 +97,12 @@ RUN chgrp -R 0 $QUAYDIR && \
chmod -R g=u $QUAYDIR
RUN mkdir /datastorage && chgrp 0 /datastorage && chmod g=u /datastorage && \
mkdir -p /var/log/nginx && chgrp 0 /var/log/nginx && chmod g=u /var/log/nginx && \
chgrp 0 /var/log/nginx && chmod g=u /var/log/nginx && \
mkdir -p /conf/stack && chgrp 0 /conf/stack && chmod g=u /conf/stack && \
mkdir -p /tmp && chgrp 0 /tmp && chmod g=u /tmp && \
mkdir /certificates && chgrp 0 /certificates && chmod g=u /certificates && \
chmod g=u /etc/passwd
RUN chgrp 0 /var/opt/rh/rh-nginx112/log/nginx && chmod g=u /var/opt/rh/rh-nginx112/log/nginx
# Allow TLS certs to be created and installed as non-root user
RUN chgrp -R 0 /etc/pki/ca-trust/extracted && \
chmod -R g=u /etc/pki/ca-trust/extracted && \
chgrp -R 0 /etc/pki/ca-trust/source/anchors && \
chmod -R g=u /etc/pki/ca-trust/source/anchors && \
chgrp -R 0 /opt/rh/python27/root/usr/lib/python2.7/site-packages/requests && \
chmod -R g=u /opt/rh/python27/root/usr/lib/python2.7/site-packages/requests && \
chgrp -R 0 /opt/rh/python27/root/usr/lib/python2.7/site-packages/certifi && \
chmod -R g=u /opt/rh/python27/root/usr/lib/python2.7/site-packages/certifi
VOLUME ["/var/log", "/datastorage", "/tmp", "/conf/stack"]
USER 1001

10
Dockerfile.centos7.osbs
View File

@ -19,8 +19,8 @@ RUN mkdir $QUAYDIR
WORKDIR $QUAYDIR
RUN INSTALL_PKGS="\
python27 \
python27-python-pip \
python36 \
python36-python-pip \
rh-nginx112 rh-nginx112-nginx \
openldap \
scl-utils \
@ -40,7 +40,7 @@ RUN INSTALL_PKGS="\
COPY . .
RUN scl enable python27 "\
RUN scl enable python36 "\
pip install --upgrade setuptools pip && \
pip install -r requirements.txt --no-cache && \
pip install -r requirements-dev.txt --no-cache && \
@ -61,8 +61,8 @@ RUN cp -r $QUAYDIR/static/ldn $QUAYDIR/config_app/static/ldn && \
# 'docutils' is a setup dependency of botocore required by s3transfer. It's under
# GPLv3, and so is manually removed.
RUN rm -Rf /opt/rh/python27/root/usr/lib/python2.7/site-packages/docutils && \
scl enable python27 "pip freeze" | grep -v '^-e' | awk -F == '{print $1}' | grep -v docutils > piplist.txt && \
scl enable python27 "xargs -a piplist.txt pip --disable-pip-version-check show" > pipinfo.txt && \
scl enable python36 "pip freeze" | grep -v '^-e' | awk -F == '{print $1}' | grep -v docutils > piplist.txt && \
scl enable python36 "xargs -a piplist.txt pip --disable-pip-version-check show" > pipinfo.txt && \
test -z "$(cat pipinfo.txt | grep GPL | grep -v LGPL)" && \
rm -f piplist.txt pipinfo.txt

7
Dockerfile.cirun
View File

@ -1,7 +0,0 @@
FROM quay-ci-base
RUN mkdir -p conf/stack
RUN rm -rf test/data/test.db
ADD cirun.config.yaml conf/stack/config.yaml
RUN /usr/bin/scl enable python27 rh-nginx112 "LOGGING_LEVEL=INFO python initdb.py"
ENTRYPOINT ["/quay-registry/quay-entrypoint.sh"]
CMD ["registry"]

12
Dockerfile.osbs
View File

@ -19,8 +19,8 @@ RUN mkdir $QUAYDIR
WORKDIR $QUAYDIR
RUN INSTALL_PKGS="\
python27 \
python27-python-pip \
python36 \
python36-python-pip \
rh-nginx112 rh-nginx112-nginx \
openldap \
scl-utils \
@ -46,8 +46,8 @@ RUN INSTALL_PKGS="\
COPY . .
RUN scl enable python27 "\
pip install --upgrade setuptools==44 pip && \
RUN scl enable python36 "\
pip install --upgrade setuptools pip && \
pip install -r requirements.txt --no-cache && \
pip freeze && \
mkdir -p $QUAYDIR/static/webfonts && \
@ -66,8 +66,8 @@ RUN cp -r $QUAYDIR/static/ldn $QUAYDIR/config_app/static/ldn && \
# 'docutils' is a setup dependency of botocore required by s3transfer. It's under
# GPLv3, and so is manually removed.
RUN rm -Rf /opt/rh/python27/root/usr/lib/python2.7/site-packages/docutils && \
scl enable python27 "pip freeze" | grep -v '^-e' | awk -F == '{print $1}' | grep -v docutils > piplist.txt && \
scl enable python27 "xargs -a piplist.txt pip --disable-pip-version-check show" > pipinfo.txt && \
scl enable python36 "pip freeze" | grep -v '^-e' | awk -F == '{print $1}' | grep -v docutils > piplist.txt && \
scl enable python36 "xargs -a piplist.txt pip --disable-pip-version-check show" > pipinfo.txt && \
test -z "$(cat pipinfo.txt | grep GPL | grep -v LGPL)" && \
rm -f piplist.txt pipinfo.txt

12
Dockerfile.rhel7
View File

@ -19,8 +19,8 @@ RUN mkdir $QUAYDIR
WORKDIR $QUAYDIR
RUN INSTALL_PKGS="\
python27 \
python27-python-pip \
python36 \
python36-python-pip \
rh-nginx112 rh-nginx112-nginx \
openldap \
scl-utils \
@ -46,8 +46,8 @@ RUN INSTALL_PKGS="\
COPY . .
RUN scl enable python27 "\
pip install --upgrade setuptools==44 pip && \
RUN scl enable python36 "\
pip install --upgrade setuptools pip && \
pip install -r requirements.txt --no-cache && \
pip freeze && \
mkdir -p $QUAYDIR/static/webfonts && \
@ -66,8 +66,8 @@ RUN cp -r $QUAYDIR/static/ldn $QUAYDIR/config_app/static/ldn && \
# 'docutils' is a setup dependency of botocore required by s3transfer. It's under
# GPLv3, and so is manually removed.
RUN rm -Rf /opt/rh/python27/root/usr/lib/python2.7/site-packages/docutils && \
scl enable python27 "pip freeze" | grep -v '^-e' | awk -F == '{print $1}' | grep -v docutils > piplist.txt && \
scl enable python27 "xargs -a piplist.txt pip --disable-pip-version-check show" > pipinfo.txt && \
scl enable python36 "pip freeze" | grep -v '^-e' | awk -F == '{print $1}' | grep -v docutils > piplist.txt && \
scl enable python36 "xargs -a piplist.txt pip --disable-pip-version-check show" > pipinfo.txt && \
test -z "$(cat pipinfo.txt | grep GPL | grep -v LGPL)" && \
rm -f piplist.txt pipinfo.txt

115
Dockerfile.rhel8 Normal file
View File

@ -0,0 +1,115 @@
FROM registry.access.redhat.com/ubi8:8.1
LABEL maintainer "thomasmckay@redhat.com"
ENV OS=linux \
ARCH=amd64 \
PYTHON_VERSION=3.6 \
PATH=$HOME/.local/bin/:$PATH \
PYTHONUNBUFFERED=1 \
PYTHONIOENCODING=UTF-8 \
LC_ALL=C.UTF-8 \
LANG=C.UTF-8 \
PIP_NO_CACHE_DIR=off
ENV QUAYDIR /quay-registry
ENV QUAYCONF /quay-registry/conf
ENV QUAYPATH "."
RUN mkdir $QUAYDIR
WORKDIR $QUAYDIR
RUN INSTALL_PKGS="\
python3 \
nginx \
openldap \
gcc-c++ git \
openldap-devel \
gpgme-devel \
python3-devel \
python3-gpg \
dnsmasq \
memcached \
openssl \
skopeo \
" && \
yum -y --setopt=tsflags=nodocs --setopt=skip_missing_names_on_install=False install $INSTALL_PKGS && \
yum -y update && \
yum -y clean all
COPY . .
RUN alternatives --set python /usr/bin/python3 && \
python -m pip install --upgrade setuptools pip && \
python -m pip install -r requirements.txt --no-cache && \
python -m pip freeze && \
mkdir -p $QUAYDIR/static/webfonts && \
mkdir -p $QUAYDIR/static/fonts && \
mkdir -p $QUAYDIR/static/ldn && \
PYTHONPATH=$QUAYPATH python -m external_libraries && \
cp -r $QUAYDIR/static/ldn $QUAYDIR/config_app/static/ldn && \
cp -r $QUAYDIR/static/fonts $QUAYDIR/config_app/static/fonts && \
cp -r $QUAYDIR/static/webfonts $QUAYDIR/config_app/static/webfonts
RUN curl --silent --location https://rpm.nodesource.com/setup_8.x | bash - && \
yum install -y nodejs && \
curl --silent --location https://dl.yarnpkg.com/rpm/yarn.repo | tee /etc/yum.repos.d/yarn.repo && \
rpm --import https://dl.yarnpkg.com/rpm/pubkey.gpg && \
yum install -y yarn && \
yarn install --ignore-engines && \
yarn build && \
yarn build-config-app
ENV JWTPROXY_VERSION=0.0.3
RUN curl -fsSL -o /usr/local/bin/jwtproxy "https://github.com/coreos/jwtproxy/releases/download/v${JWTPROXY_VERSION}/jwtproxy-${OS}-${ARCH}" && \
chmod +x /usr/local/bin/jwtproxy
ENV PUSHGATEWAY_VERSION=1.0.0
RUN curl -fsSL "https://github.com/prometheus/pushgateway/releases/download/v${PUSHGATEWAY_VERSION}/pushgateway-${PUSHGATEWAY_VERSION}.${OS}-${ARCH}.tar.gz" | \
tar xz "pushgateway-${PUSHGATEWAY_VERSION}.${OS}-${ARCH}/pushgateway" && \
mv "pushgateway-${PUSHGATEWAY_VERSION}.${OS}-${ARCH}/pushgateway" /usr/local/bin/pushgateway && \
rm -rf "pushgateway-${PUSHGATEWAY_VERSION}.${OS}-${ARCH}" && \
chmod +x /usr/local/bin/pushgateway
# Update local copy of AWS IP Ranges.
RUN curl -fsSL https://ip-ranges.amazonaws.com/ip-ranges.json -o util/ipresolver/aws-ip-ranges.json
RUN ln -s $QUAYCONF /conf && \
ln -sf /dev/stdout /var/log/nginx/access.log && \
ln -sf /dev/stdout /var/log/nginx/error.log && \
chmod -R a+rwx /var/log/nginx
# Cleanup
RUN UNINSTALL_PKGS="\
gcc-c++ git \
openldap-devel \
gpgme-devel \
python3-devel \
optipng \
kernel-headers \
" && \
yum remove -y $UNINSTALL_PKGS && \
yum clean all && \
rm -rf /var/cache/yum /tmp/* /var/tmp/* /root/.cache
EXPOSE 8080 8443 7443 9091
RUN chgrp -R 0 $QUAYDIR && \
chmod -R g=u $QUAYDIR
RUN mkdir /datastorage && chgrp 0 /datastorage && chmod g=u /datastorage && \
chgrp 0 /var/log/nginx && chmod g=u /var/log/nginx && \
mkdir -p /conf/stack && chgrp 0 /conf/stack && chmod g=u /conf/stack && \
mkdir -p /tmp && chgrp 0 /tmp && chmod g=u /tmp && \
mkdir /certificates && chgrp 0 /certificates && chmod g=u /certificates && \
chmod g=u /etc/passwd
VOLUME ["/var/log", "/datastorage", "/tmp", "/conf/stack"]
ENTRYPOINT ["/quay-registry/quay-entrypoint.sh"]
CMD ["registry"]
# root required to create and install certs
# https://jira.coreos.com/browse/QUAY-1468
# USER 1001

3
Makefile
View File

@ -173,4 +173,5 @@ yapf-test:
black:
black --line-length 100 --target-version py27 .
black --line-length 100 --target-version py36 --exclude "/(\.eggs|\.git|\.hg|\.mypy_cache|\.nox|\.tox|\.venv|_build|buck-out|build|dist|buildman)/" . # TODO(kleesc): Re-enable after buildman rewrite

2
app.py
View File

@ -136,7 +136,7 @@ if features.EXPERIMENTAL_HELM_OCI_SUPPORT:
HELM_CHART_LAYER_TYPES = ["application/tar+gzip"]
register_artifact_type(HELM_CHART_CONFIG_TYPE, HELM_CHART_LAYER_TYPES)
CONFIG_DIGEST = hashlib.sha256(json.dumps(app.config, default=str)).hexdigest()[0:8]
CONFIG_DIGEST = hashlib.sha256(json.dumps(app.config, default=str).encode("utf-8")).hexdigest()[0:8]
logger.debug("Loaded config", extra={"config": app.config})

2
auth/auth_context_type.py
View File

@ -150,7 +150,7 @@ class ValidatedAuthContext(AuthContext):
self.signed_data = signed_data
def tuple(self):
return vars(self).values()
return list(vars(self).values())
def __eq__(self, other):
return self.tuple() == other.tuple()

4
auth/basic.py
View File

@ -33,7 +33,7 @@ def validate_basic_auth(auth_header):
logger.debug("Attempt to process basic auth header")
# Parse the basic auth header.
assert isinstance(auth_header, basestring)
assert isinstance(auth_header, str)
credentials, err = _parse_basic_auth_header(auth_header)
if err is not None:
logger.debug("Got invalid basic auth header: %s", auth_header)
@ -53,7 +53,7 @@ def _parse_basic_auth_header(auth):
return None, "Invalid basic auth header"
try:
credentials = [part.decode("utf-8") for part in b64decode(normalized[1]).split(":", 1)]
credentials = [part.decode("utf-8") for part in b64decode(normalized[1]).split(b":", 1)]
except (TypeError, UnicodeDecodeError, ValueError):
logger.exception("Exception when parsing basic auth header: %s", auth)
return None, "Could not parse basic auth header"

2
auth/registry_jwt_auth.py
View File

@ -155,7 +155,7 @@ def process_registry_jwt_auth(scopes=None):
abort(
401,
message=ije.message,
message=str(ije),
headers=get_auth_headers(repository=repository, scopes=scopes),
)
else:

23
auth/test/test_basic.py
View File

@ -17,9 +17,10 @@ from test.fixtures import *
def _token(username, password):
assert isinstance(username, basestring)
assert isinstance(password, basestring)
return "basic " + b64encode("%s:%s" % (username, password))
assert isinstance(username, str)
assert isinstance(password, str)
token_bytes = b"%s:%s" % (username.encode("utf-8"), password.encode("utf-8"))
return "basic " + b64encode(token_bytes).decode("ascii")
@pytest.mark.parametrize(
@ -62,6 +63,10 @@ def _token(username, password):
error_message="This user has been disabled. Please contact your administrator.",
),
),
(
_token("usér", "passwôrd"),
ValidateResult(AuthKind.basic, error_message="Invalid Username or Password"),
),
],
)
def test_validate_basic_auth_token(token, expected_result, app):
@ -110,15 +115,15 @@ def test_valid_app_specific_token(app):
def test_invalid_unicode(app):
token = "\xebOH"
header = "basic " + b64encode(token)
token = b"\xebOH"
header = "basic " + b64encode(token).decode("ascii")
result = validate_basic_auth(header)
assert result == ValidateResult(AuthKind.basic, missing=True)
def test_invalid_unicode_2(app):
token = "“4JPCOLIVMAY32Q3XGVPHC4CBF8SKII5FWNYMASOFDIVSXTC5I5NBU”"
header = "basic " + b64encode("devtable+somerobot:%s" % token)
token = "“4JPCOLIVMAY32Q3XGVPHC4CBF8SKII5FWNYMASOFDIVSXTC5I5NBU”".encode("utf-8")
header = "basic " + b64encode(b"devtable+somerobot:%s" % token).decode("ascii")
result = validate_basic_auth(header)
assert result == ValidateResult(
AuthKind.basic,
@ -128,7 +133,9 @@ def test_invalid_unicode_2(app):
def test_invalid_unicode_3(app):
token = "sometoken"
header = "basic " + b64encode("“devtable+somerobot”:%s" % token)
auth = "“devtable+somerobot”:" + token
auth = auth.encode("utf-8")
header = "basic " + b64encode(auth).decode("ascii")
result = validate_basic_auth(header)
assert result == ValidateResult(
AuthKind.basic, error_message="Could not find robot with specified username",

2
auth/test/test_credentials.py
View File

@ -156,7 +156,7 @@ def test_invalid_unicode_robot(app):
result, kind = validate_credentials("devtable+somerobot", token)
assert kind == CredentialKind.robot
assert not result.auth_valid
msg = "Could not find robot with specified username"
msg = "Could not find robot with username: devtable+somerobot and supplied password."
assert result == ValidateResult(AuthKind.credentials, error_message=msg)

4
auth/test/test_registry_jwt.py
View File

@ -73,7 +73,7 @@ def _token(token_data, key_id=None, private_key=None, skip_header=False, alg=Non
token_headers = {}
token_data = jwt.encode(token_data, private_key, alg or "RS256", headers=token_headers)
return "Bearer {0}".format(token_data)
return "Bearer {0}".format(token_data.decode("ascii"))
def _parse_token(token):
@ -228,7 +228,7 @@ def test_mixing_keys_e2e(initialized_db):
_parse_token(deleted_key_token)
@pytest.mark.parametrize("token", [u"someunicodetoken✡", u"\xc9\xad\xbd",])
@pytest.mark.parametrize("token", ["someunicodetoken✡", "\xc9\xad\xbd",])
def test_unicode_token(token):
with pytest.raises(InvalidJWTException):
_parse_token(token)

5
avatars/avatars.py
View File

@ -4,6 +4,8 @@ import logging
from requests.exceptions import RequestException
from util.bytes import Bytes
logger = logging.getLogger(__name__)
@ -104,7 +106,8 @@ class BaseAvatar(object):
# Note: email_or_id may be None if gotten from external auth when email is disabled,
# so use the username in that case.
username_email_or_id = email_or_id or name
hash_value = hashlib.md5(username_email_or_id.strip().lower()).hexdigest()
username_email_or_id = Bytes.for_string_or_unicode(username_email_or_id).as_unicode()
hash_value = hashlib.md5(username_email_or_id.strip().lower().encode("utf-8")).hexdigest()
byte_count = int(math.ceil(math.log(len(colors), 16)))
byte_data = hash_value[0:byte_count]

17
bill-of-materials.json
View File

@ -194,16 +194,6 @@
"license": "Unlicense",
"project": "furl"
},
{
"format": "Python",
"license": "MIT License",
"project": "future"
},
{
"format": "Python",
"license": "PSF License",
"project": "futures"
},
{
"format": "Python",
"license": "Apache Software License 2.0",
@ -629,11 +619,6 @@
"license": "Apache Software License 2.0",
"project": "toposort"
},
{
"format": "Python",
"license": "Apache Software License 2.0",
"project": "trollius"
},
{
"format": "Python",
"license": "MIT License",
@ -3854,4 +3839,4 @@
"license": "MIT License",
"project": "zeroclipboard"
}
]
]

4
boot.py
View File

@ -1,7 +1,7 @@
#!/usr/bin/env python
from datetime import datetime, timedelta
from urlparse import urlunparse
from urllib.parse import urlunparse
from jinja2 import Template
from cachetools.func import lru_cache
@ -96,7 +96,7 @@ def setup_jwt_proxy():
with open(app.config["INSTANCE_SERVICE_KEY_LOCATION"], mode="w") as f:
f.truncate(0)
f.write(quay_key.exportKey())
f.write(quay_key.exportKey().decode("utf-8"))
# Generate the JWT proxy configuration.
audience = get_audience()

13
buildman/asyncutil.py
View File

@ -1,12 +1,12 @@
import asyncio
from concurrent.futures import ThreadPoolExecutor
from functools import partial
from trollius import get_event_loop, coroutine
def wrap_with_threadpool(obj, worker_threads=1):
"""
Wraps a class in an async executor so that it can be safely used in an event loop like trollius.
Wraps a class in an async executor so that it can be safely used in an event loop like asyncio.
"""
async_executor = ThreadPoolExecutor(worker_threads)
return AsyncWrapper(obj, executor=async_executor), async_executor
@ -14,12 +14,12 @@ def wrap_with_threadpool(obj, worker_threads=1):
class AsyncWrapper(object):
"""
Wrapper class which will transform a syncronous library to one that can be used with trollius
Wrapper class which will transform a syncronous library to one that can be used with asyncio
coroutines.
"""
def __init__(self, delegate, loop=None, executor=None):
self._loop = loop if loop is not None else get_event_loop()
self._loop = loop if loop is not None else asyncio.get_event_loop()
self._delegate = delegate
self._executor = executor
@ -39,7 +39,6 @@ class AsyncWrapper(object):
return wrapper
@coroutine
def __call__(self, *args, **kwargs):
async def __call__(self, *args, **kwargs):
callable_delegate_attr = partial(self._delegate, *args, **kwargs)
return self._loop.run_in_executor(self._executor, callable_delegate_attr)

2
buildman/builder.py
View File

@ -12,7 +12,7 @@ from buildman.manager.enterprise import EnterpriseManager
from buildman.manager.ephemeral import EphemeralBuilderManager
from buildman.server import BuilderServer
from trollius import SSLContext
from ssl import SSLContext
from raven.handlers.logging import SentryHandler
from raven.conf import setup_logging

139
buildman/component/buildcomponent.py
View File

@ -3,10 +3,9 @@ import os
import time
import logging
import json
import trollius
import asyncio
from autobahn.wamp.exception import ApplicationError
from trollius import From, Return
from buildman.server import BuildJobResult
from buildman.component.basecomponent import BaseComponent
@ -73,22 +72,18 @@ class BuildComponent(BaseComponent):
def onConnect(self):
self.join(self.builder_realm)
@trollius.coroutine
def onJoin(self, details):
async def onJoin(self, details):
logger.debug("Registering methods and listeners for component %s", self.builder_realm)
yield From(self.register(self._on_ready, u"io.quay.buildworker.ready"))
yield From(
self.register(self._determine_cache_tag, u"io.quay.buildworker.determinecachetag")
)
yield From(self.register(self._ping, u"io.quay.buildworker.ping"))
yield From(self.register(self._on_log_message, u"io.quay.builder.logmessagesynchronously"))
await self.register(self._on_ready, "io.quay.buildworker.ready")
await (self.register(self._determine_cache_tag, "io.quay.buildworker.determinecachetag"))
await self.register(self._ping, "io.quay.buildworker.ping")
await self.register(self._on_log_message, "io.quay.builder.logmessagesynchronously")
yield From(self.subscribe(self._on_heartbeat, u"io.quay.builder.heartbeat"))
await self.subscribe(self._on_heartbeat, "io.quay.builder.heartbeat")
yield From(self._set_status(ComponentStatus.WAITING))
await self._set_status(ComponentStatus.WAITING)
@trollius.coroutine
def start_build(self, build_job):
async def start_build(self, build_job):
"""
Starts a build.
"""
@ -100,7 +95,7 @@ class BuildComponent(BaseComponent):
self._worker_version,
self._component_status,
)
raise Return()
return
logger.debug(
"Starting build for component %s (build %s, worker version: %s)",
@ -113,7 +108,7 @@ class BuildComponent(BaseComponent):
self._build_status = StatusHandler(self.build_logs, build_job.repo_build.uuid)
self._image_info = {}
yield From(self._set_status(ComponentStatus.BUILDING))
await self._set_status(ComponentStatus.BUILDING)
# Send the notification that the build has started.
build_job.send_notification("build_start")
@ -122,8 +117,8 @@ class BuildComponent(BaseComponent):
try:
build_config = build_job.build_config
except BuildJobLoadException as irbe:
yield From(self._build_failure("Could not load build job information", irbe))
raise Return()
await self._build_failure("Could not load build job information", irbe)
return
base_image_information = {}
@ -189,8 +184,8 @@ class BuildComponent(BaseComponent):
self._current_job.repo_build.uuid,
build_arguments,
)
yield From(self._build_failure("Insufficient build arguments. No buildpack available."))
raise Return()
await self._build_failure("Insufficient build arguments. No buildpack available.")
return
# Invoke the build.
logger.debug("Invoking build: %s", self.builder_realm)
@ -200,7 +195,7 @@ class BuildComponent(BaseComponent):
"""
This function is used to execute a coroutine as the callback.
"""
trollius.ensure_future(self._build_complete(result))
asyncio.create_task(self._build_complete(result))
self.call("io.quay.builder.build", **build_arguments).add_done_callback(
build_complete_callback
@ -285,8 +280,7 @@ class BuildComponent(BaseComponent):
images, max(len(images), num_images)
)
@trollius.coroutine
def _on_log_message(self, phase, json_data):
async def _on_log_message(self, phase, json_data):
"""
Tails log messages and updates the build status.
"""
@ -320,7 +314,7 @@ class BuildComponent(BaseComponent):
# the pull/push progress, as well as the current step index.
with self._build_status as status_dict:
try:
changed_phase = yield From(
changed_phase = await (
self._build_status.set_phase(phase, log_data.get("status_data"))
)
if changed_phase:
@ -330,11 +324,11 @@ class BuildComponent(BaseComponent):
logger.debug(
"Trying to move cancelled build into phase: %s with id: %s", phase, build_id
)
raise Return(False)
return False
except InvalidRepositoryBuildException:
build_id = self._current_job.repo_build.uuid
logger.warning("Build %s was not found; repo was probably deleted", build_id)
raise Return(False)
return False
BuildComponent._process_pushpull_status(status_dict, phase, log_data, self._image_info)
@ -345,16 +339,15 @@ class BuildComponent(BaseComponent):
# If the json data contains an error, then something went wrong with a push or pull.
if "error" in log_data:
yield From(self._build_status.set_error(log_data["error"]))
await self._build_status.set_error(log_data["error"])
if current_step is not None:
yield From(self._build_status.set_command(current_status_string))
await self._build_status.set_command(current_status_string)
elif phase == BUILD_PHASE.BUILDING:
yield From(self._build_status.append_log(current_status_string))
raise Return(True)
await self._build_status.append_log(current_status_string)
return True
@trollius.coroutine
def _determine_cache_tag(
async def _determine_cache_tag(
self, command_comments, base_image_name, base_image_tag, base_image_id
):
with self._build_status as status_dict:
@ -369,14 +362,13 @@ class BuildComponent(BaseComponent):
)
tag_found = self._current_job.determine_cached_tag(base_image_id, command_comments)
raise Return(tag_found or "")
return tag_found or ""
@trollius.coroutine
def _build_failure(self, error_message, exception=None):
async def _build_failure(self, error_message, exception=None):
"""
Handles and logs a failed build.
"""
yield From(
await (
self._build_status.set_error(
error_message, {"internal_error": str(exception) if exception else None}
)
@ -386,10 +378,9 @@ class BuildComponent(BaseComponent):
logger.warning("Build %s failed with message: %s", build_id, error_message)
# Mark that the build has finished (in an error state)
yield From(self._build_finished(BuildJobResult.ERROR))
await self._build_finished(BuildJobResult.ERROR)
@trollius.coroutine
def _build_complete(self, result):
async def _build_complete(self, result):
"""
Wraps up a completed build.
@ -411,12 +402,12 @@ class BuildComponent(BaseComponent):
pass
try:
yield From(self._build_status.set_phase(BUILD_PHASE.COMPLETE))
await self._build_status.set_phase(BUILD_PHASE.COMPLETE)
except InvalidRepositoryBuildException:
logger.warning("Build %s was not found; repo was probably deleted", build_id)
raise Return()
return
yield From(self._build_finished(BuildJobResult.COMPLETE))
await self._build_finished(BuildJobResult.COMPLETE)
# Label the pushed manifests with the build metadata.
manifest_digests = kwargs.get("digests") or []
@ -444,7 +435,7 @@ class BuildComponent(BaseComponent):
worker_error = WorkerError(aex.error, aex.kwargs.get("base_error"))
# Write the error to the log.
yield From(
await (
self._build_status.set_error(
worker_error.public_message(),
worker_error.extra_data(),
@ -465,23 +456,22 @@ class BuildComponent(BaseComponent):
build_id,
worker_error.public_message(),
)
yield From(self._build_finished(BuildJobResult.INCOMPLETE))
await self._build_finished(BuildJobResult.INCOMPLETE)
else:
logger.debug("Got remote failure exception for build %s: %s", build_id, aex)
yield From(self._build_finished(BuildJobResult.ERROR))
await self._build_finished(BuildJobResult.ERROR)
# Remove the current job.
self._current_job = None
@trollius.coroutine
def _build_finished(self, job_status):
async def _build_finished(self, job_status):
"""
Alerts the parent that a build has completed and sets the status back to running.
"""
yield From(self.parent_manager.job_completed(self._current_job, job_status, self))
await self.parent_manager.job_completed(self._current_job, job_status, self)
# Set the component back to a running state.
yield From(self._set_status(ComponentStatus.RUNNING))
await self._set_status(ComponentStatus.RUNNING)
@staticmethod
def _ping():
@ -490,8 +480,7 @@ class BuildComponent(BaseComponent):
"""
return "pong"
@trollius.coroutine
def _on_ready(self, token, version):
async def _on_ready(self, token, version):
logger.debug('On ready called (token "%s")', token)
self._worker_version = version
@ -499,30 +488,29 @@ class BuildComponent(BaseComponent):
logger.warning(
'Build component (token "%s") is running an out-of-date version: %s', token, version
)
raise Return(False)
return False
if self._component_status != ComponentStatus.WAITING:
logger.warning('Build component (token "%s") is already connected', self.expected_token)
raise Return(False)
return False
if token != self.expected_token:
logger.warning(
'Builder token mismatch. Expected: "%s". Found: "%s"', self.expected_token, token
)
raise Return(False)
return False
yield From(self._set_status(ComponentStatus.RUNNING))
await self._set_status(ComponentStatus.RUNNING)
# Start the heartbeat check and updating loop.
loop = trollius.get_event_loop()
loop = asyncio.get_event_loop()
loop.create_task(self._heartbeat())
logger.debug("Build worker %s is connected and ready", self.builder_realm)
raise Return(True)
return True
@trollius.coroutine
def _set_status(self, phase):
async def _set_status(self, phase):
if phase == ComponentStatus.RUNNING:
yield From(self.parent_manager.build_component_ready(self))
await self.parent_manager.build_component_ready(self)
self._component_status = phase
@ -536,15 +524,14 @@ class BuildComponent(BaseComponent):
logger.debug("Got heartbeat on realm %s", self.builder_realm)
self._last_heartbeat = datetime.datetime.utcnow()
@trollius.coroutine
def _heartbeat(self):
async def _heartbeat(self):
"""
Coroutine that runs every HEARTBEAT_TIMEOUT seconds, both checking the worker's heartbeat
and updating the heartbeat in the build status dictionary (if applicable).
This allows the build system to catch crashes from either end.
"""
yield From(trollius.sleep(INITIAL_TIMEOUT))
await asyncio.sleep(INITIAL_TIMEOUT)
while True:
# If the component is no longer running or actively building, nothing more to do.
@ -552,7 +539,7 @@ class BuildComponent(BaseComponent):
self._component_status != ComponentStatus.RUNNING
and self._component_status != ComponentStatus.BUILDING
):
raise Return()
return
# If there is an active build, write the heartbeat to its status.
if self._build_status is not None:
@ -562,7 +549,7 @@ class BuildComponent(BaseComponent):
# Mark the build item.
current_job = self._current_job
if current_job is not None:
yield From(self.parent_manager.job_heartbeat(current_job))
await self.parent_manager.job_heartbeat(current_job)
# Check the heartbeat from the worker.
logger.debug("Checking heartbeat on realm %s", self.builder_realm)
@ -576,8 +563,8 @@ class BuildComponent(BaseComponent):
self._last_heartbeat,
)
yield From(self._timeout())
raise Return()
await self._timeout()
return
logger.debug(
"Heartbeat on realm %s is valid: %s (%s).",
@ -586,20 +573,19 @@ class BuildComponent(BaseComponent):
self._component_status,
)
yield From(trollius.sleep(HEARTBEAT_TIMEOUT))
await asyncio.sleep(HEARTBEAT_TIMEOUT)
@trollius.coroutine
def _timeout(self):
async def _timeout(self):
if self._component_status == ComponentStatus.TIMED_OUT:
raise Return()
return
yield From(self._set_status(ComponentStatus.TIMED_OUT))
await self._set_status(ComponentStatus.TIMED_OUT)
logger.warning("Build component with realm %s has timed out", self.builder_realm)
# If we still have a running job, then it has not completed and we need to tell the parent
# manager.
if self._current_job is not None:
yield From(
await (
self._build_status.set_error(
"Build worker timed out",
internal_error=True,
@ -609,7 +595,7 @@ class BuildComponent(BaseComponent):
build_id = self._current_job.build_uuid
logger.error("[BUILD INTERNAL ERROR: Timeout] Build ID: %s", build_id)
yield From(
await (
self.parent_manager.job_completed(
self._current_job, BuildJobResult.INCOMPLETE, self
)
@ -621,8 +607,7 @@ class BuildComponent(BaseComponent):
# Remove the job reference.
self._current_job = None
@trollius.coroutine
def cancel_build(self):
async def cancel_build(self):
self.parent_manager.build_component_disposed(self, True)
self._current_job = None
yield From(self._set_status(ComponentStatus.RUNNING))
await self._set_status(ComponentStatus.RUNNING)

36
buildman/jobutil/buildstatus.py
View File

@ -2,7 +2,6 @@ import datetime
import logging
from redis import RedisError
from trollius import From, Return, coroutine
from data.database import BUILD_PHASE
from data import model
@ -35,54 +34,47 @@ class StatusHandler(object):
# Write the initial status.
self.__exit__(None, None, None)
@coroutine
def _append_log_message(self, log_message, log_type=None, log_data=None):
async def _append_log_message(self, log_message, log_type=None, log_data=None):
log_data = log_data or {}
log_data["datetime"] = str(datetime.datetime.now())
try:
yield From(
self._build_logs.append_log_message(self._uuid, log_message, log_type, log_data)
)
await (self._build_logs.append_log_message(self._uuid, log_message, log_type, log_data))
except RedisError:
logger.exception("Could not save build log for build %s: %s", self._uuid, log_message)
@coroutine
def append_log(self, log_message, extra_data=None):
async def append_log(self, log_message, extra_data=None):
if log_message is None:
return
yield From(self._append_log_message(log_message, log_data=extra_data))
await self._append_log_message(log_message, log_data=extra_data)
@coroutine
def set_command(self, command, extra_data=None):
async def set_command(self, command, extra_data=None):
if self._current_command == command:
raise Return()
return
self._current_command = command
yield From(self._append_log_message(command, self._build_logs.COMMAND, extra_data))
await self._append_log_message(command, self._build_logs.COMMAND, extra_data)
@coroutine
def set_error(self, error_message, extra_data=None, internal_error=False, requeued=False):
async def set_error(self, error_message, extra_data=None, internal_error=False, requeued=False):
error_phase = (
BUILD_PHASE.INTERNAL_ERROR if internal_error and requeued else BUILD_PHASE.ERROR
)
yield From(self.set_phase(error_phase))
await self.set_phase(error_phase)
extra_data = extra_data or {}
extra_data["internal_error"] = internal_error
yield From(self._append_log_message(error_message, self._build_logs.ERROR, extra_data))
await self._append_log_message(error_message, self._build_logs.ERROR, extra_data)
@coroutine
def set_phase(self, phase, extra_data=None):
async def set_phase(self, phase, extra_data=None):
if phase == self._current_phase:
raise Return(False)
return False
self._current_phase = phase
yield From(self._append_log_message(phase, self._build_logs.PHASE, extra_data))
await self._append_log_message(phase, self._build_logs.PHASE, extra_data)
# Update the repository build with the new phase
raise Return(self._build_model.update_phase_then_close(self._uuid, phase))
return self._build_model.update_phase_then_close(self._uuid, phase)
def __enter__(self):
return self._status

51
buildman/manager/basemanager.py
View File

@ -1,11 +1,22 @@
from trollius import coroutine
from abc import abstractmethod, ABC
import inspect
class BaseManager(object):
class BaseManager(ABC):
"""
Base for all worker managers.
"""
def __new__(cls, *args, **kwargs):
"""Hack to ensure method defined as async are implemented as such. """
coroutines = inspect.getmembers(BaseManager, predicate=inspect.iscoroutinefunction)
for coroutine in coroutines:
implemented_method = getattr(cls, coroutine[0])
if not inspect.iscoroutinefunction(implemented_method):
raise RuntimeError("The method %s must be a coroutine" % implemented_method)
return super().__new__(cls, *args, **kwargs)
def __init__(
self,
register_component,
@ -22,8 +33,7 @@ class BaseManager(object):
self.manager_hostname = manager_hostname
self.heartbeat_period_sec = heartbeat_period_sec
@coroutine
def job_heartbeat(self, build_job):
async def job_heartbeat(self, build_job):
"""
Method invoked to tell the manager that a job is still running.
@ -31,13 +41,15 @@ class BaseManager(object):
"""
self.job_heartbeat_callback(build_job)
@abstractmethod
def overall_setup_time(self):
"""
Returns the number of seconds that the build system should wait before allowing the job to
be picked up again after called 'schedule'.
"""
raise NotImplementedError
pass
@abstractmethod
def shutdown(self):
"""
Indicates that the build controller server is in a shutdown state and that no new jobs or
@ -45,43 +57,45 @@ class BaseManager(object):
Existing workers should be cleaned up once their jobs have completed
"""
raise NotImplementedError
pass
@coroutine
def schedule(self, build_job):
@abstractmethod
async def schedule(self, build_job):
"""
Schedules a queue item to be built.
Returns a 2-tuple with (True, None) if the item was properly scheduled and (False, a retry
timeout in seconds) if all workers are busy or an error occurs.
"""
raise NotImplementedError
pass
@abstractmethod
def initialize(self, manager_config):
"""
Runs any initialization code for the manager.
Called once the server is in a ready state.
"""
raise NotImplementedError
pass
@coroutine
def build_component_ready(self, build_component):
@abstractmethod
async def build_component_ready(self, build_component):
"""
Method invoked whenever a build component announces itself as ready.
"""
raise NotImplementedError
pass
@abstractmethod
def build_component_disposed(self, build_component, timed_out):
"""
Method invoked whenever a build component has been disposed.
The timed_out boolean indicates whether the component's heartbeat timed out.
"""
raise NotImplementedError
pass
@coroutine
def job_completed(self, build_job, job_status, build_component):
@abstractmethod
async def job_completed(self, build_job, job_status, build_component):
"""
Method invoked once a job_item has completed, in some manner.
@ -89,12 +103,13 @@ class BaseManager(object):
should call coroutine self.job_complete_callback with a status of Incomplete if they wish
for the job to be automatically requeued.
"""
raise NotImplementedError
pass
@abstractmethod
def num_workers(self):
"""
Returns the number of active build workers currently registered.
This includes those that are currently busy and awaiting more work.
"""
raise NotImplementedError
pass

23
buildman/manager/enterprise.py
View File

@ -5,8 +5,6 @@ from buildman.component.basecomponent import BaseComponent
from buildman.component.buildcomponent import BuildComponent
from buildman.manager.basemanager import BaseManager
from trollius import From, Return, coroutine
REGISTRATION_REALM = "registration"
RETRY_TIMEOUT = 5
logger = logging.getLogger(__name__)
@ -20,9 +18,9 @@ class DynamicRegistrationComponent(BaseComponent):
def onConnect(self):
self.join(REGISTRATION_REALM)
def onJoin(self, details):
async def onJoin(self, details):
logger.debug("Registering registration method")
yield From(self.register(self._worker_register, u"io.quay.buildworker.register"))
await self.register(self._worker_register, "io.quay.buildworker.register")
def _worker_register(self):
realm = self.parent_manager.add_build_component()
@ -65,30 +63,27 @@ class EnterpriseManager(BaseManager):
self.all_components.add(new_component)
return realm
@coroutine
def schedule(self, build_job):
async def schedule(self, build_job):
"""
Schedules a build for an Enterprise Registry.
"""
if self.shutting_down or not self.ready_components:
raise Return(False, RETRY_TIMEOUT)
return False, RETRY_TIMEOUT
component = self.ready_components.pop()
yield From(component.start_build(build_job))
await component.start_build(build_job)
raise Return(True, None)
return True, None
@coroutine
def build_component_ready(self, build_component):
async def build_component_ready(self, build_component):
self.ready_components.add(build_component)
def shutdown(self):
self.shutting_down = True
@coroutine
def job_completed(self, build_job, job_status, build_component):
yield From(self.job_complete_callback(build_job, job_status))
async def job_completed(self, build_job, job_status, build_component):
await self.job_complete_callback(build_job, job_status)
def build_component_disposed(self, build_component, timed_out):
self.all_components.remove(build_component)

161
buildman/manager/ephemeral.py
View File

@ -1,3 +1,4 @@
import asyncio
import logging
import uuid
import calendar
@ -9,7 +10,6 @@ from datetime import datetime, timedelta
from six import iteritems
from prometheus_client import Counter, Histogram
from trollius import From, coroutine, Return, async, sleep
from buildman.orchestrator import (
orchestrator_from_config,
@ -98,8 +98,7 @@ class EphemeralBuilderManager(BaseManager):
def overall_setup_time(self):
return EPHEMERAL_SETUP_TIMEOUT
@coroutine
def _mark_job_incomplete(self, build_job, build_info):
async def _mark_job_incomplete(self, build_job, build_info):
"""
Marks a job as incomplete, in response to a failure to start or a timeout.
"""
@ -113,11 +112,11 @@ class EphemeralBuilderManager(BaseManager):
# Take a lock to ensure that only one manager reports the build as incomplete for this
# execution.
lock_key = slash_join(self._expired_lock_prefix, build_job.build_uuid, execution_id)
acquired_lock = yield From(self._orchestrator.lock(lock_key))
acquired_lock = await self._orchestrator.lock(lock_key)
if acquired_lock:
try:
# Clean up the bookkeeping for the job.
yield From(self._orchestrator.delete_key(self._job_key(build_job)))
await self._orchestrator.delete_key(self._job_key(build_job))
except KeyError:
logger.debug(
"Could not delete job key %s; might have been removed already",
@ -130,7 +129,7 @@ class EphemeralBuilderManager(BaseManager):
executor_name,
execution_id,
)
yield From(
await (
self.job_complete_callback(
build_job, BuildJobResult.INCOMPLETE, executor_name, update_phase=True
)
@ -138,8 +137,7 @@ class EphemeralBuilderManager(BaseManager):
else:
logger.debug("Did not get lock for job-expiration for job %s", build_job.build_uuid)
@coroutine
def _job_callback(self, key_change):
async def _job_callback(self, key_change):
"""
This is the callback invoked when keys related to jobs are changed. It ignores all events
related to the creation of new jobs. Deletes or expirations cause checks to ensure they've
@ -149,7 +147,7 @@ class EphemeralBuilderManager(BaseManager):
:type key_change: :class:`KeyChange`
"""
if key_change.event in (KeyEvent.CREATE, KeyEvent.SET):
raise Return()
return
elif key_change.event in (KeyEvent.DELETE, KeyEvent.EXPIRE):
# Handle the expiration/deletion.
@ -166,13 +164,13 @@ class EphemeralBuilderManager(BaseManager):
build_job.build_uuid,
job_metadata,
)
raise Return()
return
if key_change.event != KeyEvent.EXPIRE:
# If the etcd action was not an expiration, then it was already deleted by some manager and
# the execution was therefore already shutdown. All that's left is to remove the build info.
self._build_uuid_to_info.pop(build_job.build_uuid, None)
raise Return()
return
logger.debug(
"got expiration for job %s with metadata: %s", build_job.build_uuid, job_metadata
@ -181,7 +179,7 @@ class EphemeralBuilderManager(BaseManager):
if not job_metadata.get("had_heartbeat", False):
# If we have not yet received a heartbeat, then the node failed to boot in some way.
# We mark the job as incomplete here.
yield From(self._mark_job_incomplete(build_job, build_info))
await self._mark_job_incomplete(build_job, build_info)
# Finally, we terminate the build execution for the job. We don't do this under a lock as
# terminating a node is an atomic operation; better to make sure it is terminated than not.
@ -190,14 +188,13 @@ class EphemeralBuilderManager(BaseManager):
build_job.build_uuid,
build_info.execution_id,
)
yield From(self.kill_builder_executor(build_job.build_uuid))
await self.kill_builder_executor(build_job.build_uuid)
else:
logger.warning(
"Unexpected KeyEvent (%s) on job key: %s", key_change.event, key_change.key
)
@coroutine
def _realm_callback(self, key_change):
async def _realm_callback(self, key_change):
logger.debug("realm callback for key: %s", key_change.key)
if key_change.event == KeyEvent.CREATE:
# Listen on the realm created by ourselves or another worker.
@ -231,7 +228,7 @@ class EphemeralBuilderManager(BaseManager):
# Cleanup the job, since it never started.
logger.debug("Job %s for incomplete marking: %s", build_uuid, build_info)
if build_info is not None:
yield From(self._mark_job_incomplete(build_job, build_info))
await self._mark_job_incomplete(build_job, build_info)
# Cleanup the executor.
logger.info(
@ -241,7 +238,7 @@ class EphemeralBuilderManager(BaseManager):
executor_name,
execution_id,
)
yield From(self.terminate_executor(executor_name, execution_id))
await self.terminate_executor(executor_name, execution_id)
else:
logger.warning(
@ -278,10 +275,9 @@ class EphemeralBuilderManager(BaseManager):
def registered_executors(self):
return self._ordered_executors
@coroutine
def _register_existing_realms(self):
async def _register_existing_realms(self):
try:
all_realms = yield From(self._orchestrator.get_prefixed_keys(self._realm_prefix))
all_realms = await self._orchestrator.get_prefixed_keys(self._realm_prefix)
# Register all existing realms found.
encountered = {
@ -400,22 +396,21 @@ class EphemeralBuilderManager(BaseManager):
)
# Load components for all realms currently known to the cluster
async(self._register_existing_realms())
asyncio.create_task(self._register_existing_realms())
def shutdown(self):
logger.debug("Shutting down worker.")
if self._orchestrator is not None:
self._orchestrator.shutdown()
@coroutine
def schedule(self, build_job):
async def schedule(self, build_job):
build_uuid = build_job.job_details["build_uuid"]
logger.debug("Calling schedule with job: %s", build_uuid)
# Check if there are worker slots available by checking the number of jobs in the orchestrator
allowed_worker_count = self._manager_config.get("ALLOWED_WORKER_COUNT", 1)
try:
active_jobs = yield From(self._orchestrator.get_prefixed_keys(self._job_prefix))
active_jobs = await self._orchestrator.get_prefixed_keys(self._job_prefix)
workers_alive = len(active_jobs)
except KeyError:
workers_alive = 0
@ -423,12 +418,12 @@ class EphemeralBuilderManager(BaseManager):
logger.exception(
"Could not read job count from orchestrator for job due to orchestrator being down"
)
raise Return(False, ORCHESTRATOR_UNAVAILABLE_SLEEP_DURATION)
return False, ORCHESTRATOR_UNAVAILABLE_SLEEP_DURATION
except OrchestratorError:
logger.exception(
"Exception when reading job count from orchestrator for job: %s", build_uuid
)
raise Return(False, RETRY_IMMEDIATELY_SLEEP_DURATION)
return False, RETRY_IMMEDIATELY_SLEEP_DURATION
logger.debug("Total jobs (scheduling job %s): %s", build_uuid, workers_alive)
@ -439,7 +434,7 @@ class EphemeralBuilderManager(BaseManager):
workers_alive,
allowed_worker_count,
)
raise Return(False, TOO_MANY_WORKERS_SLEEP_DURATION)
return False, TOO_MANY_WORKERS_SLEEP_DURATION
job_key = self._job_key(build_job)
@ -466,7 +461,7 @@ class EphemeralBuilderManager(BaseManager):
)
try:
yield From(
await (
self._orchestrator.set_key(
job_key, lock_payload, overwrite=False, expiration=EPHEMERAL_SETUP_TIMEOUT
)
@ -475,15 +470,15 @@ class EphemeralBuilderManager(BaseManager):
logger.warning(
"Job: %s already exists in orchestrator, timeout may be misconfigured", build_uuid
)
raise Return(False, EPHEMERAL_API_TIMEOUT)
return False, EPHEMERAL_API_TIMEOUT
except OrchestratorConnectionError:
logger.exception(
"Exception when writing job %s to orchestrator; could not connect", build_uuid
)
raise Return(False, ORCHESTRATOR_UNAVAILABLE_SLEEP_DURATION)
return False, ORCHESTRATOR_UNAVAILABLE_SLEEP_DURATION
except OrchestratorError:
logger.exception("Exception when writing job %s to orchestrator", build_uuid)
raise Return(False, RETRY_IMMEDIATELY_SLEEP_DURATION)
return False, RETRY_IMMEDIATELY_SLEEP_DURATION
# Got a lock, now lets boot the job via one of the registered executors.
started_with_executor = None
@ -519,7 +514,7 @@ class EphemeralBuilderManager(BaseManager):
)
try:
execution_id = yield From(executor.start_builder(realm, token, build_uuid))
execution_id = await executor.start_builder(realm, token, build_uuid)
except:
logger.exception("Exception when starting builder for job: %s", build_uuid)
continue
@ -534,8 +529,8 @@ class EphemeralBuilderManager(BaseManager):
logger.error("Could not start ephemeral worker for build %s", build_uuid)
# Delete the associated build job record.
yield From(self._orchestrator.delete_key(job_key))
raise Return(False, EPHEMERAL_API_TIMEOUT)
await self._orchestrator.delete_key(job_key)
return False, EPHEMERAL_API_TIMEOUT
# Job was started!
logger.debug(
@ -551,7 +546,7 @@ class EphemeralBuilderManager(BaseManager):
)
try:
yield From(
await (
self._orchestrator.set_key(
self._metric_key(realm),
metric_spec,
@ -591,7 +586,7 @@ class EphemeralBuilderManager(BaseManager):
execution_id,
setup_time,
)
yield From(
await (
self._orchestrator.set_key(
self._realm_key(realm), realm_spec, expiration=setup_time
)
@ -600,12 +595,12 @@ class EphemeralBuilderManager(BaseManager):
logger.exception(
"Exception when writing realm %s to orchestrator for job %s", realm, build_uuid
)
raise Return(False, ORCHESTRATOR_UNAVAILABLE_SLEEP_DURATION)
return False, ORCHESTRATOR_UNAVAILABLE_SLEEP_DURATION
except OrchestratorError:
logger.exception(
"Exception when writing realm %s to orchestrator for job %s", realm, build_uuid
)
raise Return(False, setup_time)
return False, setup_time
logger.debug(
"Builder spawn complete for job %s using executor %s with ID %s ",
@ -613,10 +608,9 @@ class EphemeralBuilderManager(BaseManager):
started_with_executor.name,
execution_id,
)
raise Return(True, None)
return True, None
@coroutine
def build_component_ready(self, build_component):
async def build_component_ready(self, build_component):
logger.debug(
"Got component ready for component with realm %s", build_component.builder_realm
)
@ -631,7 +625,7 @@ class EphemeralBuilderManager(BaseManager):
"Could not find job for the build component on realm %s; component is ready",
build_component.builder_realm,
)
raise Return()
return
# Start the build job.
logger.debug(
@ -639,15 +633,13 @@ class EphemeralBuilderManager(BaseManager):
job.build_uuid,
build_component.builder_realm,
)
yield From(build_component.start_build(job))
await build_component.start_build(job)
yield From(self._write_duration_metric(build_ack_duration, build_component.builder_realm))
await self._write_duration_metric(build_ack_duration, build_component.builder_realm)
# Clean up the bookkeeping for allowing any manager to take the job.
try:
yield From(
self._orchestrator.delete_key(self._realm_key(build_component.builder_realm))
)
await (self._orchestrator.delete_key(self._realm_key(build_component.builder_realm)))
except KeyError:
logger.warning("Could not delete realm key %s", build_component.builder_realm)
@ -655,13 +647,12 @@ class EphemeralBuilderManager(BaseManager):
logger.debug("Calling build_component_disposed.")
self.unregister_component(build_component)
@coroutine
def job_completed(self, build_job, job_status, build_component):
async def job_completed(self, build_job, job_status, build_component):
logger.debug(
"Calling job_completed for job %s with status: %s", build_job.build_uuid, job_status
)
yield From(
await (
self._write_duration_metric(
build_duration, build_component.builder_realm, job_status=job_status
)
@ -671,66 +662,61 @@ class EphemeralBuilderManager(BaseManager):
# to ask for the phase to be updated as well.
build_info = self._build_uuid_to_info.get(build_job.build_uuid, None)
executor_name = build_info.executor_name if build_info else None
yield From(
self.job_complete_callback(build_job, job_status, executor_name, update_phase=False)
)
await (self.job_complete_callback(build_job, job_status, executor_name, update_phase=False))
# Kill the ephemeral builder.
yield From(self.kill_builder_executor(build_job.build_uuid))
await self.kill_builder_executor(build_job.build_uuid)
# Delete the build job from the orchestrator.
try:
job_key = self._job_key(build_job)
yield From(self._orchestrator.delete_key(job_key))
await self._orchestrator.delete_key(job_key)
except KeyError:
logger.debug("Builder is asking for job to be removed, but work already completed")
except OrchestratorConnectionError:
logger.exception("Could not remove job key as orchestrator is not available")
yield From(sleep(ORCHESTRATOR_UNAVAILABLE_SLEEP_DURATION))
raise Return()
await asyncio.sleep(ORCHESTRATOR_UNAVAILABLE_SLEEP_DURATION)
return
# Delete the metric from the orchestrator.
try:
metric_key = self._metric_key(build_component.builder_realm)
yield From(self._orchestrator.delete_key(metric_key))
await self._orchestrator.delete_key(metric_key)
except KeyError:
logger.debug("Builder is asking for metric to be removed, but key not found")
except OrchestratorConnectionError:
logger.exception("Could not remove metric key as orchestrator is not available")
yield From(sleep(ORCHESTRATOR_UNAVAILABLE_SLEEP_DURATION))
raise Return()
await asyncio.sleep(ORCHESTRATOR_UNAVAILABLE_SLEEP_DURATION)
return
logger.debug("job_completed for job %s with status: %s", build_job.build_uuid, job_status)
@coroutine
def kill_builder_executor(self, build_uuid):
async def kill_builder_executor(self, build_uuid):
logger.info("Starting termination of executor for job %s", build_uuid)
build_info = self._build_uuid_to_info.pop(build_uuid, None)
if build_info is None:
logger.debug(
"Build information not found for build %s; skipping termination", build_uuid
)
raise Return()
return
# Remove the build's component.
self._component_to_job.pop(build_info.component, None)
# Stop the build node/executor itself.
yield From(self.terminate_executor(build_info.executor_name, build_info.execution_id))
await self.terminate_executor(build_info.executor_name, build_info.execution_id)
@coroutine
def terminate_executor(self, executor_name, execution_id):
async def terminate_executor(self, executor_name, execution_id):
executor = self._executor_name_to_executor.get(executor_name)
if executor is None:
logger.error("Could not find registered executor %s", executor_name)
raise Return()
return
# Terminate the executor's execution.
logger.info("Terminating executor %s with execution id %s", executor_name, execution_id)
yield From(executor.stop_builder(execution_id))
await executor.stop_builder(execution_id)
@coroutine
def job_heartbeat(self, build_job):
async def job_heartbeat(self, build_job):
"""
:param build_job: the identifier for the build
:type build_job: str
@ -738,13 +724,12 @@ class EphemeralBuilderManager(BaseManager):
self.job_heartbeat_callback(build_job)
self._extend_job_in_orchestrator(build_job)
@coroutine
def _extend_job_in_orchestrator(self, build_job):
async def _extend_job_in_orchestrator(self, build_job):
try:
job_data = yield From(self._orchestrator.get_key(self._job_key(build_job)))
job_data = await self._orchestrator.get_key(self._job_key(build_job))
except KeyError:
logger.info("Job %s no longer exists in the orchestrator", build_job.build_uuid)
raise Return()
return
except OrchestratorConnectionError:
logger.exception("failed to connect when attempted to extend job")
@ -762,7 +747,7 @@ class EphemeralBuilderManager(BaseManager):
}
try:
yield From(
await (
self._orchestrator.set_key(
self._job_key(build_job), json.dumps(payload), expiration=ttl
)
@ -771,15 +756,14 @@ class EphemeralBuilderManager(BaseManager):
logger.exception(
"Could not update heartbeat for job as the orchestrator is not available"
)
yield From(sleep(ORCHESTRATOR_UNAVAILABLE_SLEEP_DURATION))
await asyncio.sleep(ORCHESTRATOR_UNAVAILABLE_SLEEP_DURATION)
@coroutine
def _write_duration_metric(self, metric, realm, job_status=None):
async def _write_duration_metric(self, metric, realm, job_status=None):
""" :returns: True if the metric was written, otherwise False
:rtype: bool
"""
try:
metric_data = yield From(self._orchestrator.get_key(self._metric_key(realm)))
metric_data = await self._orchestrator.get_key(self._metric_key(realm))
parsed_metric_data = json.loads(metric_data)
start_time = parsed_metric_data["start_time"]
executor = parsed_metric_data.get("executor_name", "unknown")
@ -799,25 +783,24 @@ class EphemeralBuilderManager(BaseManager):
"""
return len(self._component_to_job)
@coroutine
def _cancel_callback(self, key_change):
async def _cancel_callback(self, key_change):
if key_change.event not in (KeyEvent.CREATE, KeyEvent.SET):
raise Return()
return
build_uuid = key_change.value
build_info = self._build_uuid_to_info.get(build_uuid, None)
if build_info is None:
logger.debug('No build info for "%s" job %s', key_change.event, build_uuid)
raise Return(False)
return False
lock_key = slash_join(self._canceled_lock_prefix, build_uuid, build_info.execution_id)
lock_acquired = yield From(self._orchestrator.lock(lock_key))
lock_acquired = await self._orchestrator.lock(lock_key)
if lock_acquired:
builder_realm = build_info.component.builder_realm
yield From(self.kill_builder_executor(build_uuid))
yield From(self._orchestrator.delete_key(self._realm_key(builder_realm)))
yield From(self._orchestrator.delete_key(self._metric_key(builder_realm)))
yield From(self._orchestrator.delete_key(slash_join(self._job_prefix, build_uuid)))
await self.kill_builder_executor(build_uuid)
await self._orchestrator.delete_key(self._realm_key(builder_realm))
await self._orchestrator.delete_key(self._metric_key(builder_realm))
await self._orchestrator.delete_key(slash_join(self._job_prefix, build_uuid))
# This is outside the lock so we can un-register the component wherever it is registered to.
yield From(build_info.component.cancel_build())
await build_info.component.cancel_build()

63
buildman/manager/executor.py
View File

@ -1,3 +1,4 @@
import asyncio
import datetime
import hashlib
import logging
@ -16,7 +17,6 @@ import requests
from container_cloud_config import CloudConfigContext
from jinja2 import FileSystemLoader, Environment
from trollius import coroutine, sleep, From, Return, get_event_loop
from prometheus_client import Histogram
import release
@ -99,8 +99,7 @@ class BuilderExecutor(object):
"""
return self.executor_config.get("SETUP_TIME")
@coroutine
def start_builder(self, realm, token, build_uuid):
async def start_builder(self, realm, token, build_uuid):
"""
Create a builder with the specified config.
@ -108,8 +107,7 @@ class BuilderExecutor(object):
"""
raise NotImplementedError
@coroutine
def stop_builder(self, builder_id):
async def stop_builder(self, builder_id):
"""
Stop a builder which is currently running.
"""
@ -129,7 +127,7 @@ class BuilderExecutor(object):
# in the first X% of the character space, we allow this executor to be used.
staged_rollout = self.executor_config.get("STAGED_ROLLOUT")
if staged_rollout is not None:
bucket = int(hashlib.sha256(namespace).hexdigest()[-2:], 16)
bucket = int(hashlib.sha256(namespace.encode("utf-8")).hexdigest()[-2:], 16)
return bucket < (256 * staged_rollout)
# If there are no restrictions in place, we are free to use this executor.
@ -189,7 +187,7 @@ class EC2Executor(BuilderExecutor):
)
def __init__(self, *args, **kwargs):
self._loop = get_event_loop()
self._loop = asyncio.get_event_loop()
super(EC2Executor, self).__init__(*args, **kwargs)
def _get_conn(self):
@ -214,16 +212,15 @@ class EC2Executor(BuilderExecutor):
stack_amis = dict([stack.split("=") for stack in stack_list_string.split("|")])
return stack_amis[ec2_region]
@coroutine
@async_observe(build_start_duration, "ec2")
def start_builder(self, realm, token, build_uuid):
async def start_builder(self, realm, token, build_uuid):
region = self.executor_config["EC2_REGION"]
channel = self.executor_config.get("COREOS_CHANNEL", "stable")
coreos_ami = self.executor_config.get("COREOS_AMI", None)
if coreos_ami is None:
get_ami_callable = partial(self._get_coreos_ami, region, channel)
coreos_ami = yield From(self._loop.run_in_executor(None, get_ami_callable))
coreos_ami = await self._loop.run_in_executor(None, get_ami_callable)
user_data = self.generate_cloud_config(
realm, token, build_uuid, channel, self.manager_hostname
@ -250,7 +247,7 @@ class EC2Executor(BuilderExecutor):
interfaces = boto.ec2.networkinterface.NetworkInterfaceCollection(interface)
try:
reservation = yield From(
reservation = await (
ec2_conn.run_instances(
coreos_ami,
instance_type=self.executor_config["EC2_INSTANCE_TYPE"],
@ -273,12 +270,12 @@ class EC2Executor(BuilderExecutor):
launched = AsyncWrapper(reservation.instances[0])
# Sleep a few seconds to wait for AWS to spawn the instance.
yield From(sleep(_TAG_RETRY_SLEEP))
await asyncio.sleep(_TAG_RETRY_SLEEP)
# Tag the instance with its metadata.
for i in range(0, _TAG_RETRY_COUNT):
try:
yield From(
await (
launched.add_tags(
{
"Name": "Quay Ephemeral Builder",
@ -297,7 +294,7 @@ class EC2Executor(BuilderExecutor):
build_uuid,
i,
)
yield From(sleep(_TAG_RETRY_SLEEP))
await asyncio.sleep(_TAG_RETRY_SLEEP)
continue
raise ExecutorException("Unable to find builder instance.")
@ -305,13 +302,12 @@ class EC2Executor(BuilderExecutor):
logger.exception("Failed to write EC2 tags (attempt #%s)", i)
logger.debug("Machine with ID %s started for build %s", launched.id, build_uuid)
raise Return(launched.id)
return launched.id
@coroutine
def stop_builder(self, builder_id):
async def stop_builder(self, builder_id):
try:
ec2_conn = self._get_conn()
terminated_instances = yield From(ec2_conn.terminate_instances([builder_id]))
terminated_instances = await ec2_conn.terminate_instances([builder_id])
except boto.exception.EC2ResponseError as ec2e:
if ec2e.error_code == "InvalidInstanceID.NotFound":
logger.debug("Instance %s already terminated", builder_id)
@ -333,9 +329,8 @@ class PopenExecutor(BuilderExecutor):
self._jobs = {}
super(PopenExecutor, self).__init__(executor_config, manager_hostname)
@coroutine
@async_observe(build_start_duration, "fork")
def start_builder(self, realm, token, build_uuid):
async def start_builder(self, realm, token, build_uuid):
# Now start a machine for this job, adding the machine id to the etcd information
logger.debug("Forking process for build")
@ -362,10 +357,9 @@ class PopenExecutor(BuilderExecutor):
builder_id = str(uuid.uuid4())
self._jobs[builder_id] = (spawned, logpipe)
logger.debug("Builder spawned with id: %s", builder_id)
raise Return(builder_id)
return builder_id
@coroutine
def stop_builder(self, builder_id):
async def stop_builder(self, builder_id):
if builder_id not in self._jobs:
raise ExecutorException("Builder id not being tracked by executor.")
@ -384,14 +378,13 @@ class KubernetesExecutor(BuilderExecutor):
def __init__(self, *args, **kwargs):
super(KubernetesExecutor, self).__init__(*args, **kwargs)
self._loop = get_event_loop()
self._loop = asyncio.get_event_loop()
self.namespace = self.executor_config.get("BUILDER_NAMESPACE", "builder")
self.image = self.executor_config.get(
"BUILDER_VM_CONTAINER_IMAGE", "quay.io/quay/quay-builder-qemu-coreos:stable"
)
@coroutine
def _request(self, method, path, **kwargs):
async def _request(self, method, path, **kwargs):
request_options = dict(kwargs)
tls_cert = self.executor_config.get("K8S_API_TLS_CERT")
@ -422,7 +415,7 @@ class KubernetesExecutor(BuilderExecutor):
logger.debug("Kubernetes request: %s %s: %s", method, url, request_options)
res = requests.request(method, url, **request_options)
logger.debug("Kubernetes response: %s: %s", res.status_code, res.text)
raise Return(res)
return res
def _jobs_path(self):
return "/apis/batch/v1/namespaces/%s/jobs" % self.namespace
@ -566,9 +559,8 @@ class KubernetesExecutor(BuilderExecutor):
return job_resource
@coroutine
@async_observe(build_start_duration, "k8s")
def start_builder(self, realm, token, build_uuid):
async def start_builder(self, realm, token, build_uuid):
# generate resource
channel = self.executor_config.get("COREOS_CHANNEL", "stable")
user_data = self.generate_cloud_config(
@ -579,7 +571,7 @@ class KubernetesExecutor(BuilderExecutor):
logger.debug("Generated kubernetes resource:\n%s", resource)
# schedule
create_job = yield From(self._request("POST", self._jobs_path(), json=resource))
create_job = await self._request("POST", self._jobs_path(), json=resource)
if int(create_job.status_code / 100) != 2:
raise ExecutorException(
"Failed to create job: %s: %s: %s"
@ -587,24 +579,21 @@ class KubernetesExecutor(BuilderExecutor):
)
job = create_job.json()
raise Return(job["metadata"]["name"])
return job["metadata"]["name"]
@coroutine
def stop_builder(self, builder_id):
async def stop_builder(self, builder_id):
pods_path = "/api/v1/namespaces/%s/pods" % self.namespace
# Delete the job itself.
try:
yield From(self._request("DELETE", self._job_path(builder_id)))
await self._request("DELETE", self._job_path(builder_id))
except:
logger.exception("Failed to send delete job call for job %s", builder_id)
# Delete the pod(s) for the job.
selectorString = "job-name=%s" % builder_id
try:
yield From(
self._request("DELETE", pods_path, params=dict(labelSelector=selectorString))
)
await (self._request("DELETE", pods_path, params=dict(labelSelector=selectorString)))
except:
logger.exception("Failed to send delete pod call for job %s", builder_id)

202
buildman/orchestrator.py
View File

@ -1,6 +1,7 @@
from abc import ABCMeta, abstractmethod
from collections import namedtuple
import asyncio
import datetime
import json
import logging
@ -9,7 +10,6 @@ import time
from enum import IntEnum, unique
from six import add_metaclass, iteritems
from trollius import async, coroutine, From, Return
from urllib3.exceptions import ReadTimeoutError, ProtocolError
import etcd
@ -62,18 +62,18 @@ def orchestrator_from_config(manager_config, canceller_only=False):
}
# Sanity check that legacy prefixes are no longer being used.
for key in manager_config["ORCHESTRATOR"].keys():
for key in list(manager_config["ORCHESTRATOR"].keys()):
words = key.split("_")
if len(words) > 1 and words[-1].lower() == "prefix":
raise AssertionError("legacy prefix used, use ORCHESTRATOR_PREFIX instead")
def _dict_key_prefix(d):
"""
:param d: the dict that has keys prefixed with underscore
:type d: {str: any}
:rtype: str
"""
return d.keys()[0].split("_", 1)[0].lower()
:param d: the dict that has keys prefixed with underscore
:type d: {str: any}
:rtype: str
"""
return list(d.keys())[0].split("_", 1)[0].lower()
orchestrator_name = _dict_key_prefix(manager_config["ORCHESTRATOR"])
@ -153,40 +153,37 @@ class Orchestrator(object):
@abstractmethod
def get_prefixed_keys(self, prefix):
"""
:returns: a dict of key value pairs beginning with prefix
:rtype: {str: str}
"""
:returns: a dict of key value pairs beginning with prefix
:rtype: {str: str}
"""
pass
@abstractmethod
def get_key(self, key):
"""
:returns: the value stored at the provided key
:rtype: str
"""
:returns: the value stored at the provided key
:rtype: str
"""
pass
@abstractmethod
def set_key(self, key, value, overwrite=False, expiration=None):
"""
:param key: the identifier for the value
:type key: str
:param value: the value being stored
:type value: str
:param overwrite: whether or not a KeyError is thrown if the key already exists
:type overwrite: bool
:param expiration: the duration in seconds that a key should be available
:type expiration: int
"""
:param key: the identifier for the value
:type key: str
:param value: the value being stored
:type value: str
:param overwrite: whether or not a KeyError is thrown if the key already exists
:type overwrite: bool
:param expiration: the duration in seconds that a key should be available
:type expiration: int
"""
pass
@abstractmethod
def set_key_sync(self, key, value, overwrite=False, expiration=None):
"""
set_key, but without trollius coroutines.
set_key, but without asyncio coroutines.
"""
pass
@ -224,8 +221,8 @@ class Orchestrator(object):
def _sleep_orchestrator():
"""
This function blocks the trollius event loop by sleeping in order to backoff if a failure such
as a ConnectionError has occurred.
This function blocks the asyncio event loop by sleeping in order to backoff if a failure
such as a ConnectionError has occurred.
"""
logger.exception(
"Connecting to etcd failed; sleeping for %s and then trying again",
@ -262,7 +259,7 @@ class Etcd2Orchestrator(Orchestrator):
ca_cert=None,
client_threads=5,
canceller_only=False,
**kwargs
**kwargs,
):
self.is_canceller_only = canceller_only
@ -322,7 +319,7 @@ class Etcd2Orchestrator(Orchestrator):
logger.debug("Etcd moved forward too quickly. Restarting watch cycle.")
new_index = None
if restarter is not None:
async(restarter())
asyncio.create_task(restarter())
except (KeyError, etcd.EtcdKeyError):
logger.debug("Etcd key already cleared: %s", key)
@ -334,7 +331,7 @@ class Etcd2Orchestrator(Orchestrator):
except etcd.EtcdException as eex:
# TODO: This is a quick and dirty hack and should be replaced with a proper
# exception check.
if str(eex.message).find("Read timed out") >= 0:
if str(eex).find("Read timed out") >= 0:
logger.debug("Read-timeout on etcd watch %s, rescheduling", key)
else:
logger.exception("Exception on etcd watch: %s", key)
@ -346,7 +343,7 @@ class Etcd2Orchestrator(Orchestrator):
self._watch_etcd(key, callback, start_index=new_index, restarter=restarter)
if etcd_result and etcd_result.value is not None:
async(callback(self._etcd_result_to_keychange(etcd_result)))
asyncio.create_task(callback(self._etcd_result_to_keychange(etcd_result)))
if not self._shutting_down:
logger.debug("Scheduling watch of key: %s at start index %s", key, start_index)
@ -355,7 +352,7 @@ class Etcd2Orchestrator(Orchestrator):
)
watch_future.add_done_callback(callback_wrapper)
self._watch_tasks[key] = async(watch_future)
self._watch_tasks[key] = asyncio.create_task(watch_future)
@staticmethod
def _etcd_result_to_keychange(etcd_result):
@ -384,13 +381,12 @@ class Etcd2Orchestrator(Orchestrator):
logger.debug("creating watch on %s", key)
self._watch_etcd(key, callback, restarter=restarter)
@coroutine
def get_prefixed_keys(self, prefix):
async def get_prefixed_keys(self, prefix):
assert not self.is_canceller_only
try:
etcd_result = yield From(self._etcd_client.read(prefix, recursive=True))
raise Return({leaf.key: leaf.value for leaf in etcd_result.leaves})
etcd_result = await self._etcd_client.read(prefix, recursive=True)
return {leaf.key: leaf.value for leaf in etcd_result.leaves}
except etcd.EtcdKeyError:
raise KeyError
except etcd.EtcdConnectionFailed as ex:
@ -398,14 +394,13 @@ class Etcd2Orchestrator(Orchestrator):
except etcd.EtcdException as ex:
raise OrchestratorError(ex)
@coroutine
def get_key(self, key):
async def get_key(self, key):
assert not self.is_canceller_only
try:
# Ignore pylint: the value property on EtcdResult is added dynamically using setattr.
etcd_result = yield From(self._etcd_client.read(key))
raise Return(etcd_result.value)
etcd_result = await self._etcd_client.read(key)
return etcd_result.value
except etcd.EtcdKeyError:
raise KeyError
except etcd.EtcdConnectionFailed as ex:
@ -413,11 +408,10 @@ class Etcd2Orchestrator(Orchestrator):
except etcd.EtcdException as ex:
raise OrchestratorError(ex)
@coroutine
def set_key(self, key, value, overwrite=False, expiration=None):
async def set_key(self, key, value, overwrite=False, expiration=None):
assert not self.is_canceller_only
yield From(
await (
self._etcd_client.write(
key, value, prevExists=overwrite, ttl=self._sanity_check_ttl(expiration)
)
@ -428,12 +422,11 @@ class Etcd2Orchestrator(Orchestrator):
key, value, prevExists=overwrite, ttl=self._sanity_check_ttl(expiration)
)
@coroutine
def delete_key(self, key):
async def delete_key(self, key):
assert not self.is_canceller_only
try:
yield From(self._etcd_client.delete(key))
await self._etcd_client.delete(key)
except etcd.EtcdKeyError:
raise KeyError
except etcd.EtcdConnectionFailed as ex:
@ -441,22 +434,21 @@ class Etcd2Orchestrator(Orchestrator):
except etcd.EtcdException as ex:
raise OrchestratorError(ex)
@coroutine
def lock(self, key, expiration=DEFAULT_LOCK_EXPIRATION):
async def lock(self, key, expiration=DEFAULT_LOCK_EXPIRATION):
assert not self.is_canceller_only
try:
yield From(
await (
self._etcd_client.write(
key, {}, prevExist=False, ttl=self._sanity_check_ttl(expiration)
)
)
raise Return(True)
return True
except (KeyError, etcd.EtcdKeyError):
raise Return(False)
return False
except etcd.EtcdConnectionFailed:
logger.exception("Could not get etcd atomic lock as etcd is down")
raise Return(False)
return False
except etcd.EtcdException as ex:
raise OrchestratorError(ex)
@ -467,7 +459,7 @@ class Etcd2Orchestrator(Orchestrator):
if self.is_canceller_only:
return
for (key, _), task in self._watch_tasks.items():
for (key, _), task in list(self._watch_tasks.items()):
if not task.done():
logger.debug("Canceling watch task for %s", key)
task.cancel()
@ -487,16 +479,13 @@ class MemoryOrchestrator(Orchestrator):
def on_key_change(self, key, callback, restarter=None):
self.callbacks[key] = callback
@coroutine
def get_prefixed_keys(self, prefix):
raise Return({k: value for (k, value) in self.state.items() if k.startswith(prefix)})
async def get_prefixed_keys(self, prefix):
return {k: value for (k, value) in list(self.state.items()) if k.startswith(prefix)}
@coroutine
def get_key(self, key):
raise Return(self.state[key])
async def get_key(self, key):
return self.state[key]
@coroutine
def set_key(self, key, value, overwrite=False, expiration=None):
async def set_key(self, key, value, overwrite=False, expiration=None):
preexisting_key = "key" in self.state
if preexisting_key and not overwrite:
raise KeyError
@ -509,11 +498,11 @@ class MemoryOrchestrator(Orchestrator):
event = KeyEvent.CREATE if not preexisting_key else KeyEvent.SET
for callback in self._callbacks_prefixed(key):
yield From(callback(KeyChange(event, key, value)))
await callback(KeyChange(event, key, value))
def set_key_sync(self, key, value, overwrite=False, expiration=None):
"""
set_key, but without trollius coroutines.
set_key, but without asyncio coroutines.
"""
preexisting_key = "key" in self.state
if preexisting_key and not overwrite:
@ -529,20 +518,18 @@ class MemoryOrchestrator(Orchestrator):
for callback in self._callbacks_prefixed(key):
callback(KeyChange(event, key, value))
@coroutine
def delete_key(self, key):
async def delete_key(self, key):
value = self.state[key]
del self.state[key]
for callback in self._callbacks_prefixed(key):
yield From(callback(KeyChange(KeyEvent.DELETE, key, value)))
await callback(KeyChange(KeyEvent.DELETE, key, value))
@coroutine
def lock(self, key, expiration=DEFAULT_LOCK_EXPIRATION):
async def lock(self, key, expiration=DEFAULT_LOCK_EXPIRATION):
if key in self.state:
raise Return(False)
return False
self.state.set(key, None, expires=expiration)
raise Return(True)
return True
def shutdown(self):
self.state = None
@ -562,7 +549,7 @@ class RedisOrchestrator(Orchestrator):
ssl=False,
skip_keyspace_event_setup=False,
canceller_only=False,
**kwargs
**kwargs,
):
self.is_canceller_only = canceller_only
(cert, key) = tuple(cert_and_key) if cert_and_key is not None else (None, None)
@ -632,16 +619,16 @@ class RedisOrchestrator(Orchestrator):
keychange = self._publish_to_keychange(event_value)
for watched_key, callback in iteritems(self._watched_keys):
if keychange.key.startswith(watched_key):
async(callback(keychange))
asyncio.create_task(callback(keychange))
if not self._shutting_down:
logger.debug("Scheduling watch of publish stream")
watch_future = self._pubsub.parse_response()
watch_future.add_done_callback(published_callback_wrapper)
self._tasks["pub"] = async(watch_future)
self._tasks["pub"] = asyncio.create_task(watch_future)
def _watch_expiring_key(self):
def expiring_callback_wrapper(event_future):
async def expiring_callback_wrapper(event_future):
logger.debug("expiring callback called")
event_result = None
@ -651,11 +638,11 @@ class RedisOrchestrator(Orchestrator):
if self._is_expired_keyspace_event(event_result):
# Get the value of the original key before the expiration happened.
key = self._key_from_expiration(event_future)
expired_value = yield From(self._client.get(key))
expired_value = await self._client.get(key)
# $KEY/expiring is gone, but the original key still remains, set an expiration for it
# so that other managers have time to get the event and still read the expired value.
yield From(self._client.expire(key, ONE_DAY))
await self._client.expire(key, ONE_DAY)
except redis.ConnectionError:
_sleep_orchestrator()
except redis.RedisError:
@ -668,13 +655,15 @@ class RedisOrchestrator(Orchestrator):
if self._is_expired_keyspace_event(event_result) and expired_value is not None:
for watched_key, callback in iteritems(self._watched_keys):
if key.startswith(watched_key):
async(callback(KeyChange(KeyEvent.EXPIRE, key, expired_value)))
asyncio.create_task(
callback(KeyChange(KeyEvent.EXPIRE, key, expired_value))
)
if not self._shutting_down:
logger.debug("Scheduling watch of expiration")
watch_future = self._pubsub_expiring.parse_response()
watch_future.add_done_callback(expiring_callback_wrapper)
self._tasks["expire"] = async(watch_future)
self._tasks["expire"] = asyncio.create_task(watch_future)
def on_key_change(self, key, callback, restarter=None):
assert not self.is_canceller_only
@ -709,49 +698,46 @@ class RedisOrchestrator(Orchestrator):
e = json.loads(event_value)
return KeyChange(KeyEvent(e["event"]), e["key"], e["value"])
@coroutine
def get_prefixed_keys(self, prefix):
async def get_prefixed_keys(self, prefix):
assert not self.is_canceller_only
# TODO: This can probably be done with redis pipelines to make it transactional.
keys = yield From(self._client.keys(prefix + "*"))
keys = await self._client.keys(prefix + "*")
# Yielding to the event loop is required, thus this cannot be written as a dict comprehension.
results = {}
for key in keys:
if key.endswith(REDIS_EXPIRING_SUFFIX):
continue
ttl = yield From(self._client.ttl(key))
ttl = await self._client.ttl(key)
if ttl != REDIS_NONEXPIRING_KEY:
# Only redis keys without expirations are live build manager keys.
value = yield From(self._client.get(key))
value = await self._client.get(key)
results.update({key: value})
raise Return(results)
return results
@coroutine
def get_key(self, key):
async def get_key(self, key):
assert not self.is_canceller_only
value = yield From(self._client.get(key))
raise Return(value)
value = await self._client.get(key)
return value
@coroutine
def set_key(self, key, value, overwrite=False, expiration=None):
async def set_key(self, key, value, overwrite=False, expiration=None):
assert not self.is_canceller_only
already_exists = yield From(self._client.exists(key))
already_exists = await self._client.exists(key)
yield From(self._client.set(key, value, xx=overwrite))
await self._client.set(key, value, xx=overwrite)
if expiration is not None:
yield From(
await (
self._client.set(
slash_join(key, REDIS_EXPIRING_SUFFIX), value, xx=overwrite, ex=expiration
)
)
key_event = KeyEvent.SET if already_exists else KeyEvent.CREATE
yield From(self._publish(event=key_event, key=key, value=value))
await self._publish(event=key_event, key=key, value=value)
def set_key_sync(self, key, value, overwrite=False, expiration=None):
already_exists = self._sync_client.exists(key)
@ -773,31 +759,27 @@ class RedisOrchestrator(Orchestrator):
),
)
@coroutine
def _publish(self, **kwargs):
async def _publish(self, **kwargs):
kwargs["event"] = int(kwargs["event"])
event_json = json.dumps(kwargs)
logger.debug("publishing event: %s", event_json)
yield From(self._client.publish(self._pubsub_key, event_json))
await self._client.publish(self._pubsub_key, event_json)
@coroutine
def delete_key(self, key):
async def delete_key(self, key):
assert not self.is_canceller_only
value = yield From(self._client.get(key))
yield From(self._client.delete(key))
yield From(self._client.delete(slash_join(key, REDIS_EXPIRING_SUFFIX)))
yield From(self._publish(event=KeyEvent.DELETE, key=key, value=value))
value = await self._client.get(key)
await self._client.delete(key)
await self._client.delete(slash_join(key, REDIS_EXPIRING_SUFFIX))
await self._publish(event=KeyEvent.DELETE, key=key, value=value)
@coroutine
def lock(self, key, expiration=DEFAULT_LOCK_EXPIRATION):
async def lock(self, key, expiration=DEFAULT_LOCK_EXPIRATION):
assert not self.is_canceller_only
yield From(self.set_key(key, "", ex=expiration))
raise Return(True)
await self.set_key(key, "", ex=expiration)
return True
@coroutine
def shutdown(self):
async def shutdown(self):
logger.debug("Shutting down redis client.")
self._shutting_down = True

35
buildman/server.py
View File

@ -4,14 +4,13 @@ import json
from datetime import timedelta
from threading import Event
import trollius
import asyncio
from aiowsgi import create_server as create_wsgi_server
from autobahn.asyncio.wamp import RouterFactory, RouterSessionFactory
from autobahn.asyncio.websocket import WampWebSocketServerFactory
from autobahn.wamp import types
from flask import Flask
from trollius.coroutines import From
from app import app
from buildman.enums import BuildJobResult, BuildServerStatus, RESULT_PHASES
@ -108,7 +107,7 @@ class BuilderServer(object):
self._lifecycle_manager.initialize(self._lifecycle_manager_config)
logger.debug("Initializing all members of the event loop")
loop = trollius.get_event_loop()
loop = asyncio.get_event_loop()
logger.debug(
"Starting server on port %s, with controller on port %s",
@ -175,8 +174,7 @@ class BuilderServer(object):
minimum_extension=MINIMUM_JOB_EXTENSION,
)
@trollius.coroutine
def _job_complete(self, build_job, job_status, executor_name=None, update_phase=False):
async def _job_complete(self, build_job, job_status, executor_name=None, update_phase=False):
if job_status == BuildJobResult.INCOMPLETE:
logger.warning(
"[BUILD INCOMPLETE: job complete] Build ID: %s. No retry restore.",
@ -194,19 +192,18 @@ class BuilderServer(object):
if update_phase:
status_handler = StatusHandler(self._build_logs, build_job.repo_build.uuid)
yield From(status_handler.set_phase(RESULT_PHASES[job_status]))
await status_handler.set_phase(RESULT_PHASES[job_status])
self._job_count = self._job_count - 1
if self._current_status == BuildServerStatus.SHUTDOWN and not self._job_count:
self._shutdown_event.set()
@trollius.coroutine
def _work_checker(self):
async def _work_checker(self):
logger.debug("Initializing work checker")
while self._current_status == BuildServerStatus.RUNNING:
with database.CloseForLongOperation(app.config):
yield From(trollius.sleep(WORK_CHECK_TIMEOUT))
await asyncio.sleep(WORK_CHECK_TIMEOUT)
logger.debug(
"Checking for more work for %d active workers",
@ -237,9 +234,7 @@ class BuilderServer(object):
)
try:
schedule_success, retry_timeout = yield From(
self._lifecycle_manager.schedule(build_job)
)
schedule_success, retry_timeout = await self._lifecycle_manager.schedule(build_job)
except:
logger.warning(
"[BUILD INCOMPLETE: scheduling] Build ID: %s. Retry restored.",
@ -253,7 +248,7 @@ class BuilderServer(object):
if schedule_success:
logger.debug("Marking build %s as scheduled", build_job.repo_build.uuid)
status_handler = StatusHandler(self._build_logs, build_job.repo_build.uuid)
yield From(status_handler.set_phase(database.BUILD_PHASE.BUILD_SCHEDULED))
await status_handler.set_phase(database.BUILD_PHASE.BUILD_SCHEDULED)
self._job_count = self._job_count + 1
logger.debug(
@ -273,18 +268,16 @@ class BuilderServer(object):
)
self._queue.incomplete(job_item, restore_retry=True, retry_after=retry_timeout)
@trollius.coroutine
def _queue_metrics_updater(self):
async def _queue_metrics_updater(self):
logger.debug("Initializing queue metrics updater")
while self._current_status == BuildServerStatus.RUNNING:
logger.debug("Writing metrics")
self._queue.update_metrics()
logger.debug("Metrics going to sleep for 30 seconds")
yield From(trollius.sleep(30))
await asyncio.sleep(30)
@trollius.coroutine
def _initialize(self, loop, host, websocket_port, controller_port, ssl=None):
async def _initialize(self, loop, host, websocket_port, controller_port, ssl=None):
self._loop = loop
# Create the WAMP server.
@ -295,10 +288,10 @@ class BuilderServer(object):
create_wsgi_server(
self._controller_app, loop=loop, host=host, port=controller_port, ssl=ssl
)
yield From(loop.create_server(transport_factory, host, websocket_port, ssl=ssl))
await loop.create_server(transport_factory, host, websocket_port, ssl=ssl)
# Initialize the metrics updater
trollius.async(self._queue_metrics_updater())
asyncio.create_task(self._queue_metrics_updater())
# Initialize the work queue checker.
yield From(self._work_checker())
await self._work_checker()

104
buildman/test/test_buildman.py
View File

@ -1,10 +1,10 @@
import asyncio
import unittest
import json
import uuid
from mock import Mock, ANY
from six import iteritems
from trollius import coroutine, get_event_loop, From, Future, Return
from buildman.asyncutil import AsyncWrapper
from buildman.component.buildcomponent import BuildComponent
@ -21,9 +21,9 @@ REALM_ID = "1234-realm"
def async_test(f):
def wrapper(*args, **kwargs):
coro = coroutine(f)
coro = asyncio.coroutine(f)
future = coro(*args, **kwargs)
loop = get_event_loop()
loop = asyncio.get_event_loop()
loop.run_until_complete(future)
return wrapper
@ -33,19 +33,16 @@ class TestExecutor(BuilderExecutor):
job_started = None
job_stopped = None
@coroutine
def start_builder(self, realm, token, build_uuid):
async def start_builder(self, realm, token, build_uuid):
self.job_started = str(uuid.uuid4())
raise Return(self.job_started)
return self.job_started
@coroutine
def stop_builder(self, execution_id):
async def stop_builder(self, execution_id):
self.job_stopped = execution_id
class BadExecutor(BuilderExecutor):
@coroutine
def start_builder(self, realm, token, build_uuid):
async def start_builder(self, realm, token, build_uuid):
raise ExecutorException("raised on purpose!")
@ -57,7 +54,7 @@ class EphemeralBuilderTestCase(unittest.TestCase):
@staticmethod
def _create_completed_future(result=None):
def inner(*args, **kwargs):
new_future = Future()
new_future = asyncio.Future()
new_future.set_result(result)
return new_future
@ -69,9 +66,8 @@ class EphemeralBuilderTestCase(unittest.TestCase):
def tearDown(self):
EphemeralBuilderManager.EXECUTORS = self._existing_executors
@coroutine
def _register_component(self, realm_spec, build_component, token):
raise Return("hello")
async def _register_component(self, realm_spec, build_component, token):
return "hello"
def _create_build_job(self, namespace="namespace", retries=3):
mock_job = Mock()
@ -99,7 +95,7 @@ class TestEphemeralLifecycle(EphemeralBuilderTestCase):
def _create_completed_future(self, result=None):
def inner(*args, **kwargs):
new_future = Future()
new_future = asyncio.Future()
new_future.set_result(result)
return new_future
@ -149,14 +145,13 @@ class TestEphemeralLifecycle(EphemeralBuilderTestCase):
super(TestEphemeralLifecycle, self).tearDown()
self.manager.shutdown()
@coroutine
def _setup_job_for_managers(self):
async def _setup_job_for_managers(self):
test_component = Mock(spec=BuildComponent)
test_component.builder_realm = REALM_ID
test_component.start_build = Mock(side_effect=self._create_completed_future())
self.register_component_callback.return_value = test_component
is_scheduled = yield From(self.manager.schedule(self.mock_job))
is_scheduled = await self.manager.schedule(self.mock_job)
self.assertTrue(is_scheduled)
self.assertEqual(self.test_executor.start_builder.call_count, 1)
@ -168,7 +163,7 @@ class TestEphemeralLifecycle(EphemeralBuilderTestCase):
realm_for_build = self._find_realm_key(self.manager._orchestrator, BUILD_UUID)
raw_realm_data = yield From(
raw_realm_data = await (
self.manager._orchestrator.get_key(slash_join("realm", realm_for_build))
)
realm_data = json.loads(raw_realm_data)
@ -178,7 +173,7 @@ class TestEphemeralLifecycle(EphemeralBuilderTestCase):
self.assertEqual(self.register_component_callback.call_count, 0)
# Fire off a realm changed with the same data.
yield From(
await (
self.manager._realm_callback(
KeyChange(
KeyEvent.CREATE, slash_join(REALM_PREFIX, REALM_ID), json.dumps(realm_data)
@ -193,7 +188,7 @@ class TestEphemeralLifecycle(EphemeralBuilderTestCase):
# Ensure that the build info exists.
self.assertIsNotNone(self.manager._build_uuid_to_info.get(BUILD_UUID))
raise Return(test_component)
return test_component
@staticmethod
def _find_realm_key(orchestrator, build_uuid):
@ -209,15 +204,15 @@ class TestEphemeralLifecycle(EphemeralBuilderTestCase):
@async_test
def test_schedule_and_complete(self):
# Test that a job is properly registered with all of the managers
test_component = yield From(self._setup_job_for_managers())
test_component = await self._setup_job_for_managers()
# Take the job ourselves
yield From(self.manager.build_component_ready(test_component))
await self.manager.build_component_ready(test_component)
self.assertIsNotNone(self.manager._build_uuid_to_info.get(BUILD_UUID))
# Finish the job
yield From(
await (
self.manager.job_completed(self.mock_job, BuildJobResult.COMPLETE, test_component)
)
@ -231,9 +226,9 @@ class TestEphemeralLifecycle(EphemeralBuilderTestCase):
@async_test
def test_another_manager_takes_job(self):
# Prepare a job to be taken by another manager
test_component = yield From(self._setup_job_for_managers())
test_component = await self._setup_job_for_managers()
yield From(
await (
self.manager._realm_callback(
KeyChange(
KeyEvent.DELETE,
@ -260,7 +255,7 @@ class TestEphemeralLifecycle(EphemeralBuilderTestCase):
self.assertIsNotNone(self.manager._build_uuid_to_info.get(BUILD_UUID))
# Delete the job once it has "completed".
yield From(
await (
self.manager._job_callback(
KeyChange(
KeyEvent.DELETE,
@ -281,7 +276,7 @@ class TestEphemeralLifecycle(EphemeralBuilderTestCase):
self.assertIn(JOB_PREFIX, callback_keys)
# Send a signal to the callback that the job has been created.
yield From(
await (
self.manager._job_callback(
KeyChange(
KeyEvent.CREATE,
@ -301,7 +296,7 @@ class TestEphemeralLifecycle(EphemeralBuilderTestCase):
self.assertIn(JOB_PREFIX, callback_keys)
# Send a signal to the callback that a worker has expired
yield From(
await (
self.manager._job_callback(
KeyChange(
KeyEvent.EXPIRE,
@ -316,13 +311,13 @@ class TestEphemeralLifecycle(EphemeralBuilderTestCase):
@async_test
def test_expiring_worker_started(self):
test_component = yield From(self._setup_job_for_managers())
test_component = await self._setup_job_for_managers()
# Ensure that that the building callbacks have been registered
callback_keys = [key for key in self.manager._orchestrator.callbacks]
self.assertIn(JOB_PREFIX, callback_keys)
yield From(
await (
self.manager._job_callback(
KeyChange(
KeyEvent.EXPIRE,
@ -337,14 +332,14 @@ class TestEphemeralLifecycle(EphemeralBuilderTestCase):
@async_test
def test_buildjob_deleted(self):
test_component = yield From(self._setup_job_for_managers())
test_component = await self._setup_job_for_managers()
# Ensure that that the building callbacks have been registered
callback_keys = [key for key in self.manager._orchestrator.callbacks]
self.assertIn(JOB_PREFIX, callback_keys)
# Send a signal to the callback that a worker has expired
yield From(
await (
self.manager._job_callback(
KeyChange(
KeyEvent.DELETE,
@ -360,14 +355,14 @@ class TestEphemeralLifecycle(EphemeralBuilderTestCase):
@async_test
def test_builder_never_starts(self):
test_component = yield From(self._setup_job_for_managers())
test_component = await self._setup_job_for_managers()
# Ensure that that the building callbacks have been registered
callback_keys = [key for key in self.manager._orchestrator.callbacks]
self.assertIn(JOB_PREFIX, callback_keys)
# Send a signal to the callback that a worker has expired
yield From(
await (
self.manager._job_callback(
KeyChange(
KeyEvent.EXPIRE,
@ -382,7 +377,7 @@ class TestEphemeralLifecycle(EphemeralBuilderTestCase):
# Ensure the job was marked as incomplete, with an update_phase to True (so the DB record and
# logs are updated as well)
yield From(
await (
self.job_complete_callback.assert_called_once_with(
ANY, BuildJobResult.INCOMPLETE, "MockExecutor", update_phase=True
)
@ -396,10 +391,10 @@ class TestEphemeralLifecycle(EphemeralBuilderTestCase):
@async_test
def test_realm_expired(self):
test_component = yield From(self._setup_job_for_managers())
test_component = await self._setup_job_for_managers()
# Send a signal to the callback that a realm has expired
yield From(
await (
self.manager._realm_callback(
KeyChange(
KeyEvent.EXPIRE,
@ -433,9 +428,8 @@ class TestEphemeral(EphemeralBuilderTestCase):
unregister_component_callback = Mock()
job_heartbeat_callback = Mock()
@coroutine
def job_complete_callback(*args, **kwargs):
raise Return()
async def job_complete_callback(*args, **kwargs):
return
self.manager = EphemeralBuilderManager(
self._register_component,
@ -542,12 +536,12 @@ class TestEphemeral(EphemeralBuilderTestCase):
# Try with a build job in an invalid namespace.
build_job = self._create_build_job(namespace="somethingelse")
result = yield From(self.manager.schedule(build_job))
result = await self.manager.schedule(build_job)
self.assertFalse(result[0])
# Try with a valid namespace.
build_job = self._create_build_job(namespace="something")
result = yield From(self.manager.schedule(build_job))
result = await self.manager.schedule(build_job)
self.assertTrue(result[0])
@async_test
@ -562,12 +556,12 @@ class TestEphemeral(EphemeralBuilderTestCase):
# Try with a build job that has too few retries.
build_job = self._create_build_job(retries=1)
result = yield From(self.manager.schedule(build_job))
result = await self.manager.schedule(build_job)
self.assertFalse(result[0])
# Try with a valid job.
build_job = self._create_build_job(retries=2)
result = yield From(self.manager.schedule(build_job))
result = await self.manager.schedule(build_job)
self.assertTrue(result[0])
@async_test
@ -593,7 +587,7 @@ class TestEphemeral(EphemeralBuilderTestCase):
# Try a job not matching the primary's namespace filter. Should schedule on secondary.
build_job = self._create_build_job(namespace="somethingelse")
result = yield From(self.manager.schedule(build_job))
result = await self.manager.schedule(build_job)
self.assertTrue(result[0])
self.assertIsNone(self.manager.registered_executors[0].job_started)
@ -604,7 +598,7 @@ class TestEphemeral(EphemeralBuilderTestCase):
# Try a job not matching the primary's retry minimum. Should schedule on secondary.
build_job = self._create_build_job(namespace="something", retries=2)
result = yield From(self.manager.schedule(build_job))
result = await self.manager.schedule(build_job)
self.assertTrue(result[0])
self.assertIsNone(self.manager.registered_executors[0].job_started)
@ -615,7 +609,7 @@ class TestEphemeral(EphemeralBuilderTestCase):
# Try a job matching the primary. Should schedule on the primary.
build_job = self._create_build_job(namespace="something", retries=3)
result = yield From(self.manager.schedule(build_job))
result = await self.manager.schedule(build_job)
self.assertTrue(result[0])
self.assertIsNotNone(self.manager.registered_executors[0].job_started)
@ -626,7 +620,7 @@ class TestEphemeral(EphemeralBuilderTestCase):
# Try a job not matching either's restrictions.
build_job = self._create_build_job(namespace="somethingelse", retries=1)
result = yield From(self.manager.schedule(build_job))
result = await self.manager.schedule(build_job)
self.assertFalse(result[0])
self.assertIsNone(self.manager.registered_executors[0].job_started)
@ -649,14 +643,14 @@ class TestEphemeral(EphemeralBuilderTestCase):
)
build_job = self._create_build_job(namespace="something", retries=3)
result = yield From(self.manager.schedule(build_job))
result = await self.manager.schedule(build_job)
self.assertTrue(result[0])
self.assertIsNotNone(self.manager.registered_executors[0].job_started)
self.manager.registered_executors[0].job_started = None
build_job = self._create_build_job(namespace="something", retries=0)
result = yield From(self.manager.schedule(build_job))
result = await self.manager.schedule(build_job)
self.assertTrue(result[0])
self.assertIsNotNone(self.manager.registered_executors[0].job_started)
@ -671,7 +665,7 @@ class TestEphemeral(EphemeralBuilderTestCase):
)
build_job = self._create_build_job(namespace="something", retries=3)
result = yield From(self.manager.schedule(build_job))
result = await self.manager.schedule(build_job)
self.assertFalse(result[0])
@async_test
@ -684,14 +678,14 @@ class TestEphemeral(EphemeralBuilderTestCase):
# Start the build job.
build_job = self._create_build_job(namespace="something", retries=3)
result = yield From(self.manager.schedule(build_job))
result = await self.manager.schedule(build_job)
self.assertTrue(result[0])
executor = self.manager.registered_executors[0]
self.assertIsNotNone(executor.job_started)
# Register the realm so the build information is added.
yield From(
await (
self.manager._register_realm(
{
"realm": str(uuid.uuid4()),
@ -705,7 +699,7 @@ class TestEphemeral(EphemeralBuilderTestCase):
)
# Stop the build job.
yield From(self.manager.kill_builder_executor(build_job.build_uuid))
await self.manager.kill_builder_executor(build_job.build_uuid)
self.assertEqual(executor.job_stopped, executor.job_started)

2
buildtrigger/basehandler.py
View File

@ -277,7 +277,7 @@ class BuildTriggerHandler(object):
"""
Returns whether the file is named Dockerfile or follows the convention <name>.Dockerfile.
"""
return file_name.endswith(".Dockerfile") or u"Dockerfile" == file_name
return file_name.endswith(".Dockerfile") or "Dockerfile" == file_name
@classmethod
def service_name(cls):

2
buildtrigger/customhandler.py
View File

@ -147,7 +147,7 @@ class CustomBuildTrigger(BuildTriggerHandler):
return "custom-git"
def is_active(self):
return self.config.has_key("credentials")
return "credentials" in self.config
def _metadata_from_payload(self, payload, git_url):
# Parse the JSON payload.

3
buildtrigger/githubhandler.py
View File

@ -352,8 +352,7 @@ class GithubBuildTrigger(BuildTriggerHandler):
elem.path
for elem in commit_tree.tree
if (
elem.type == u"blob"
and self.filename_is_dockerfile(os.path.basename(elem.path))
elem.type == "blob" and self.filename_is_dockerfile(os.path.basename(elem.path))
)
]
except GithubException as ghe:

5
buildtrigger/gitlabhandler.py
View File

@ -231,7 +231,10 @@ class GitLabBuildTrigger(BuildTriggerHandler):
]
key = gl_project.keys.create(
{"title": "%s Builder" % app.config["REGISTRY_TITLE"], "key": public_key,}
{
"title": "%s Builder" % app.config["REGISTRY_TITLE"],
"key": public_key.decode("ascii"),
}
)
if not key:

4
buildtrigger/test/gitlabmock.py
View File

@ -342,7 +342,7 @@ def dockerfile_handler(_, request):
"file_path": "Dockerfile",
"size": 10,
"encoding": "base64",
"content": base64.b64encode("hello world"),
"content": base64.b64encode(b"hello world").decode("ascii"),
"ref": "master",
"blob_id": "79f7bbd25901e8334750839545a9bd021f0e4c83",
"commit_id": "d5a3ff139356ce33e37e73add446f16869741b50",
@ -368,7 +368,7 @@ def sub_dockerfile_handler(_, request):
"file_path": "somesubdir/Dockerfile",
"size": 10,
"encoding": "base64",
"content": base64.b64encode("hi universe"),
"content": base64.b64encode(b"hi universe").decode("ascii"),
"ref": "master",
"blob_id": "79f7bbd25901e8334750839545a9bd021f0e4c83",
"commit_id": "d5a3ff139356ce33e37e73add446f16869741b50",

6
buildtrigger/test/test_basehandler.py
View File

@ -8,10 +8,10 @@ from buildtrigger.basehandler import BuildTriggerHandler
[
("Dockerfile", True),
("server.Dockerfile", True),
(u"Dockerfile", True),
(u"server.Dockerfile", True),
("Dockerfile", True),
("server.Dockerfile", True),
("bad file name", False),
("bad file name", False),
(u"bad file name", False),
],
)
def test_path_is_dockerfile(input, output):

3
buildtrigger/test/test_customhandler.py
View File

@ -18,7 +18,7 @@ from util.morecollections import AttrDict
(
'{"commit": "foo", "ref": "refs/heads/something", "default_branch": "baz"}',
InvalidPayloadException,
"u'foo' does not match '^([A-Fa-f0-9]{7,})$'",
"'foo' does not match '^([A-Fa-f0-9]{7,})$'",
),
(
'{"commit": "11d6fbc", "ref": "refs/heads/something", "default_branch": "baz"}',
@ -48,6 +48,7 @@ def test_handle_trigger_request(payload, expected_error, expected_message):
if expected_error is not None:
with pytest.raises(expected_error) as ipe:
trigger.handle_trigger_request(request)
assert str(ipe.value) == expected_message
else:
assert isinstance(trigger.handle_trigger_request(request), PreparedBuild)

4
buildtrigger/test/test_githosthandler.py
View File

@ -93,9 +93,9 @@ def test_list_build_source_namespaces():
]
found = get_bitbucket_trigger().list_build_source_namespaces()
found.sort()
found = sorted(found, key=lambda d: sorted(d.items()))
namespaces_expected.sort()
namespaces_expected = sorted(namespaces_expected, key=lambda d: sorted(d.items()))
assert found == namespaces_expected

4
buildtrigger/test/test_githubhandler.py
View File

@ -129,7 +129,7 @@ def test_list_build_source_namespaces(github_trigger):
]
found = github_trigger.list_build_source_namespaces()
found.sort()
sorted(found, key=lambda d: sorted(d.items()))
namespaces_expected.sort()
sorted(namespaces_expected, key=lambda d: sorted(d.items()))
assert found == namespaces_expected

38
buildtrigger/test/test_gitlabhandler.py
View File

@ -27,8 +27,8 @@ def test_list_build_subdirs(gitlab_trigger):
@pytest.mark.parametrize(
"dockerfile_path, contents",
[
("/Dockerfile", "hello world"),
("somesubdir/Dockerfile", "hi universe"),
("/Dockerfile", b"hello world"),
("somesubdir/Dockerfile", b"hi universe"),
("unknownpath", None),
],
)
@ -68,19 +68,19 @@ def test_list_build_sources():
assert sources == [
{
"last_updated": 1380548762,
"name": u"someproject",
"url": u"http://example.com/someorg/someproject",
"name": "someproject",
"url": "http://example.com/someorg/someproject",
"private": True,
"full_name": u"someorg/someproject",
"full_name": "someorg/someproject",
"has_admin_permissions": False,
"description": "",
},
{
"last_updated": 1380548762,
"name": u"anotherproject",
"url": u"http://example.com/someorg/anotherproject",
"name": "anotherproject",
"url": "http://example.com/someorg/anotherproject",
"private": False,
"full_name": u"someorg/anotherproject",
"full_name": "someorg/anotherproject",
"has_admin_permissions": True,
"description": "",
},
@ -93,8 +93,8 @@ def test_null_avatar():
expected = {
"avatar_url": None,
"personal": False,
"title": u"someorg",
"url": u"http://gitlab.com/groups/someorg",
"title": "someorg",
"url": "http://gitlab.com/groups/someorg",
"score": 1,
"id": "2",
}
@ -239,10 +239,10 @@ def test_list_field_values(name, expected, gitlab_trigger):
[
{
"last_updated": 1380548762,
"name": u"anotherproject",
"url": u"http://example.com/knownuser/anotherproject",
"name": "anotherproject",
"url": "http://example.com/knownuser/anotherproject",
"private": False,
"full_name": u"knownuser/anotherproject",
"full_name": "knownuser/anotherproject",
"has_admin_permissions": True,
"description": "",
},
@ -253,19 +253,19 @@ def test_list_field_values(name, expected, gitlab_trigger):
[
{
"last_updated": 1380548762,
"name": u"someproject",
"url": u"http://example.com/someorg/someproject",
"name": "someproject",
"url": "http://example.com/someorg/someproject",
"private": True,
"full_name": u"someorg/someproject",
"full_name": "someorg/someproject",
"has_admin_permissions": False,
"description": "",
},
{
"last_updated": 1380548762,
"name": u"anotherproject",
"url": u"http://example.com/someorg/anotherproject",
"name": "anotherproject",
"url": "http://example.com/someorg/anotherproject",
"private": False,
"full_name": u"someorg/anotherproject",
"full_name": "someorg/anotherproject",
"has_admin_permissions": True,
"description": "",
},

292
buildtrigger/test/test_prepare_trigger.py
View File

@ -38,25 +38,25 @@ def assertSchema(filename, expected, processor, *args, **kwargs):
def test_custom_custom():
expected = {
u"commit": u"1c002dd",
u"commit_info": {
u"url": u"gitsoftware.com/repository/commits/1234567",
u"date": u"timestamp",
u"message": u"initial commit",
u"committer": {
u"username": u"user",
u"url": u"gitsoftware.com/users/user",
u"avatar_url": u"gravatar.com/user.png",
"commit": "1c002dd",
"commit_info": {
"url": "gitsoftware.com/repository/commits/1234567",
"date": "timestamp",
"message": "initial commit",
"committer": {
"username": "user",
"url": "gitsoftware.com/users/user",
"avatar_url": "gravatar.com/user.png",
},
u"author": {
u"username": u"user",
u"url": u"gitsoftware.com/users/user",
u"avatar_url": u"gravatar.com/user.png",
"author": {
"username": "user",
"url": "gitsoftware.com/users/user",
"avatar_url": "gravatar.com/user.png",
},
},
u"ref": u"refs/heads/master",
u"default_branch": u"master",
u"git_url": u"foobar",
"ref": "refs/heads/master",
"default_branch": "master",
"git_url": "foobar",
}
assertSchema("custom_webhook", expected, custom_trigger_payload, git_url="foobar")
@ -64,13 +64,13 @@ def test_custom_custom():
def test_custom_gitlab():
expected = {
"commit": u"fb88379ee45de28a0a4590fddcbd8eff8b36026e",
"ref": u"refs/heads/master",
"git_url": u"git@gitlab.com:jsmith/somerepo.git",
"commit": "fb88379ee45de28a0a4590fddcbd8eff8b36026e",
"ref": "refs/heads/master",
"git_url": "git@gitlab.com:jsmith/somerepo.git",
"commit_info": {
"url": u"https://gitlab.com/jsmith/somerepo/commit/fb88379ee45de28a0a4590fddcbd8eff8b36026e",
"date": u"2015-08-13T19:33:18+00:00",
"message": u"Fix link\n",
"url": "https://gitlab.com/jsmith/somerepo/commit/fb88379ee45de28a0a4590fddcbd8eff8b36026e",
"date": "2015-08-13T19:33:18+00:00",
"message": "Fix link\n",
},
}
@ -84,16 +84,16 @@ def test_custom_gitlab():
def test_custom_github():
expected = {
"commit": u"410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"ref": u"refs/heads/master",
"default_branch": u"master",
"git_url": u"git@github.com:jsmith/anothertest.git",
"commit": "410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"ref": "refs/heads/master",
"default_branch": "master",
"git_url": "git@github.com:jsmith/anothertest.git",
"commit_info": {
"url": u"https://github.com/jsmith/anothertest/commit/410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"date": u"2015-09-11T14:26:16-04:00",
"message": u"Update Dockerfile",
"committer": {"username": u"jsmith",},
"author": {"username": u"jsmith",},
"url": "https://github.com/jsmith/anothertest/commit/410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"date": "2015-09-11T14:26:16-04:00",
"message": "Update Dockerfile",
"committer": {"username": "jsmith",},
"author": {"username": "jsmith",},
},
}
@ -107,20 +107,20 @@ def test_custom_github():
def test_custom_bitbucket():
expected = {
"commit": u"af64ae7188685f8424040b4735ad12941b980d75",
"ref": u"refs/heads/master",
"git_url": u"git@bitbucket.org:jsmith/another-repo.git",
"commit": "af64ae7188685f8424040b4735ad12941b980d75",
"ref": "refs/heads/master",
"git_url": "git@bitbucket.org:jsmith/another-repo.git",
"commit_info": {
"url": u"https://bitbucket.org/jsmith/another-repo/commits/af64ae7188685f8424040b4735ad12941b980d75",
"date": u"2015-09-10T20:40:54+00:00",
"message": u"Dockerfile edited online with Bitbucket",
"url": "https://bitbucket.org/jsmith/another-repo/commits/af64ae7188685f8424040b4735ad12941b980d75",
"date": "2015-09-10T20:40:54+00:00",
"message": "Dockerfile edited online with Bitbucket",
"author": {
"username": u"John Smith",
"avatar_url": u"https://bitbucket.org/account/jsmith/avatar/32/",
"username": "John Smith",
"avatar_url": "https://bitbucket.org/account/jsmith/avatar/32/",
},
"committer": {
"username": u"John Smith",
"avatar_url": u"https://bitbucket.org/account/jsmith/avatar/32/",
"username": "John Smith",
"avatar_url": "https://bitbucket.org/account/jsmith/avatar/32/",
},
},
}
@ -180,15 +180,15 @@ def test_bitbucket_commit():
return {"user": {"display_name": "cooluser", "avatar": "http://some/avatar/url"}}
expected = {
"commit": u"abdeaf1b2b4a6b9ddf742c1e1754236380435a62",
"ref": u"refs/heads/somebranch",
"git_url": u"git@bitbucket.org:foo/bar.git",
"default_branch": u"somebranch",
"commit": "abdeaf1b2b4a6b9ddf742c1e1754236380435a62",
"ref": "refs/heads/somebranch",
"git_url": "git@bitbucket.org:foo/bar.git",
"default_branch": "somebranch",
"commit_info": {
"url": u"https://bitbucket.org/foo/bar/commits/abdeaf1b2b4a6b9ddf742c1e1754236380435a62",
"date": u"2012-07-24 00:26:36",
"message": u"making some changes\n",
"author": {"avatar_url": u"http://some/avatar/url", "username": u"cooluser",},
"url": "https://bitbucket.org/foo/bar/commits/abdeaf1b2b4a6b9ddf742c1e1754236380435a62",
"date": "2012-07-24 00:26:36",
"message": "making some changes\n",
"author": {"avatar_url": "http://some/avatar/url", "username": "cooluser",},
},
}
@ -199,20 +199,20 @@ def test_bitbucket_commit():
def test_bitbucket_webhook_payload():
expected = {
"commit": u"af64ae7188685f8424040b4735ad12941b980d75",
"ref": u"refs/heads/master",
"git_url": u"git@bitbucket.org:jsmith/another-repo.git",
"commit": "af64ae7188685f8424040b4735ad12941b980d75",
"ref": "refs/heads/master",
"git_url": "git@bitbucket.org:jsmith/another-repo.git",
"commit_info": {
"url": u"https://bitbucket.org/jsmith/another-repo/commits/af64ae7188685f8424040b4735ad12941b980d75",
"date": u"2015-09-10T20:40:54+00:00",
"message": u"Dockerfile edited online with Bitbucket",
"url": "https://bitbucket.org/jsmith/another-repo/commits/af64ae7188685f8424040b4735ad12941b980d75",
"date": "2015-09-10T20:40:54+00:00",
"message": "Dockerfile edited online with Bitbucket",
"author": {
"username": u"John Smith",
"avatar_url": u"https://bitbucket.org/account/jsmith/avatar/32/",
"username": "John Smith",
"avatar_url": "https://bitbucket.org/account/jsmith/avatar/32/",
},
"committer": {
"username": u"John Smith",
"avatar_url": u"https://bitbucket.org/account/jsmith/avatar/32/",
"username": "John Smith",
"avatar_url": "https://bitbucket.org/account/jsmith/avatar/32/",
},
},
}
@ -222,16 +222,16 @@ def test_bitbucket_webhook_payload():
def test_github_webhook_payload_slash_branch():
expected = {
"commit": u"410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"ref": u"refs/heads/slash/branch",
"default_branch": u"master",
"git_url": u"git@github.com:jsmith/anothertest.git",
"commit": "410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"ref": "refs/heads/slash/branch",
"default_branch": "master",
"git_url": "git@github.com:jsmith/anothertest.git",
"commit_info": {
"url": u"https://github.com/jsmith/anothertest/commit/410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"date": u"2015-09-11T14:26:16-04:00",
"message": u"Update Dockerfile",
"committer": {"username": u"jsmith",},
"author": {"username": u"jsmith",},
"url": "https://github.com/jsmith/anothertest/commit/410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"date": "2015-09-11T14:26:16-04:00",
"message": "Update Dockerfile",
"committer": {"username": "jsmith",},
"author": {"username": "jsmith",},
},
}
@ -240,16 +240,16 @@ def test_github_webhook_payload_slash_branch():
def test_github_webhook_payload():
expected = {
"commit": u"410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"ref": u"refs/heads/master",
"default_branch": u"master",
"git_url": u"git@github.com:jsmith/anothertest.git",
"commit": "410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"ref": "refs/heads/master",
"default_branch": "master",
"git_url": "git@github.com:jsmith/anothertest.git",
"commit_info": {
"url": u"https://github.com/jsmith/anothertest/commit/410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"date": u"2015-09-11T14:26:16-04:00",
"message": u"Update Dockerfile",
"committer": {"username": u"jsmith",},
"author": {"username": u"jsmith",},
"url": "https://github.com/jsmith/anothertest/commit/410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"date": "2015-09-11T14:26:16-04:00",
"message": "Update Dockerfile",
"committer": {"username": "jsmith",},
"author": {"username": "jsmith",},
},
}
@ -258,23 +258,23 @@ def test_github_webhook_payload():
def test_github_webhook_payload_with_lookup():
expected = {
"commit": u"410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"ref": u"refs/heads/master",
"default_branch": u"master",
"git_url": u"git@github.com:jsmith/anothertest.git",
"commit": "410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"ref": "refs/heads/master",
"default_branch": "master",
"git_url": "git@github.com:jsmith/anothertest.git",
"commit_info": {
"url": u"https://github.com/jsmith/anothertest/commit/410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"date": u"2015-09-11T14:26:16-04:00",
"message": u"Update Dockerfile",
"url": "https://github.com/jsmith/anothertest/commit/410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"date": "2015-09-11T14:26:16-04:00",
"message": "Update Dockerfile",
"committer": {
"username": u"jsmith",
"url": u"http://github.com/jsmith",
"avatar_url": u"http://some/avatar/url",
"username": "jsmith",
"url": "http://github.com/jsmith",
"avatar_url": "http://some/avatar/url",
},
"author": {
"username": u"jsmith",
"url": u"http://github.com/jsmith",
"avatar_url": u"http://some/avatar/url",
"username": "jsmith",
"url": "http://github.com/jsmith",
"avatar_url": "http://some/avatar/url",
},
},
}
@ -287,14 +287,14 @@ def test_github_webhook_payload_with_lookup():
def test_github_webhook_payload_missing_fields_with_lookup():
expected = {
"commit": u"410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"ref": u"refs/heads/master",
"default_branch": u"master",
"git_url": u"git@github.com:jsmith/anothertest.git",
"commit": "410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"ref": "refs/heads/master",
"default_branch": "master",
"git_url": "git@github.com:jsmith/anothertest.git",
"commit_info": {
"url": u"https://github.com/jsmith/anothertest/commit/410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"date": u"2015-09-11T14:26:16-04:00",
"message": u"Update Dockerfile",
"url": "https://github.com/jsmith/anothertest/commit/410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"date": "2015-09-11T14:26:16-04:00",
"message": "Update Dockerfile",
},
}
@ -309,13 +309,13 @@ def test_github_webhook_payload_missing_fields_with_lookup():
def test_gitlab_webhook_payload():
expected = {
"commit": u"fb88379ee45de28a0a4590fddcbd8eff8b36026e",
"ref": u"refs/heads/master",
"git_url": u"git@gitlab.com:jsmith/somerepo.git",
"commit": "fb88379ee45de28a0a4590fddcbd8eff8b36026e",
"ref": "refs/heads/master",
"git_url": "git@gitlab.com:jsmith/somerepo.git",
"commit_info": {
"url": u"https://gitlab.com/jsmith/somerepo/commit/fb88379ee45de28a0a4590fddcbd8eff8b36026e",
"date": u"2015-08-13T19:33:18+00:00",
"message": u"Fix link\n",
"url": "https://gitlab.com/jsmith/somerepo/commit/fb88379ee45de28a0a4590fddcbd8eff8b36026e",
"date": "2015-08-13T19:33:18+00:00",
"message": "Fix link\n",
},
}
@ -340,14 +340,14 @@ def test_github_webhook_payload_known_issue():
def test_github_webhook_payload_missing_fields():
expected = {
"commit": u"410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"ref": u"refs/heads/master",
"default_branch": u"master",
"git_url": u"git@github.com:jsmith/anothertest.git",
"commit": "410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"ref": "refs/heads/master",
"default_branch": "master",
"git_url": "git@github.com:jsmith/anothertest.git",
"commit_info": {
"url": u"https://github.com/jsmith/anothertest/commit/410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"date": u"2015-09-11T14:26:16-04:00",
"message": u"Update Dockerfile",
"url": "https://github.com/jsmith/anothertest/commit/410f4cdf8ff09b87f245b13845e8497f90b90a4c",
"date": "2015-09-11T14:26:16-04:00",
"message": "Update Dockerfile",
},
}
@ -360,13 +360,13 @@ def test_gitlab_webhook_nocommit_payload():
def test_gitlab_webhook_multiple_commits():
expected = {
"commit": u"9a052a0b2fbe01d4a1a88638dd9fe31c1c56ef53",
"ref": u"refs/heads/master",
"git_url": u"git@gitlab.com:jsmith/some-test-project.git",
"commit": "9a052a0b2fbe01d4a1a88638dd9fe31c1c56ef53",
"ref": "refs/heads/master",
"git_url": "git@gitlab.com:jsmith/some-test-project.git",
"commit_info": {
"url": u"https://gitlab.com/jsmith/some-test-project/commit/9a052a0b2fbe01d4a1a88638dd9fe31c1c56ef53",
"date": u"2016-09-29T15:02:41+00:00",
"message": u"Merge branch 'foobar' into 'master'\r\n\r\nAdd changelog\r\n\r\nSome merge thing\r\n\r\nSee merge request !1",
"url": "https://gitlab.com/jsmith/some-test-project/commit/9a052a0b2fbe01d4a1a88638dd9fe31c1c56ef53",
"date": "2016-09-29T15:02:41+00:00",
"message": "Merge branch 'foobar' into 'master'\r\n\r\nAdd changelog\r\n\r\nSome merge thing\r\n\r\nSee merge request !1",
"author": {
"username": "jsmith",
"url": "http://gitlab.com/jsmith",
@ -387,7 +387,7 @@ def test_gitlab_webhook_multiple_commits():
def test_gitlab_webhook_for_tag():
expected = {
"commit": u"82b3d5ae55f7080f1e6022629cdb57bfae7cccc7",
"commit": "82b3d5ae55f7080f1e6022629cdb57bfae7cccc7",
"commit_info": {
"author": {
"avatar_url": "http://some/avatar/url",
@ -398,8 +398,8 @@ def test_gitlab_webhook_for_tag():
"message": "Fix link\n",
"url": "https://some/url",
},
"git_url": u"git@example.com:jsmith/example.git",
"ref": u"refs/tags/v1.0.0",
"git_url": "git@example.com:jsmith/example.git",
"ref": "refs/tags/v1.0.0",
}
def lookup_user(_):
@ -441,13 +441,13 @@ def test_gitlab_webhook_for_tag_commit_sha_null():
def test_gitlab_webhook_for_tag_known_issue():
expected = {
"commit": u"770830e7ca132856991e6db4f7fc0f4dbe20bd5f",
"ref": u"refs/tags/thirdtag",
"git_url": u"git@gitlab.com:someuser/some-test-project.git",
"commit": "770830e7ca132856991e6db4f7fc0f4dbe20bd5f",
"ref": "refs/tags/thirdtag",
"git_url": "git@gitlab.com:someuser/some-test-project.git",
"commit_info": {
"url": u"https://gitlab.com/someuser/some-test-project/commit/770830e7ca132856991e6db4f7fc0f4dbe20bd5f",
"date": u"2019-10-17T18:07:48Z",
"message": u"Update Dockerfile",
"url": "https://gitlab.com/someuser/some-test-project/commit/770830e7ca132856991e6db4f7fc0f4dbe20bd5f",
"date": "2019-10-17T18:07:48Z",
"message": "Update Dockerfile",
"author": {
"username": "someuser",
"url": "http://gitlab.com/someuser",
@ -468,13 +468,13 @@ def test_gitlab_webhook_for_tag_known_issue():
def test_gitlab_webhook_payload_known_issue():
expected = {
"commit": u"770830e7ca132856991e6db4f7fc0f4dbe20bd5f",
"ref": u"refs/tags/fourthtag",
"git_url": u"git@gitlab.com:someuser/some-test-project.git",
"commit": "770830e7ca132856991e6db4f7fc0f4dbe20bd5f",
"ref": "refs/tags/fourthtag",
"git_url": "git@gitlab.com:someuser/some-test-project.git",
"commit_info": {
"url": u"https://gitlab.com/someuser/some-test-project/commit/770830e7ca132856991e6db4f7fc0f4dbe20bd5f",
"date": u"2019-10-17T18:07:48Z",
"message": u"Update Dockerfile",
"url": "https://gitlab.com/someuser/some-test-project/commit/770830e7ca132856991e6db4f7fc0f4dbe20bd5f",
"date": "2019-10-17T18:07:48Z",
"message": "Update Dockerfile",
},
}
@ -501,13 +501,13 @@ def test_gitlab_webhook_for_other():
def test_gitlab_webhook_payload_with_lookup():
expected = {
"commit": u"fb88379ee45de28a0a4590fddcbd8eff8b36026e",
"ref": u"refs/heads/master",
"git_url": u"git@gitlab.com:jsmith/somerepo.git",
"commit": "fb88379ee45de28a0a4590fddcbd8eff8b36026e",
"ref": "refs/heads/master",
"git_url": "git@gitlab.com:jsmith/somerepo.git",
"commit_info": {
"url": u"https://gitlab.com/jsmith/somerepo/commit/fb88379ee45de28a0a4590fddcbd8eff8b36026e",
"date": u"2015-08-13T19:33:18+00:00",
"message": u"Fix link\n",
"url": "https://gitlab.com/jsmith/somerepo/commit/fb88379ee45de28a0a4590fddcbd8eff8b36026e",
"date": "2015-08-13T19:33:18+00:00",
"message": "Fix link\n",
"author": {
"username": "jsmith",
"url": "http://gitlab.com/jsmith",
@ -528,20 +528,20 @@ def test_gitlab_webhook_payload_with_lookup():
def test_github_webhook_payload_deleted_commit():
expected = {
"commit": u"456806b662cb903a0febbaed8344f3ed42f27bab",
"commit": "456806b662cb903a0febbaed8344f3ed42f27bab",
"commit_info": {
"author": {"username": u"jsmith"},
"committer": {"username": u"jsmith"},
"date": u"2015-12-08T18:07:03-05:00",
"author": {"username": "jsmith"},
"committer": {"username": "jsmith"},
"date": "2015-12-08T18:07:03-05:00",
"message": (
u"Merge pull request #1044 from jsmith/errerror\n\n"
"Merge pull request #1044 from jsmith/errerror\n\n"
+ "Assign the exception to a variable to log it"
),
"url": u"https://github.com/jsmith/somerepo/commit/456806b662cb903a0febbaed8344f3ed42f27bab",
"url": "https://github.com/jsmith/somerepo/commit/456806b662cb903a0febbaed8344f3ed42f27bab",
},
"git_url": u"git@github.com:jsmith/somerepo.git",
"ref": u"refs/heads/master",
"default_branch": u"master",
"git_url": "git@github.com:jsmith/somerepo.git",
"ref": "refs/heads/master",
"default_branch": "master",
}
def lookup_user(_):

5
conf/gunicorn_registry.py
View File

@ -1,3 +1,8 @@
# NOTE: Must be before we import or call anything that may be synchronous.
from gevent import monkey
monkey.patch_all()
import sys
import os

5
conf/gunicorn_secscan.py
View File

@ -1,3 +1,8 @@
# NOTE: Must be before we import or call anything that may be synchronous.
from gevent import monkey
monkey.patch_all()
import sys
import os

5
conf/gunicorn_web.py
View File

@ -1,3 +1,8 @@
# NOTE: Must be before we import or call anything that may be synchronous.
from gevent import monkey
monkey.patch_all()
import sys
import os

4
conf/init/02_get_kube_certs.py
View File

@ -63,7 +63,7 @@ def main():
service_token = f.read()
secret_data = _lookup_secret(service_token).get("data", {})
cert_keys = filter(is_extra_cert, secret_data.keys())
cert_keys = list(filter(is_extra_cert, list(secret_data.keys())))
for cert_key in cert_keys:
if not os.path.exists(KUBE_EXTRA_CA_CERTDIR):
@ -71,7 +71,7 @@ def main():
cert_value = base64.b64decode(secret_data[cert_key])
cert_filename = cert_key.replace(EXTRA_CA_DIRECTORY_PREFIX, "")
print "Found an extra cert %s in config-secret, copying to kube ca dir"
print("Found an extra cert %s in config-secret, copying to kube ca dir")
with open(os.path.join(KUBE_EXTRA_CA_CERTDIR, cert_filename), "w") as f:
f.write(cert_value)

2
conf/init/certs_install.sh
View File

@ -6,7 +6,7 @@ QUAYCONFIG=${QUAYCONFIG:-"$QUAYCONF/stack"}
CERTDIR=${CERTDIR:-"$QUAYCONFIG/extra_ca_certs"}
SYSTEM_CERTDIR=${SYSTEM_CERTDIR:-"/etc/pki/ca-trust/source/anchors"}
PYTHON_ROOT=${PYTHON_ROOT:-"/opt/rh/python27/root/usr/lib/python2.7"}
PYTHON_ROOT=${PYTHON_ROOT:-"/usr/local/lib/python3.6"}
# If we're running under kube, the previous script (02_get_kube_certs.sh) will put the certs in a different location
if [[ "$KUBERNETES_SERVICE_HOST" != "" ]];then

4
conf/init/supervisord_conf_create.py
View File

@ -61,7 +61,7 @@ def limit_services(config, enabled_services):
if enabled_services == []:
return
for service in config.keys():
for service in list(config.keys()):
if service in enabled_services:
config[service]["autostart"] = "true"
else:
@ -72,7 +72,7 @@ def override_services(config, override_services):
if override_services == []:
return
for service in config.keys():
for service in list(config.keys()):
if service + "=true" in override_services:
config[service]["autostart"] = "true"
elif service + "=false" in override_services:

2
conf/init/test/test_supervisord_conf_create.py
View File

@ -50,7 +50,7 @@ def test_supervisord_conf_create_defaults():
):
opts = ServerOptions()
with tempfile.NamedTemporaryFile() as f:
with tempfile.NamedTemporaryFile(mode="w") as f:
f.write(rendered_config_file)
f.flush()

2
conf/nginx/http-base.conf
View File

@ -9,7 +9,7 @@ log_format lb_logs '$remote_addr ($proxy_protocol_addr) '
'($request_time $request_length $upstream_response_time)';
types_hash_max_size 2048;
include /etc/opt/rh/rh-nginx112/nginx/mime.types;
include /etc/nginx/mime.types;
default_type application/octet-stream;

2
config_app/conf/http-base.conf
View File

@ -9,7 +9,7 @@ log_format lb_logs '$remote_addr ($proxy_protocol_addr) '
'($request_time $request_length $upstream_response_time)';
types_hash_max_size 2048;
include /etc/opt/rh/rh-nginx112/nginx/mime.types;
include /etc/nginx/mime.types;
default_type application/octet-stream;

5
config_app/config_application.py
View File

@ -1,10 +1,9 @@
import logging
from config_app import config_web
from config_app.c_app import app as application
from util.log import logfile_path
# Bind all of the blueprints
import config_web
if __name__ == "__main__":
logging.config.fileConfig(logfile_path(debug=True), disable_existing_loggers=False)

4
config_app/config_endpoints/api/__init__.py
View File

@ -117,7 +117,7 @@ def define_json_response(schema_name):
try:
validate(resp, schema)
except ValidationError as ex:
raise InvalidResponse(ex.message)
raise InvalidResponse(str(ex))
return resp
@ -141,7 +141,7 @@ def validate_json_request(schema_name, optional=False):
validate(json_data, schema)
return func(self, *args, **kwargs)
except ValidationError as ex:
raise InvalidRequest(ex.message)
raise InvalidRequest(str(ex))
return wrapped

4
config_app/config_endpoints/api/discovery.py
View File

@ -196,7 +196,7 @@ def generate_route_data():
"404": {"description": "Not found",},
}
for _, body in responses.items():
for _, body in list(responses.items()):
body["schema"] = {"$ref": "#/definitions/ApiError"}
if method_name == "DELETE":
@ -229,7 +229,7 @@ def generate_route_data():
path_swagger[method_name.lower()] = operation_swagger
tags.sort(key=lambda t: t["name"])
paths = OrderedDict(sorted(paths.items(), key=lambda p: p[1]["x-tag"]))
paths = OrderedDict(sorted(list(paths.items()), key=lambda p: p[1]["x-tag"]))
if compact:
return {"paths": paths}

4
config_app/config_endpoints/api/kube_endpoints.py
View File

@ -108,7 +108,7 @@ class QEDeploymentRollback(ApiResource):
kube_accessor.rollback_deployment(name)
except K8sApiException as e:
logger.exception("Failed to rollback deployment.")
return make_response(e.message, 503)
return make_response(str(e), 503)
return make_response("Ok", 204)
@ -127,7 +127,7 @@ class SuperUserKubernetesConfiguration(ApiResource):
KubernetesAccessorSingleton.get_instance().replace_qe_secret(new_secret)
except K8sApiException as e:
logger.exception("Failed to deploy qe config secret to kubernetes.")
return make_response(e.message, 503)
return make_response(str(e), 503)
return make_response("Ok", 201)

4
config_app/config_endpoints/api/superuser.py
View File

@ -138,11 +138,11 @@ class SuperUserCustomCertificates(ApiResource):
)
except CertInvalidException as cie:
cert_views.append(
{"path": extra_cert_path, "error": cie.message,}
{"path": extra_cert_path, "error": str(cie),}
)
except IOError as ioe:
cert_views.append(
{"path": extra_cert_path, "error": ioe.message,}
{"path": extra_cert_path, "error": str(ioe),}
)
return {

4
config_app/config_endpoints/common.py
View File

@ -23,7 +23,7 @@ logger = logging.getLogger(__name__)
TYPE_CONVERTER = {
truthy_bool: "boolean",
str: "string",
basestring: "string",
str: "string",
reqparse.text_type: "string",
int: "integer",
}
@ -67,7 +67,7 @@ def render_page_template(name, route_data=None, js_bundle_name=DEFAULT_JS_BUNDLE
external_scripts=external_scripts,
config_set=frontend_visible_config(app.config),
kubernetes_namespace=IS_KUBERNETES and get_k8s_namespace(),
**kwargs
**kwargs,
)
resp = make_response(contents)

28
config_app/config_test/__init__.py
View File

@ -1,8 +1,8 @@
import json as py_json
import unittest
from contextlib import contextmanager
from urllib import urlencode
from urlparse import urlparse, parse_qs, urlunparse
from urllib.parse import urlencode
from urllib.parse import urlparse, parse_qs, urlunparse
from config_app.c_app import app, config_provider
from config_app.config_endpoints.api import api
@ -64,7 +64,7 @@ class ApiTestCase(unittest.TestCase):
def getJsonResponse(self, resource_name, params={}, expected_code=200):
rv = self.app.get(api.url_for(resource_name, **params))
self.assertEquals(expected_code, rv.status_code)
self.assertEqual(expected_code, rv.status_code)
data = rv.data
parsed = py_json.loads(data)
return parsed
@ -82,12 +82,12 @@ class ApiTestCase(unittest.TestCase):
headers = None
rv = self.app.post(self.url_for(resource_name, params), data=data, headers=headers)
self.assertEquals(rv.status_code, expected_code)
self.assertEqual(rv.status_code, expected_code)
return rv.data
def getResponse(self, resource_name, params={}, expected_code=200):
rv = self.app.get(api.url_for(resource_name, **params))
self.assertEquals(rv.status_code, expected_code)
self.assertEqual(rv.status_code, expected_code)
return rv.data
def putResponse(self, resource_name, params={}, data={}, expected_code=200):
@ -96,22 +96,22 @@ class ApiTestCase(unittest.TestCase):
data=py_json.dumps(data),
headers={"Content-Type": "application/json"},
)
self.assertEquals(rv.status_code, expected_code)
self.assertEqual(rv.status_code, expected_code)
return rv.data
def deleteResponse(self, resource_name, params={}, expected_code=204):
rv = self.app.delete(self.url_for(resource_name, params))
if rv.status_code != expected_code:
print "Mismatch data for resource DELETE %s: %s" % (resource_name, rv.data)
print("Mismatch data for resource DELETE %s: %s" % (resource_name, rv.data))
self.assertEquals(rv.status_code, expected_code)
self.assertEqual(rv.status_code, expected_code)
return rv.data
def deleteEmptyResponse(self, resource_name, params={}, expected_code=204):
rv = self.app.delete(self.url_for(resource_name, params))
self.assertEquals(rv.status_code, expected_code)
self.assertEquals(rv.data, "") # ensure response body empty
self.assertEqual(rv.status_code, expected_code)
self.assertEqual(rv.data, "") # ensure response body empty
return
def postJsonResponse(self, resource_name, params={}, data={}, expected_code=200):
@ -122,9 +122,9 @@ class ApiTestCase(unittest.TestCase):
)
if rv.status_code != expected_code:
print "Mismatch data for resource POST %s: %s" % (resource_name, rv.data)
print("Mismatch data for resource POST %s: %s" % (resource_name, rv.data))
self.assertEquals(rv.status_code, expected_code)
self.assertEqual(rv.status_code, expected_code)
data = rv.data
parsed = py_json.loads(data)
return parsed
@ -139,9 +139,9 @@ class ApiTestCase(unittest.TestCase):
)
if rv.status_code != expected_code:
print "Mismatch data for resource PUT %s: %s" % (resource_name, rv.data)
print("Mismatch data for resource PUT %s: %s" % (resource_name, rv.data))
self.assertEquals(rv.status_code, expected_code)
self.assertEqual(rv.status_code, expected_code)
data = rv.data
parsed = py_json.loads(data)
return parsed

36
config_app/config_test/test_api_usage.py
View File

@ -1,4 +1,4 @@
from StringIO import StringIO
from io import StringIO
from mockldap import MockLdap
from data import database, model
@ -56,7 +56,7 @@ class TestSuperUserCreateInitialSuperUser(ApiTestCase):
# Ensure that the current user is a superuser in the config.
json = self.getJsonResponse(SuperUserConfig)
self.assertEquals(["newsuper"], json["config"]["SUPER_USERS"])
self.assertEqual(["newsuper"], json["config"]["SUPER_USERS"])
# Ensure that the current user is a superuser in memory by trying to call an API
# that will fail otherwise.
@ -67,7 +67,7 @@ class TestSuperUserConfig(ApiTestCase):
def test_get_status_update_config(self):
# With no config the status should be 'config-db'.
json = self.getJsonResponse(SuperUserRegistryStatus)
self.assertEquals("config-db", json["status"])
self.assertEqual("config-db", json["status"])
# Add some fake config.
fake_config = {
@ -78,9 +78,9 @@ class TestSuperUserConfig(ApiTestCase):
json = self.putJsonResponse(
SuperUserConfig, data=dict(config=fake_config, hostname="fakehost")
)
self.assertEquals("fakekey", json["config"]["SECRET_KEY"])
self.assertEquals("fakehost", json["config"]["SERVER_HOSTNAME"])
self.assertEquals("Database", json["config"]["AUTHENTICATION_TYPE"])
self.assertEqual("fakekey", json["config"]["SECRET_KEY"])
self.assertEqual("fakehost", json["config"]["SERVER_HOSTNAME"])
self.assertEqual("Database", json["config"]["AUTHENTICATION_TYPE"])
# With config the status should be 'setup-db'.
# TODO: fix this test
@ -167,12 +167,12 @@ class TestSuperUserCustomCertificates(ApiTestCase):
# Make sure it is present.
json = self.getJsonResponse(SuperUserCustomCertificates)
self.assertEquals(1, len(json["certs"]))
self.assertEqual(1, len(json["certs"]))
cert_info = json["certs"][0]
self.assertEquals("testcert.crt", cert_info["path"])
self.assertEqual("testcert.crt", cert_info["path"])
self.assertEquals(set(["somecoolhost", "bar", "baz"]), set(cert_info["names"]))
self.assertEqual(set(["somecoolhost", "bar", "baz"]), set(cert_info["names"]))
self.assertFalse(cert_info["expired"])
# Remove the certificate.
@ -180,7 +180,7 @@ class TestSuperUserCustomCertificates(ApiTestCase):
# Make sure it is gone.
json = self.getJsonResponse(SuperUserCustomCertificates)
self.assertEquals(0, len(json["certs"]))
self.assertEqual(0, len(json["certs"]))
def test_expired_custom_certificate(self):
# Upload a certificate.
@ -194,12 +194,12 @@ class TestSuperUserCustomCertificates(ApiTestCase):
# Make sure it is present.
json = self.getJsonResponse(SuperUserCustomCertificates)
self.assertEquals(1, len(json["certs"]))
self.assertEqual(1, len(json["certs"]))
cert_info = json["certs"][0]
self.assertEquals("testcert.crt", cert_info["path"])
self.assertEqual("testcert.crt", cert_info["path"])
self.assertEquals(set(["somecoolhost"]), set(cert_info["names"]))
self.assertEqual(set(["somecoolhost"]), set(cert_info["names"]))
self.assertTrue(cert_info["expired"])
def test_invalid_custom_certificate(self):
@ -213,11 +213,11 @@ class TestSuperUserCustomCertificates(ApiTestCase):
# Make sure it is present but invalid.
json = self.getJsonResponse(SuperUserCustomCertificates)
self.assertEquals(1, len(json["certs"]))
self.assertEqual(1, len(json["certs"]))
cert_info = json["certs"][0]
self.assertEquals("testcert.crt", cert_info["path"])
self.assertEquals("no start line", cert_info["error"])
self.assertEqual("testcert.crt", cert_info["path"])
self.assertEqual("no start line", cert_info["error"])
def test_path_sanitization(self):
# Upload a certificate.
@ -231,7 +231,7 @@ class TestSuperUserCustomCertificates(ApiTestCase):
# Make sure it is present.
json = self.getJsonResponse(SuperUserCustomCertificates)
self.assertEquals(1, len(json["certs"]))
self.assertEqual(1, len(json["certs"]))
cert_info = json["certs"][0]
self.assertEquals("foobar.crt", cert_info["path"])
self.assertEqual("foobar.crt", cert_info["path"])

12
config_app/config_test/test_suconfig_api.py
View File

@ -38,7 +38,7 @@ class TestSuperUserRegistryStatus(ApiTestCase):
def test_registry_status_no_config(self):
with FreshConfigProvider():
json = self.getJsonResponse(SuperUserRegistryStatus)
self.assertEquals("config-db", json["status"])
self.assertEqual("config-db", json["status"])
@mock.patch(
"config_app.config_endpoints.api.suconfig.database_is_valid", mock.Mock(return_value=False)
@ -47,7 +47,7 @@ class TestSuperUserRegistryStatus(ApiTestCase):
with FreshConfigProvider():
config_provider.save_config({"key": "value"})
json = self.getJsonResponse(SuperUserRegistryStatus)
self.assertEquals("setup-db", json["status"])
self.assertEqual("setup-db", json["status"])
@mock.patch(
"config_app.config_endpoints.api.suconfig.database_is_valid", mock.Mock(return_value=True)
@ -56,7 +56,7 @@ class TestSuperUserRegistryStatus(ApiTestCase):
with FreshConfigProvider():
config_provider.save_config({"key": "value"})
json = self.getJsonResponse(SuperUserRegistryStatus)
self.assertEquals("config", json["status"])
self.assertEqual("config", json["status"])
@mock.patch(
"config_app.config_endpoints.api.suconfig.database_is_valid", mock.Mock(return_value=True)
@ -68,7 +68,7 @@ class TestSuperUserRegistryStatus(ApiTestCase):
with FreshConfigProvider():
config_provider.save_config({"key": "value"})
json = self.getJsonResponse(SuperUserRegistryStatus)
self.assertEquals("create-superuser", json["status"])
self.assertEqual("create-superuser", json["status"])
@mock.patch(
"config_app.config_endpoints.api.suconfig.database_is_valid", mock.Mock(return_value=True)
@ -80,7 +80,7 @@ class TestSuperUserRegistryStatus(ApiTestCase):
with FreshConfigProvider():
config_provider.save_config({"key": "value", "SETUP_COMPLETE": True})
json = self.getJsonResponse(SuperUserRegistryStatus)
self.assertEquals("config", json["status"])
self.assertEqual("config", json["status"])
class TestSuperUserConfigFile(ApiTestCase):
@ -151,7 +151,7 @@ class TestSuperUserCreateInitialSuperUser(ApiTestCase):
# Verify the superuser was placed into the config.
result = self.getJsonResponse(SuperUserConfig)
self.assertEquals(["cooluser"], result["config"]["SUPER_USERS"])
self.assertEqual(["cooluser"], result["config"]["SUPER_USERS"])
class TestSuperUserConfigValidate(ApiTestCase):

9
config_app/config_util/config/__init__.py
View File

@ -28,13 +28,14 @@ def get_config_as_kube_secret(config_path):
certs_dir = os.path.join(config_path, EXTRA_CA_DIRECTORY)
if os.path.exists(certs_dir):
for extra_cert in os.listdir(certs_dir):
with open(os.path.join(certs_dir, extra_cert)) as f:
data[EXTRA_CA_DIRECTORY_PREFIX + extra_cert] = base64.b64encode(f.read())
file_path = os.path.join(certs_dir, extra_cert)
with open(file_path, "rb") as f:
data[EXTRA_CA_DIRECTORY_PREFIX + extra_cert] = base64.b64encode(f.read()).decode()
for name in os.listdir(config_path):
file_path = os.path.join(config_path, name)
if not os.path.isdir(file_path):
with open(file_path) as f:
data[name] = base64.b64encode(f.read())
with open(file_path, "rb") as f:
data[name] = base64.b64encode(f.read()).decode()
return data

4
config_app/config_util/config/baseprovider.py
View File

@ -37,7 +37,7 @@ def import_yaml(config_obj, config_file):
if isinstance(c, str):
raise Exception("Invalid YAML config file: " + str(c))
for key in c.iterkeys():
for key in c.keys():
if key.isupper():
config_obj[key] = c[key]
@ -54,7 +54,7 @@ def import_yaml(config_obj, config_file):
def get_yaml(config_obj):
return yaml.safe_dump(config_obj, encoding="utf-8", allow_unicode=True)
return yaml.safe_dump(config_obj, allow_unicode=True)
def export_yaml(config_obj, config_file):

22
config_app/config_util/config/test/test_helpers.py
View File

@ -11,7 +11,7 @@ from util.config.validator import EXTRA_CA_DIRECTORY
def _create_temp_file_structure(file_structure):
temp_dir = TemporaryDirectory()
for filename, data in file_structure.iteritems():
for filename, data in file_structure.items():
if filename == EXTRA_CA_DIRECTORY:
extra_ca_dir_path = os.path.join(temp_dir.name, EXTRA_CA_DIRECTORY)
os.mkdir(extra_ca_dir_path)
@ -36,14 +36,17 @@ def _create_temp_file_structure(file_structure):
),
pytest.param(
{"config.yaml": "test:true", "otherfile.ext": "im a file"},
{"config.yaml": "dGVzdDp0cnVl", "otherfile.ext": base64.b64encode("im a file")},
{
"config.yaml": "dGVzdDp0cnVl",
"otherfile.ext": base64.b64encode(b"im a file").decode("ascii"),
},
id="config and another file",
),
pytest.param(
{"config.yaml": "test:true", "extra_ca_certs": [("cert.crt", "im a cert!"),]},
{
"config.yaml": "dGVzdDp0cnVl",
"extra_ca_certs_cert.crt": base64.b64encode("im a cert!"),
"extra_ca_certs_cert.crt": base64.b64encode(b"im a cert!").decode("ascii"),
},
id="config and an extra cert",
),
@ -58,12 +61,19 @@ def _create_temp_file_structure(file_structure):
},
{
"config.yaml": "dGVzdDp0cnVl",
"otherfile.ext": base64.b64encode("im a file"),
"extra_ca_certs_cert.crt": base64.b64encode("im a cert!"),
"extra_ca_certs_another.crt": base64.b64encode("im a different cert!"),
"otherfile.ext": base64.b64encode(b"im a file").decode("ascii"),
"extra_ca_certs_cert.crt": base64.b64encode(b"im a cert!").decode("ascii"),
"extra_ca_certs_another.crt": base64.b64encode(b"im a different cert!").decode(
"ascii"
),
},
id="config, files, and extra certs!",
),
pytest.param(
{"config.yaml": "First line\nSecond line"},
{"config.yaml": "Rmlyc3QgbGluZQpTZWNvbmQgbGluZQ=="},
id="certificate includes newline characters",
),
],
)
def test_get_config_as_kube_secret(file_structure, expected_secret):

6
config_app/config_util/config/test/test_transient_dir_provider.py
View File

@ -36,7 +36,7 @@ from config_app.config_util.config.TransientDirectoryProvider import TransientDi
def test_transient_dir_copy_config_dir(files_to_write, operations, expected_new_dir):
config_provider = TransientDirectoryProvider("", "", "")
for name, data in files_to_write.iteritems():
for name, data in files_to_write.items():
config_provider.write_volume_file(name, data)
config_provider.create_copy_of_config_dir()
@ -53,7 +53,7 @@ def test_transient_dir_copy_config_dir(files_to_write, operations, expected_new_
config_provider.remove_volume_file(delete)
# check that the new directory matches expected state
for filename, data in expected_new_dir.iteritems():
for filename, data in expected_new_dir.items():
with open(os.path.join(config_provider.get_config_dir_path(), filename)) as f:
new_data = f.read()
assert new_data == data
@ -61,7 +61,7 @@ def test_transient_dir_copy_config_dir(files_to_write, operations, expected_new_
# Now check that the old dir matches the original state
saved = config_provider.get_old_config_dir()
for filename, data in files_to_write.iteritems():
for filename, data in files_to_write.items():
with open(os.path.join(saved, filename)) as f:
new_data = f.read()
assert new_data == data

2
config_app/config_util/k8saccessor.py
View File

@ -129,7 +129,7 @@ class KubernetesAccessorSingleton(object):
extra_ca_dir_path = os.path.join(dir_path, EXTRA_CA_DIRECTORY)
os.mkdir(extra_ca_dir_path)
for secret_filename, data in secret_data.iteritems():
for secret_filename, data in secret_data.items():
write_path = os.path.join(dir_path, secret_filename)
if EXTRA_CA_DIRECTORY_PREFIX in secret_filename:

4
config_app/config_util/ssl.py
View File

@ -28,7 +28,7 @@ def load_certificate(cert_contents):
cert = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_PEM, cert_contents)
return SSLCertificate(cert)
except OpenSSL.crypto.Error as ex:
raise CertInvalidException(ex.message[0][2])
raise CertInvalidException(str(ex))
_SUBJECT_ALT_NAME = "subjectAltName"
@ -55,7 +55,7 @@ class SSLCertificate(object):
context.use_privatekey_file(private_key_path)
context.check_privatekey()
except OpenSSL.SSL.Error as ex:
raise KeyInvalidException(ex.message[0][2])
raise KeyInvalidException(str(ex))
def matches_name(self, check_name):
"""

2
data/appr_model/manifest.py
View File

@ -19,7 +19,7 @@ def _ensure_sha256_header(digest):
def _digest(manifestjson):
return _ensure_sha256_header(
hashlib.sha256(json.dumps(manifestjson, sort_keys=True)).hexdigest()
hashlib.sha256(json.dumps(manifestjson, sort_keys=True).encode("utf-8")).hexdigest()
)

4
data/appr_model/manifest_list.py
View File

@ -16,7 +16,7 @@ def _ensure_sha256_header(digest):
def _digest(manifestjson):
return _ensure_sha256_header(
hashlib.sha256(json.dumps(manifestjson, sort_keys=True)).hexdigest()
hashlib.sha256(json.dumps(manifestjson, sort_keys=True).encode("utf-8")).hexdigest()
)
@ -49,7 +49,7 @@ def create_manifestlistmanifest(manifestlist, manifest_ids, manifest_list_json,
From a manifestlist, manifests, and the manifest list blob, create if doesn't exist the
manfiestlistmanifest for each manifest.
"""
for pos in xrange(len(manifest_ids)):
for pos in range(len(manifest_ids)):
manifest_id = manifest_ids[pos]
manifest_json = manifest_list_json[pos]
get_or_create_manifestlistmanifest(

2
data/appr_model/tag.py
View File

@ -144,6 +144,6 @@ def get_most_recent_tag_lifetime_start(repository_ids, models_ref, tag_kind="rel
),
Tag,
)
to_seconds = lambda ms: ms / 1000 if ms is not None else None
to_seconds = lambda ms: ms // 1000 if ms is not None else None
return {t.repository.id: to_seconds(t.lifetime_start) for t in tags}

2
data/buildlogs.py
View File

@ -139,7 +139,7 @@ class RedisBuildLogs(object):
connection.get(self._health_key())
return (True, None)
except redis.RedisError as re:
return (False, "Could not connect to redis: %s" % re.message)
return (False, "Could not connect to redis: %s" % str(re))
class BuildLogs(object):

9
data/database.py
View File

@ -22,7 +22,7 @@ from playhouse.pool import PooledMySQLDatabase, PooledPostgresqlDatabase, Pooled
from sqlalchemy.engine.url import make_url
import resumablehashlib
import rehash
from cachetools.func import lru_cache
from data.fields import (
@ -405,7 +405,7 @@ def _db_from_url(
db_kwargs.pop("timeout", None)
db_kwargs.pop("max_connections", None)
for key, value in _EXTRA_ARGS.get(parsed_url.drivername, {}).iteritems():
for key, value in _EXTRA_ARGS.get(parsed_url.drivername, {}).items():
if key not in db_kwargs:
db_kwargs[key] = value
@ -1112,7 +1112,7 @@ class Image(BaseModel):
"""
Returns an integer list of ancestor ids, ordered chronologically from root to direct parent.
"""
return map(int, self.ancestors.split("/")[1:-1])
return list(map(int, self.ancestors.split("/")[1:-1]))
class DerivedStorageForImage(BaseModel):
@ -1418,7 +1418,8 @@ class BlobUpload(BaseModel):
repository = ForeignKeyField(Repository)
uuid = CharField(index=True, unique=True)
byte_count = BigIntegerField(default=0)
sha_state = ResumableSHA256Field(null=True, default=resumablehashlib.sha256)
# TODO(kleesc): Verify that this is backward compatible with resumablehashlib
sha_state = ResumableSHA256Field(null=True, default=rehash.sha256)
location = ForeignKeyField(ImageStorageLocation)
storage_metadata = JSONField(null=True, default={})
chunk_count = IntegerField(default=0)

4
data/encryption.py
View File

@ -27,7 +27,7 @@ def _encrypt_ccm(secret_key, value, field_max_length=None):
aesccm = AESCCM(secret_key)
nonce = os.urandom(AES_CCM_NONCE_LENGTH)
ct = aesccm.encrypt(nonce, value.encode("utf-8"), None)
encrypted = base64.b64encode(nonce + ct)
encrypted = base64.b64encode(nonce + ct).decode("utf-8")
if field_max_length:
msg = "Tried to encode a value too large for this field"
assert (len(encrypted) + _RESERVED_FIELD_SPACE) <= field_max_length, msg
@ -54,7 +54,7 @@ _VERSIONS = {
"v0": EncryptionVersion("v0", _encrypt_ccm, _decrypt_ccm),
}
_RESERVED_FIELD_SPACE = len(_SEPARATOR) + max([len(k) for k in _VERSIONS.keys()])
_RESERVED_FIELD_SPACE = len(_SEPARATOR) + max([len(k) for k in list(_VERSIONS.keys())])
class FieldEncrypter(object):

52
data/fields.py
View File

@ -1,14 +1,16 @@
import base64
import pickle
import string
import json
from random import SystemRandom
import bcrypt
import resumablehashlib
import rehash
from peewee import TextField, CharField, SmallIntegerField
from data.text import prefix_search
from util.bytes import Bytes
def random_string(length=16):
@ -17,42 +19,44 @@ def random_string(length=16):
class _ResumableSHAField(TextField):
"""
Base Class used to store the state of an in-progress hash in the database. This is particularly
useful for working with large byte streams and allows the hashing to be paused and resumed
as needed.
"""
def _create_sha(self):
raise NotImplementedError
def db_value(self, value):
"""
Serialize the Hasher's state for storage in the database as plain-text.
"""
if value is None:
return None
sha_state = value.state()
# One of the fields is a byte string, let's base64 encode it to make sure
# we can store and fetch it regardless of default collocation.
sha_state[3] = base64.b64encode(sha_state[3])
return json.dumps(sha_state)
serialized_state = base64.b64encode(pickle.dumps(value)).decode("ascii")
return serialized_state
def python_value(self, value):
"""
Restore the Hasher from its state stored in the database.
"""
if value is None:
return None
sha_state = json.loads(value)
# We need to base64 decode the data bytestring.
sha_state[3] = base64.b64decode(sha_state[3])
to_resume = self._create_sha()
to_resume.set_state(sha_state)
return to_resume
hasher = pickle.loads(base64.b64decode(value.encode("ascii")))
return hasher
class ResumableSHA256Field(_ResumableSHAField):
def _create_sha(self):
return resumablehashlib.sha256()
return rehash.sha256()
class ResumableSHA1Field(_ResumableSHAField):
def _create_sha(self):
return resumablehashlib.sha1()
return rehash.sha1()
class JSONField(TextField):
@ -69,12 +73,12 @@ class Base64BinaryField(TextField):
def db_value(self, value):
if value is None:
return None
return base64.b64encode(value)
return base64.b64encode(value).decode("ascii")
def python_value(self, value):
if value is None:
return None
return base64.b64decode(value)
return base64.b64decode(value.encode("ascii"))
class DecryptedValue(object):
@ -84,7 +88,6 @@ class DecryptedValue(object):
def __init__(self, decrypted_value):
assert decrypted_value is not None
assert isinstance(decrypted_value, basestring)
self.value = decrypted_value
def decrypt(self):
@ -180,6 +183,9 @@ def _add_encryption(field_class, requires_length_check=True):
return LazyEncryptedValue(value, self)
def __hash__(self):
return field_class.__hash__(self)
def __eq__(self, _):
raise Exception("Disallowed operation; use `matches`")
@ -322,15 +328,15 @@ class CredentialField(CharField):
if value is None:
return None
if isinstance(value, basestring):
if isinstance(value, str):
raise Exception(
"A string cannot be given to a CredentialField; please wrap in a Credential"
)
return value.hashed
return Bytes.for_string_or_unicode(value.hashed).as_unicode()
def python_value(self, value):
if value is None:
return None
return Credential(value)
return Credential(Bytes.for_string_or_unicode(value).as_encoded_str())

3
data/logs_model/combined_model.py
View File

@ -30,7 +30,8 @@ def _merge_aggregated_log_counts(*args):
matching_keys[kind_date_key] = (kind_id, dt, count)
return [
AggregatedLogCount(kind_id, count, dt) for (kind_id, dt, count) in matching_keys.values()
AggregatedLogCount(kind_id, count, dt)
for (kind_id, dt, count) in list(matching_keys.values())
]

2
data/logs_model/elastic_logs.py
View File

@ -231,7 +231,7 @@ class ElasticsearchLogs(object):
def list_indices(self):
self._initialize()
try:
return self._client.indices.get(self._index_prefix + "*").keys()
return list(self._client.indices.get(self._index_prefix + "*").keys())
except NotFoundError as nfe:
logger.exception("`%s` indices not found: %s", self._index_prefix, nfe.info)
return []

2
data/logs_model/inmemory_model.py
View File

@ -177,7 +177,7 @@ class InMemoryModel(ActionLogsDataInterface):
else:
entries[key] = AggregatedLogCount(entry.kind_id, 1, synthetic_date)
return entries.values()
return list(entries.values())
def count_repository_actions(self, repository, day):
count = 0

8
data/logs_model/logs_producer/kinesis_stream_logs_producer.py
View File

@ -30,9 +30,13 @@ def _partition_key(number_of_shards=None):
key = None
if number_of_shards is not None:
shard_number = random.randrange(0, number_of_shards)
key = hashlib.sha1(KINESIS_PARTITION_KEY_PREFIX + str(shard_number)).hexdigest()
key = hashlib.sha1(
(KINESIS_PARTITION_KEY_PREFIX + str(shard_number)).encode("utf-8")
).hexdigest()
else:
key = hashlib.sha1(KINESIS_PARTITION_KEY_PREFIX + str(random.getrandbits(256))).hexdigest()
key = hashlib.sha1(
(KINESIS_PARTITION_KEY_PREFIX + str(random.getrandbits(256))).encode("utf-8")
).hexdigest()
return key

6
data/logs_model/logs_producer/test/test_json_logs_serializer.py
View File

@ -15,7 +15,7 @@ logger = logging.getLogger(__name__)
TEST_DATETIME = datetime.utcnow()
TEST_JSON_STRING = '{"a": "b", "c": "d"}'
TEST_JSON_STRING_WITH_UNICODE = u'{"éëê": "îôû"}'
TEST_JSON_STRING_WITH_UNICODE = '{"éëê": "îôû"}'
VALID_LOGENTRY = LogEntry(
random_id="123-45", ip="0.0.0.0", metadata_json=TEST_JSON_STRING, datetime=TEST_DATETIME
@ -30,11 +30,11 @@ VALID_LOGENTRY_WITH_UNICODE = LogEntry(
VALID_LOGENTRY_EXPECTED_OUTPUT = (
'{"datetime": "%s", "ip": "0.0.0.0", "metadata_json": "{\\"a\\": \\"b\\", \\"c\\": \\"d\\"}", "random_id": "123-45"}'
% TEST_DATETIME.isoformat()
)
).encode("ascii")
VALID_LOGENTRY_WITH_UNICODE_EXPECTED_OUTPUT = (
'{"datetime": "%s", "ip": "0.0.0.0", "metadata_json": "{\\"\\u00e9\\u00eb\\u00ea\\": \\"\\u00ee\\u00f4\\u00fb\\"}", "random_id": "123-45"}'
% TEST_DATETIME.isoformat()
)
).encode("ascii")
@pytest.mark.parametrize(

2
data/logs_model/shared.py
View File

@ -57,7 +57,7 @@ class SharedModel:
def epoch_ms(dt):
return (timegm(dt.timetuple()) * 1000) + (dt.microsecond / 1000)
return (timegm(dt.timetuple()) * 1000) + (dt.microsecond // 1000)
def get_kinds_filter(kinds):

2
data/logs_model/table_logs_model.py
View File

@ -199,7 +199,7 @@ class TableLogsModel(SharedModel, ActionLogsDataInterface):
else:
entries[key] = AggregatedLogCount(entry.kind_id, entry.count, synthetic_date)
return entries.values()
return list(entries.values())
def count_repository_actions(self, repository, day):
return model.repositoryactioncount.count_repository_actions(repository, day)

31
data/logs_model/test/fake_elasticsearch.py
View File

@ -37,7 +37,7 @@ def fake_elasticsearch(allow_wildcard=True):
# fields here.
if field_name == "datetime":
if isinstance(value, int):
return datetime.utcfromtimestamp(value / 1000)
return datetime.utcfromtimestamp(value // 1000)
parsed = dateutil.parser.parse(value)
return parsed
@ -75,7 +75,7 @@ def fake_elasticsearch(allow_wildcard=True):
def index_delete(url, request):
index_name_or_pattern = url.path[1:]
to_delete = []
for index_name in docs.keys():
for index_name in list(docs.keys()):
if not fnmatch.fnmatch(index_name, index_name_or_pattern):
continue
@ -94,7 +94,7 @@ def fake_elasticsearch(allow_wildcard=True):
def index_lookup(url, request):
index_name_or_pattern = url.path[1:]
found = {}
for index_name in docs.keys():
for index_name in list(docs.keys()):
if not fnmatch.fnmatch(index_name, index_name_or_pattern):
continue
@ -115,7 +115,7 @@ def fake_elasticsearch(allow_wildcard=True):
found = []
found_index = False
for index_name in docs.keys():
for index_name in list(docs.keys()):
if not allow_wildcard and index_name_or_pattern.find("*") >= 0:
break
@ -128,8 +128,8 @@ def fake_elasticsearch(allow_wildcard=True):
if current_query is None:
return True
for filter_type, filter_params in current_query.iteritems():
for field_name, filter_props in filter_params.iteritems():
for filter_type, filter_params in current_query.items():
for field_name, filter_props in filter_params.items():
if filter_type == "range":
lt = transform(filter_props["lt"], field_name)
gte = transform(filter_props["gte"], field_name)
@ -244,7 +244,7 @@ def fake_elasticsearch(allow_wildcard=True):
source = item["_source"]
key = ""
for sort_config in sort:
for sort_key, direction in sort_config.iteritems():
for sort_key, direction in sort_config.items():
assert direction == "desc"
sort_key = sort_key.replace(".keyword", "")
key += str(transform(source[sort_key], sort_key))
@ -258,11 +258,11 @@ def fake_elasticsearch(allow_wildcard=True):
if search_after:
sort_fields = []
for sort_config in sort:
if isinstance(sort_config, unicode):
if isinstance(sort_config, str):
sort_fields.append(sort_config)
continue
for sort_key, _ in sort_config.iteritems():
for sort_key, _ in sort_config.items():
sort_key = sort_key.replace(".keyword", "")
sort_fields.append(sort_key)
@ -304,7 +304,7 @@ def fake_elasticsearch(allow_wildcard=True):
def _by_field(agg_field_params, results):
aggregated_by_field = defaultdict(list)
for agg_means, agg_means_params in agg_field_params.iteritems():
for agg_means, agg_means_params in agg_field_params.items():
if agg_means == "terms":
field_name = agg_means_params["field"]
for result in results:
@ -324,7 +324,7 @@ def fake_elasticsearch(allow_wildcard=True):
# Invoke the aggregation recursively.
buckets = []
for field_value, field_results in aggregated_by_field.iteritems():
for field_value, field_results in aggregated_by_field.items():
aggregated = _aggregate(agg_field_params, field_results)
if isinstance(aggregated, list):
aggregated = {"doc_count": len(aggregated)}
@ -335,12 +335,12 @@ def fake_elasticsearch(allow_wildcard=True):
return {"buckets": buckets}
def _aggregate(query_config, results):
agg_params = query_config.get(u"aggs")
agg_params = query_config.get("aggs")
if not agg_params:
return results
by_field_name = {}
for agg_field_name, agg_field_params in agg_params.iteritems():
for agg_field_name, agg_field_params in agg_params.items():
by_field_name[agg_field_name] = _by_field(agg_field_params, results)
return by_field_name
@ -364,10 +364,7 @@ def fake_elasticsearch(allow_wildcard=True):
@urlmatch(netloc=FAKE_ES_HOST)
def catchall_handler(url, request):
print "Unsupported URL: %s %s" % (
request.method,
url,
)
print("Unsupported URL: %s %s" % (request.method, url,))
return {"status_code": 501}
handlers = [

4
data/logs_model/test/test_elasticsearch.py
View File

@ -14,7 +14,7 @@ from httmock import urlmatch, HTTMock
from data.model.log import _json_serialize
from data.logs_model.elastic_logs import ElasticsearchLogs, INDEX_NAME_PREFIX, INDEX_DATE_FORMAT
from data.logs_model import configure, LogsModelProxy
from mock_elasticsearch import *
from .mock_elasticsearch import *
FAKE_ES_HOST = "fakees"
FAKE_ES_HOST_PATTERN = r"fakees.*"
@ -195,7 +195,7 @@ def mock_elasticsearch():
window_size = query["scroll"]
maximum_result_size = int(query["size"])
return mock.search_scroll_create(window_size, maximum_result_size, json.loads(req.body))
elif "aggs" in req.body:
elif b"aggs" in req.body:
return mock.search_aggs(json.loads(req.body))
else:
return mock.search_after(json.loads(req.body))

8
data/logs_model/test/test_logs_producer.py
View File

@ -7,14 +7,14 @@ import botocore
from data.logs_model import configure
from test_elasticsearch import (
from .test_elasticsearch import (
app_config,
logs_model_config,
logs_model,
mock_elasticsearch,
mock_db_model,
)
from mock_elasticsearch import *
from .mock_elasticsearch import *
logger = logging.getLogger(__name__)
@ -65,7 +65,7 @@ def test_kafka_logs_producers(
producer_config = kafka_logs_producer_config
with patch("kafka.client_async.KafkaClient.check_version"), patch(
"kafka.KafkaProducer.send"
) as mock_send:
) as mock_send, patch("kafka.KafkaProducer._max_usable_produce_magic"):
configure(producer_config)
logs_model.log_action(
"pull_repo",
@ -104,4 +104,4 @@ def test_kinesis_logs_producers(
# Check that a PutRecord api call is made.
# NOTE: The second arg of _make_api_call uses a randomized PartitionKey
mock_send.assert_called_once_with(u"PutRecord", mock_send.call_args_list[0][0][1])
mock_send.assert_called_once_with("PutRecord", mock_send.call_args_list[0][0][1])

6
data/migrations/env.py
View File

@ -2,8 +2,8 @@ import logging
import os
from logging.config import fileConfig
from urllib import unquote
from functools import partial
from urllib.parse import unquote
from alembic import context, op as alembic_op
from alembic.script.revision import ResolutionError
@ -81,7 +81,7 @@ def get_progress_reporter():
labels = {
_process_label_key(k): v
for k, v in os.environ.items()
for k, v in list(os.environ.items())
if k.startswith(PROM_LABEL_PREFIX)
}
@ -130,7 +130,7 @@ def run_migrations_online():
"""
if isinstance(db.obj, SqliteDatabase) and not "DB_URI" in os.environ:
print "Skipping Sqlite migration!"
print("Skipping Sqlite migration!")
return
progress_reporter = get_progress_reporter()

2
data/migrations/tester.py
View File

@ -142,7 +142,7 @@ class PopulateTestDataTester(MigrationTester):
"INSERT INTO %s (%s) VALUES (%s)"
% (table_name, ", ".join(field_names), ", ".join(field_name_vars))
)
logger.info("Executing test query %s with values %s", query, columns.values())
logger.info("Executing test query %s with values %s", query, list(columns.values()))
op.get_bind().execute(query, **columns)
def populate_column(self, table_name, col_name, field_type):

4
data/migrations/versions/5248ddf35167_repository_mirror.py
View File

@ -117,7 +117,7 @@ def upgrade(op, tables, tester):
)
op.add_column(
u"repository", sa.Column("state", sa.Integer(), nullable=False, server_default="0")
"repository", sa.Column("state", sa.Integer(), nullable=False, server_default="0")
)
op.create_index("repository_state", "repository", ["state"], unique=False)
@ -176,7 +176,7 @@ def upgrade(op, tables, tester):
def downgrade(op, tables, tester):
op.drop_column(u"repository", "state")
op.drop_column("repository", "state")
op.drop_table("repomirrorconfig")

8
data/migrations/versions/61cadbacb9fc_add_ability_for_build_triggers_to_be_.py
View File

@ -31,10 +31,10 @@ def upgrade(op, tables, tester):
op.bulk_insert(tables.logentrykind, [{"name": "toggle_repo_trigger"},])
op.add_column(
u"repositorybuildtrigger", sa.Column("disabled_reason_id", sa.Integer(), nullable=True)
"repositorybuildtrigger", sa.Column("disabled_reason_id", sa.Integer(), nullable=True)
)
op.add_column(
u"repositorybuildtrigger",
"repositorybuildtrigger",
sa.Column("enabled", sa.Boolean(), nullable=False, server_default=sa.sql.expression.true()),
)
op.create_index(
@ -68,8 +68,8 @@ def downgrade(op, tables, tester):
type_="foreignkey",
)
op.drop_index("repositorybuildtrigger_disabled_reason_id", table_name="repositorybuildtrigger")
op.drop_column(u"repositorybuildtrigger", "enabled")
op.drop_column(u"repositorybuildtrigger", "disabled_reason_id")
op.drop_column("repositorybuildtrigger", "enabled")
op.drop_column("repositorybuildtrigger", "disabled_reason_id")
op.drop_table("disablereason")
# ### end Alembic commands ###

2
data/migrations/versions/703298a825c2_backfill_new_encrypted_fields.py
View File

@ -72,7 +72,7 @@ def _decrypted(value):
if value is None:
return None
assert isinstance(value, basestring)
assert isinstance(value, str)
return DecryptedValue(value)

4
data/migrations/versions/b4df55dea4b3_add_repository_kind.py
View File

@ -28,7 +28,7 @@ def upgrade(op, tables, tester):
)
op.add_column(
u"repository", sa.Column("kind_id", sa.Integer(), nullable=False, server_default="1")
"repository", sa.Column("kind_id", sa.Integer(), nullable=False, server_default="1")
)
op.create_index("repository_kind_id", "repository", ["kind_id"], unique=False)
op.create_foreign_key(
@ -49,5 +49,5 @@ def downgrade(op, tables, tester):
op.f("fk_repository_kind_id_repositorykind"), "repository", type_="foreignkey"
)
op.drop_index("repository_kind_id", table_name="repository")
op.drop_column(u"repository", "kind_id")
op.drop_column("repository", "kind_id")
op.drop_table("repositorykind")

4
data/migrations/versions/c059b952ed76_remove_unencrypted_fields_and_data.py
View File

@ -24,7 +24,7 @@ logger = logging.getLogger(__name__)
def upgrade(op, tables, tester):
# ### commands auto generated by Alembic - please adjust! ###
op.drop_index("oauthaccesstoken_refresh_token", table_name="oauthaccesstoken")
op.drop_column(u"oauthaccesstoken", "refresh_token")
op.drop_column("oauthaccesstoken", "refresh_token")
op.drop_column("accesstoken", "code")
@ -82,7 +82,7 @@ def upgrade(op, tables, tester):
def downgrade(op, tables, tester):
# ### commands auto generated by Alembic - please adjust! ###
op.add_column(
u"oauthaccesstoken", sa.Column("refresh_token", sa.String(length=255), nullable=True)
"oauthaccesstoken", sa.Column("refresh_token", sa.String(length=255), nullable=True)
)
op.create_index(
"oauthaccesstoken_refresh_token", "oauthaccesstoken", ["refresh_token"], unique=False

56
data/migrations/versions/c13c8052f7a6_add_new_fields_and_tables_for_encrypted_.py
View File

@ -32,46 +32,42 @@ def upgrade(op, tables, tester):
"robotaccounttoken_robot_account_id", "robotaccounttoken", ["robot_account_id"], unique=True
)
op.add_column(u"accesstoken", sa.Column("token_code", sa.String(length=255), nullable=True))
op.add_column(u"accesstoken", sa.Column("token_name", sa.String(length=255), nullable=True))
op.add_column("accesstoken", sa.Column("token_code", sa.String(length=255), nullable=True))
op.add_column("accesstoken", sa.Column("token_name", sa.String(length=255), nullable=True))
op.create_index("accesstoken_token_name", "accesstoken", ["token_name"], unique=True)
op.add_column(
u"appspecificauthtoken", sa.Column("token_name", sa.String(length=255), nullable=True)
"appspecificauthtoken", sa.Column("token_name", sa.String(length=255), nullable=True)
)
op.add_column(
u"appspecificauthtoken", sa.Column("token_secret", sa.String(length=255), nullable=True)
"appspecificauthtoken", sa.Column("token_secret", sa.String(length=255), nullable=True)
)
op.create_index(
"appspecificauthtoken_token_name", "appspecificauthtoken", ["token_name"], unique=True
)
op.add_column(
u"emailconfirmation", sa.Column("verification_code", sa.String(length=255), nullable=True)
"emailconfirmation", sa.Column("verification_code", sa.String(length=255), nullable=True)
)
op.add_column(
u"oauthaccesstoken", sa.Column("token_code", sa.String(length=255), nullable=True)
)
op.add_column(
u"oauthaccesstoken", sa.Column("token_name", sa.String(length=255), nullable=True)
)
op.add_column("oauthaccesstoken", sa.Column("token_code", sa.String(length=255), nullable=True))
op.add_column("oauthaccesstoken", sa.Column("token_name", sa.String(length=255), nullable=True))
op.create_index("oauthaccesstoken_token_name", "oauthaccesstoken", ["token_name"], unique=True)
op.add_column(
u"oauthapplication", sa.Column("secure_client_secret", sa.String(length=255), nullable=True)
"oauthapplication", sa.Column("secure_client_secret", sa.String(length=255), nullable=True)
)
op.add_column(
u"oauthapplication",
"oauthapplication",
sa.Column("fully_migrated", sa.Boolean(), server_default="0", nullable=False),
)
op.add_column(
u"oauthauthorizationcode",
"oauthauthorizationcode",
sa.Column("code_credential", sa.String(length=255), nullable=True),
)
op.add_column(
u"oauthauthorizationcode", sa.Column("code_name", sa.String(length=255), nullable=True)
"oauthauthorizationcode", sa.Column("code_name", sa.String(length=255), nullable=True)
)
op.create_index(
"oauthauthorizationcode_code_name", "oauthauthorizationcode", ["code_name"], unique=True
@ -80,14 +76,14 @@ def upgrade(op, tables, tester):
op.create_index("oauthauthorizationcode_code", "oauthauthorizationcode", ["code"], unique=True)
op.add_column(
u"repositorybuildtrigger",
"repositorybuildtrigger",
sa.Column("secure_auth_token", sa.String(length=255), nullable=True),
)
op.add_column(
u"repositorybuildtrigger", sa.Column("secure_private_key", sa.Text(), nullable=True)
"repositorybuildtrigger", sa.Column("secure_private_key", sa.Text(), nullable=True)
)
op.add_column(
u"repositorybuildtrigger",
"repositorybuildtrigger",
sa.Column("fully_migrated", sa.Boolean(), server_default="0", nullable=False),
)
# ### end Alembic commands ###
@ -114,30 +110,30 @@ def upgrade(op, tables, tester):
def downgrade(op, tables, tester):
# ### commands auto generated by Alembic - please adjust! ###
op.drop_column(u"repositorybuildtrigger", "secure_private_key")
op.drop_column(u"repositorybuildtrigger", "secure_auth_token")
op.drop_column("repositorybuildtrigger", "secure_private_key")
op.drop_column("repositorybuildtrigger", "secure_auth_token")
op.drop_index("oauthauthorizationcode_code", table_name="oauthauthorizationcode")
op.create_index("oauthauthorizationcode_code", "oauthauthorizationcode", ["code"], unique=False)
op.drop_index("oauthauthorizationcode_code_name", table_name="oauthauthorizationcode")
op.drop_column(u"oauthauthorizationcode", "code_name")
op.drop_column(u"oauthauthorizationcode", "code_credential")
op.drop_column("oauthauthorizationcode", "code_name")
op.drop_column("oauthauthorizationcode", "code_credential")
op.drop_column(u"oauthapplication", "secure_client_secret")
op.drop_column("oauthapplication", "secure_client_secret")
op.drop_index("oauthaccesstoken_token_name", table_name="oauthaccesstoken")
op.drop_column(u"oauthaccesstoken", "token_name")
op.drop_column(u"oauthaccesstoken", "token_code")
op.drop_column("oauthaccesstoken", "token_name")
op.drop_column("oauthaccesstoken", "token_code")
op.drop_column(u"emailconfirmation", "verification_code")
op.drop_column("emailconfirmation", "verification_code")
op.drop_index("appspecificauthtoken_token_name", table_name="appspecificauthtoken")
op.drop_column(u"appspecificauthtoken", "token_secret")
op.drop_column(u"appspecificauthtoken", "token_name")
op.drop_column("appspecificauthtoken", "token_secret")
op.drop_column("appspecificauthtoken", "token_name")
op.drop_index("accesstoken_token_name", table_name="accesstoken")
op.drop_column(u"accesstoken", "token_name")
op.drop_column(u"accesstoken", "token_code")
op.drop_column("accesstoken", "token_name")
op.drop_column("accesstoken", "token_code")
op.drop_table("robotaccounttoken")
# ### end Alembic commands ###

2
data/migrationutil.py
View File

@ -65,7 +65,7 @@ class DefinedDataMigration(DataMigration):
@property
def _error_suffix(self):
message = "Available values for this migration: %s. " % (self.phases.keys())
message = "Available values for this migration: %s. " % (list(self.phases.keys()))
message += "If this is a new installation, please use `new-installation`."
return message

4
data/model/_basequery.py
View File

@ -24,6 +24,7 @@ from data.database import (
db_count_estimator,
db,
)
from functools import reduce
logger = logging.getLogger(__name__)
@ -36,7 +37,8 @@ def reduce_as_tree(queries_to_reduce):
This works around a bug in peewee SQL generation where reducing linearly generates a chain of
queries that will exceed the recursion depth limit when it has around 80 queries.
"""
mid = len(queries_to_reduce) / 2
mid = len(queries_to_reduce) // 2
left = queries_to_reduce[:mid]
right = queries_to_reduce[mid:]

3
data/model/appspecifictoken.py
View File

@ -8,6 +8,7 @@ from data.model._basequery import update_last_accessed
from data.fields import DecryptedValue
from util.timedeltastring import convert_to_timedelta
from util.unicode import remove_unicode
from util.bytes import Bytes
logger = logging.getLogger(__name__)
@ -132,7 +133,7 @@ def access_valid_token(token_code):
If found, the token's last_accessed field is set to now and the token is returned. If not found,
returns None.
"""
token_code = remove_unicode(token_code)
token_code = remove_unicode(Bytes.for_string_or_unicode(token_code).as_encoded_str())
prefix = token_code[:TOKEN_NAME_PREFIX_LENGTH]
if len(prefix) != TOKEN_NAME_PREFIX_LENGTH:

2
data/model/blob.py
View File

@ -261,7 +261,7 @@ def get_or_create_shared_blob(digest, byte_data, storage):
special empty gzipped tar layer that Docker no longer pushes to us.
"""
assert digest
assert byte_data is not None
assert byte_data is not None and isinstance(byte_data, bytes)
assert storage
try:

2
data/model/gc.py
View File

@ -166,7 +166,7 @@ def _chunk_iterate_for_deletion(query, chunk_size=10):
while True:
results = list(query.limit(chunk_size))
if not results:
raise StopIteration
return
yield results

4
data/model/health.py
View File

@ -13,11 +13,11 @@ def check_health(app_config):
try:
validate_database_url(app_config["DB_URI"], {}, connect_timeout=3)
except Exception as ex:
return (False, "Could not connect to the database: %s" % ex.message)
return (False, "Could not connect to the database: %s" % str(ex))
# We will connect to the db, check that it contains some team role kinds
try:
okay = bool(list(TeamRole.select().limit(1)))
return (okay, "Could not connect to the database" if not okay else None)
except Exception as ex:
return (False, "Could not connect to the database: %s" % ex.message)
return (False, "Could not connect to the database: %s" % str(ex))

4
data/model/image.py
View File

@ -76,7 +76,7 @@ def get_parent_images(namespace_name, repository_name, image_obj):
parents = _get_repository_images_and_storages(
namespace_name, repository_name, filter_to_parents
)
id_to_image = {unicode(image.id): image for image in parents}
id_to_image = {str(image.id): image for image in parents}
try:
return [id_to_image[parent_id] for parent_id in reversed(parent_db_ids)]
except KeyError as ke:
@ -560,7 +560,7 @@ def _get_uniqueness_hash(varying_metadata):
if not varying_metadata:
return None
return hashlib.sha256(json.dumps(canonicalize(varying_metadata))).hexdigest()
return hashlib.sha256(json.dumps(canonicalize(varying_metadata)).encode("utf-8")).hexdigest()
def find_or_create_derived_storage(

2
data/model/notification.py
View File

@ -132,7 +132,7 @@ def delete_matching_notifications(target, kind_name, **kwargs):
except:
continue
for (key, value) in kwargs.iteritems():
for (key, value) in kwargs.items():
if not key in metadata or metadata[key] != value:
matches = False
break

8
data/model/oauth.py
View File

@ -3,8 +3,8 @@ import json
from flask import url_for
from datetime import datetime, timedelta
from oauth2lib.provider import AuthorizationProvider
from oauth2lib import utils
from oauth.provider import AuthorizationProvider
from oauth import utils
from data.database import (
OAuthApplication,
@ -281,12 +281,12 @@ def create_application(org, name, application_uri, redirect_uri, **kwargs):
application_uri=application_uri,
redirect_uri=redirect_uri,
secure_client_secret=DecryptedValue(client_secret),
**kwargs
**kwargs,
)
def validate_access_token(access_token):
assert isinstance(access_token, basestring)
assert isinstance(access_token, str)
token_name = access_token[:ACCESS_TOKEN_PREFIX_LENGTH]
if not token_name:
return None

10
data/model/oci/manifest.py
View File

@ -294,7 +294,7 @@ def _create_manifest(
# Create the manifest and its blobs.
media_type = Manifest.media_type.get_id(manifest_interface_instance.media_type)
storage_ids = {storage.id for storage in blob_map.values()}
storage_ids = {storage.id for storage in list(blob_map.values())}
with db_transaction():
# Check for the manifest. This is necessary because Postgres doesn't handle IntegrityErrors
@ -349,7 +349,7 @@ def _create_manifest(
if child_manifest_rows:
children_to_insert = [
dict(manifest=manifest, child_manifest=child_manifest, repository=repository_id)
for child_manifest in child_manifest_rows.values()
for child_manifest in list(child_manifest_rows.values())
]
ManifestChild.insert_many(children_to_insert).execute()
@ -366,7 +366,7 @@ def _create_manifest(
# application to the manifest occur under the transaction.
labels = manifest_interface_instance.get_manifest_labels(retriever)
if labels:
for key, value in labels.iteritems():
for key, value in labels.items():
# NOTE: There can technically be empty label keys via Dockerfile's. We ignore any
# such `labels`, as they don't really mean anything.
if not key:
@ -381,11 +381,11 @@ def _create_manifest(
# to ensure that any action performed is defined in all manifests.
labels_to_apply = labels or {}
if child_manifest_label_dicts:
labels_to_apply = child_manifest_label_dicts[0].viewitems()
labels_to_apply = child_manifest_label_dicts[0].items()
for child_manifest_label_dict in child_manifest_label_dicts[1:]:
# Intersect the key+values of the labels to ensure we get the exact same result
# for all the child manifests.
labels_to_apply = labels_to_apply & child_manifest_label_dict.viewitems()
labels_to_apply = labels_to_apply & child_manifest_label_dict.items()
labels_to_apply = dict(labels_to_apply)

3
data/model/oci/retriever.py
View File

@ -4,6 +4,7 @@ from image.shared.interfaces import ContentRetriever
from data.database import Manifest
from data.model.oci.blob import get_repository_blob_by_digest
from data.model.storage import get_layer_path
from util.bytes import Bytes
RETRY_COUNT = 5
RETRY_DELAY = 0.3 # seconds
@ -34,7 +35,7 @@ class RepositoryContentRetriever(ContentRetriever):
)
try:
return query.get().manifest_bytes
return Bytes.for_string_or_unicode(query.get().manifest_bytes).as_encoded_str()
except Manifest.DoesNotExist:
return None

Some files were not shown because too many files have changed in this diff Show More