Skip to content

Commit c9b20bf

Browse files
edenhilledenhill
committed
Docker-based manylinux packaging and testing
Provides prebuilt dependency-less (or dependency-included to be correct) binary wheels for Linux.
1 parent d0b8563 commit c9b20bf

File tree

4 files changed

+211
-15
lines changed

4 files changed

+211
-15
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,3 +15,4 @@ confluent?kafka.egg-info
1515
confluent-kafka-0.*.*
1616
tmp-build
1717
.tox
18+
wheelhouse

‎README.md

Lines changed: 17 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ c.close()
6868
**AvroProducer**
6969

7070
```python
71-
from confluent_kafka import avro
71+
from confluent_kafka import avro
7272
from confluent_kafka.avro import AvroProducer
7373

7474
value_schema = avro.load('ValueSchema.avsc')
@@ -103,7 +103,7 @@ while running:
103103
except SerializerError as e:
104104
print("Message deserialization failed for %s: %s" % (msg, e))
105105
running = False
106-
106+
107107
c.close()
108108
```
109109

@@ -120,12 +120,12 @@ by the broker, thus you will need to hint the Python client what protocol
120120
version it may use. This is done through two configuration settings:
121121

122122
* `broker.version.fallback=YOUR_BROKER_VERSION` (default 0.9.0.1)
123-
* `api.version.request=true|false` (default false)
123+
* `api.version.request=true|false` (default true)
124124

125-
When using a Kafka 0.10 broker or later you only need to set
126-
`api.version.request=true`.
127-
If you use Kafka broker 0.9 or 0.8 you should leave
128-
`api.version.request=false` (default) and set
125+
When using a Kafka 0.10 broker or later you don't need to do anything
126+
(`api.version.request=true` is the default).
127+
If you use Kafka broker 0.9 or 0.8 you must set
128+
`api.version.request=false` and set
129129
`broker.version.fallback` to your broker version,
130130
e.g `broker.version.fallback=0.9.0.1`.
131131

@@ -136,17 +136,19 @@ https://github.com/edenhill/librdkafka/wiki/Broker-version-compatibility
136136
Prerequisites
137137
=============
138138

139-
* Python >= 2.7 or Python 3.x
140-
* [librdkafka](https://github.com/edenhill/librdkafka) >= 0.9.1
141-
142-
143-
For **Debian/Ubuntu**** based systems, add this APT repo and then do `sudo apt-get install librdkafka-dev python-dev`:
139+
* Python >= 2.6 or Python 3.x
140+
* [librdkafka](https://github.com/edenhill/librdkafka) >= 0.9.1 (embedded in Linux wheels)
141+
142+
librdkafka is embedded in the manylinux wheels, for other platforms or
143+
when a specific version of librdkafka is desired, following these guidelines:
144+
145+
* For **Debian/Ubuntu**** based systems, add this APT repo and then do `sudo apt-get install librdkafka-dev python-dev`:
144146
http://docs.confluent.io/current/installation.html#installation-apt
145147

146-
For **RedHat** and **RPM**-based distros, add this YUM repo and then do `sudo yum install librdkafka-devel python-devel`:
148+
* For **RedHat** and **RPM**-based distros, add this YUM repo and then do `sudo yum install librdkafka-devel python-devel`:
147149
http://docs.confluent.io/current/installation.html#rpm-packages-via-yum
148150

149-
On **OSX**, use **homebrew** and do `sudo brew install librdkafka`
151+
* On **OSX**, use **homebrew** and do `sudo brew install librdkafka`
150152

151153

152154
Install
@@ -155,7 +157,7 @@ Install
155157
**Install from PyPi:**
156158

157159
$ pip install confluent-kafka
158-
160+
159161
# for AvroProducer or AvroConsumer
160162
$ pip install confluent-kafka[avro]
161163

‎tools/build-manylinux.sh

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
#!/bin/bash
2+
#
3+
#
4+
# Builds autonomous Python packages including all dependencies
5+
# using the excellent manylinux docker images and the equally awesome
6+
# auditwheel tool.
7+
#
8+
# This script should be run in a docker image where the confluent-kafka-python
9+
# directory is mapped as /io .
10+
#
11+
# Usage on host:
12+
# tools/build-manylinux.sh <librdkafka_tag>
13+
#
14+
# Usage in container:
15+
# docker run -t -v $(pwd):/io quay.io/pypa/manylinux1_x86_64:latest /io/tools/build-manylinux.sh <librdkafka_tag>
16+
17+
# NOTE: Keep this updated to make sure we always build the latest
18+
# version of OpenSSL in the 1.0 release train.
19+
OPENSSL_VERSION=1.0.2l
20+
21+
LIBRDKAFKA_VERSION=$1
22+
23+
if [[ -z "$LIBRDKAFKA_VERSION" ]]; then
24+
echo "Usage: $0 <librdkafka_tag>"
25+
exit 1
26+
fi
27+
28+
set -ex
29+
30+
if [[ ! -f /.dockerenv ]]; then
31+
#
32+
# Running on host, fire up a docker container a run it.
33+
#
34+
35+
if [[ ! -f tools/$(basename $0) ]]; then
36+
echo "Must be called from confluent-kafka-python root directory"
37+
exit 1
38+
fi
39+
40+
docker run -t -v $(pwd):/io quay.io/pypa/manylinux1_x86_64:latest /io/tools/build-manylinux.sh "$LIBRDKAFKA_VERSION"
41+
42+
exit $?
43+
fi
44+
45+
46+
#
47+
# Running in container
48+
#
49+
50+
function install_openssl {
51+
rm -rf build-openssl
52+
mkdir -p build-openssl
53+
pushd build-openssl
54+
curl -s -l https://www.openssl.org/source/openssl-${OPENSSL_VERSION}.tar.gz | \
55+
tar -xz --strip-components=1 -f -
56+
./config --prefix=/usr/local zlib no-krb5 zlib shared
57+
echo "## building openssl"
58+
make 2>&1 | tail -50
59+
echo "## testing openssl"
60+
make test 2>&1 | tail -50
61+
echo "## installing openssl"
62+
make install 2>&1 | tail -50
63+
popd
64+
}
65+
66+
echo "# Installing basic system dependencies"
67+
yum install -y zlib-devel gcc-c++
68+
69+
echo "# Building OpenSSL ${OPENSSL_VERSION}"
70+
install_openssl
71+
72+
echo "# Building librdkafka ${LIBRDKAFKA_VERSION}"
73+
$(dirname $0)/bootstrap-librdkafka.sh ${LIBRDKAFKA_VERSION} /usr/local
74+
75+
# Compile wheels
76+
echo "# Compile"
77+
for PYBIN in /opt/python/*/bin; do
78+
echo "## Compiling $PYBIN"
79+
CFLAGS="-Werror -Wno-strict-aliasing -Wno-parentheses" \
80+
"${PYBIN}/pip" wheel /io/ -w unrepaired-wheelhouse/
81+
done
82+
83+
# Bundle external shared libraries into the wheels
84+
echo "# auditwheel repair"
85+
mkdir -p /io/wheelhouse
86+
for whl in unrepaired-wheelhouse/*.whl; do
87+
echo "## Repairing $whl"
88+
auditwheel repair "$whl" -w /io/wheelhouse
89+
done
90+
91+
echo "# Repaired wheels"
92+
for whl in /io/wheelhouse/*.whl; do
93+
echo "## Repaired wheel $whl"
94+
auditwheel show "$whl"
95+
done
96+
97+
# Install packages and test
98+
echo "# Installing wheels"
99+
for PYBIN in /opt/python/*/bin/; do
100+
echo "## Installing $PYBIN"
101+
"${PYBIN}/pip" install confluent_kafka --no-index -f /io/wheelhouse
102+
"${PYBIN}/python" -c 'import confluent_kafka; print(confluent_kafka.libversion())'
103+
echo "## Uninstalling $PYBIN"
104+
"${PYBIN}/pip" uninstall -y confluent_kafka
105+
done
106+
107+
108+

‎tools/test-manylinux.sh

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
#!/bin/bash
2+
#
3+
#
4+
# Tests the manylinux wheels on a plethora of bare-bone Linux docker images.
5+
6+
set -ex
7+
8+
9+
function setup_centos {
10+
# CentOS container setup
11+
yum install -q -y python epel-release
12+
yum install -q -y python-pip
13+
}
14+
15+
function setup_ubuntu {
16+
# Ubuntu container setup
17+
apt-get update
18+
apt-get install -y python python-pip
19+
}
20+
21+
22+
function run_single_in_docker {
23+
# Run single test inside docker container
24+
25+
# Detect OS
26+
if grep -qi centos /etc/system-release ; then
27+
setup_centos
28+
elif grep -qi ubuntu /etc/os-release ; then
29+
setup_ubuntu
30+
else
31+
echo "WARNING: Don't know what platform I'm on: $(uname -a)"
32+
fi
33+
34+
# Make sure pip itself is up to date
35+
pip install -U pip
36+
hash -r # let go of previous 'pip'
37+
38+
# Install modules
39+
pip install confluent_kafka --no-index -f /io/wheelhouse
40+
pip install pytest
41+
42+
43+
pushd /io/tests
44+
# Remove cached files from previous runs
45+
rm -rf __pycache__ *.pyc
46+
# Test
47+
pytest --import-mode=append --ignore=avro
48+
popd
49+
50+
}
51+
52+
function run_all_with_docker {
53+
# Run tests in all listed docker containers.
54+
# This is executed on the host.
55+
56+
[[ ! -z $DOCKER_IMAGES ]] || \
57+
DOCKER_IMAGES="ubuntu:14.04 ubuntu:devel centos:6.6 centos:latest"
58+
59+
60+
_wheels="wheelhouse/*manylinux*.whl"
61+
if [[ -z $_wheels ]]; then
62+
echo "No wheels in wheelhouse/, must run build-manylinux.sh first"
63+
exit 1
64+
fi
65+
66+
for DOCKER_IMAGE in $DOCKER_IMAGES; do
67+
echo "# Testing on $DOCKER_IMAGE"
68+
docker run -v $(pwd):/io $DOCKER_IMAGE /io/tools/test-manylinux.sh || \
69+
(echo "Failed on $DOCKER_IMAGE" ; false)
70+
71+
done
72+
}
73+
74+
75+
76+
if [[ -f /.dockerenv && -d /io ]]; then
77+
# Called from within a docker container
78+
run_single_in_docker
79+
80+
else
81+
# Run from host, trigger runs for all docker images.
82+
run_all_with_docker
83+
fi
84+
85+

0 commit comments

Comments
 (0)
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy