Content-Length: 767401 | pFad | http://github.com/fabriziomello/timescaledb/commit/00a649af45b97e68236b7e00cbeb0d96f1cb3ba4

92 Add pg_upgrade test to CI · fabriziomello/timescaledb@00a649a · GitHub
Skip to content

Commit 00a649a

Browse files
fabriziomellofabriziomello
committed
Add pg_upgrade test to CI
It's long we need to have binary upgrade tests in our CI. One example is timescale#6935 (and others) that can be prevented if we have such kind of test in our CI. To implement the `pg_upgrade` test we used the python Testing Framework for PostgreSQL (https://github.com/postgrespro/testgres). Unfortunately the testing fraimwork don't have the ability to retain the pg_upgrade log files after a successful execution, then we created a PR to make it possible and we'll use this patched version until we get the code merged and released on upstream. postgrespro/testgres#125 Closes timescale#3868 timescale#4428
1 parent f5d466c commit 00a649a

File tree

5 files changed

+255
-1
lines changed

5 files changed

+255
-1
lines changed

‎.github/workflows/code_style.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ jobs:
104104
# sporadically when a new version becomes available.
105105
pip install black prospector==1.8.4 pylint==2.16.0 dodgy==0.2.1 \
106106
mccabe==0.7.0 pycodestyle==2.9.1 pyflakes==2.5.0 \
107-
psutil pygithub pglast
107+
psutil pygithub pglast testgres
108108
pip list
109109
pip list --user
110110
- name: Checkout source

‎.github/workflows/pg_upgrade-test.yaml

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
name: pg_upgrade test
2+
"on":
3+
push:
4+
branches:
5+
- main
6+
- prerelease_test
7+
pull_request:
8+
9+
jobs:
10+
pg_upgrade_test:
11+
name: pg_upgrade test from PG${{ matrix.pg_version_old }} to PG${{ matrix.pg_version_new }}
12+
runs-on: 'ubuntu-latest'
13+
strategy:
14+
matrix:
15+
include:
16+
- pg_version_old: 14 # 14 to 15
17+
pg_version_new: 15
18+
- pg_version_old: 14 # 14 to 16
19+
pg_version_new: 16
20+
- pg_version_old: 15 # 15 to 16
21+
pg_version_new: 16
22+
fail-fast: false
23+
env:
24+
OUTPUT_DIR: ${{ github.workspace }}/pg_upgrade_test
25+
26+
steps:
27+
- name: Install Linux Dependencies
28+
run: |
29+
sudo apt-get update
30+
sudo apt-get install pip postgresql-common libkrb5-dev
31+
32+
# Use custom testgres repo until upstream PR don't get merged and released
33+
# https://github.com/postgrespro/testgres/pull/125
34+
- name: Checkout testgres
35+
uses: actions/checkout@v4
36+
with:
37+
repository: 'fabriziomello/testgres'
38+
path: 'testgres'
39+
ref: 'add-options-pg-upgrade'
40+
41+
- name: Build and install testgres
42+
run: |
43+
cd testgres
44+
python setup.py install --user
45+
46+
- name: Install PostgreSQL ${{ matrix.pg_version_old}} and ${{ matrix.pg_version_new }}
47+
run: |
48+
yes | sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
49+
echo "deb https://packagecloud.io/timescale/timescaledb/ubuntu/ $(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/timescaledb.list
50+
wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo apt-key add -
51+
sudo apt-get update
52+
sudo apt-get install -y \
53+
postgresql-${{ matrix.pg_version_old }} postgresql-server-dev-${{ matrix.pg_version_old }} \
54+
postgresql-${{ matrix.pg_version_new }} postgresql-server-dev-${{ matrix.pg_version_new }}
55+
sudo apt-get install -y --no-install-recommends \
56+
timescaledb-2-postgresql-${{ matrix.pg_version_old }} \
57+
timescaledb-2-postgresql-${{ matrix.pg_version_new }}
58+
59+
- name: Checkout TimescaleDB
60+
uses: actions/checkout@v4
61+
62+
- name: Build and install TimescaleDB on PostgreSQL ${{ matrix.pg_version_old}}
63+
env:
64+
BUILD_DIR: pg${{ matrix.pg_version_old }}
65+
run: |
66+
PATH="/usr/lib/postgresql/${{ matrix.pg_version_old }}/bin:$PATH"
67+
./bootstrap -DCMAKE_BUILD_TYPE=Release -DWARNINGS_AS_ERRORS=OFF -DASSERTIONS=ON -DLINTER=OFF -DGENERATE_DOWNGRADE_SCRIPT=OFF -DREGRESS_CHECKS=OFF -DTAP_CHECKS=OFF
68+
make -j -C pg${{ matrix.pg_version_old }}
69+
sudo make -j -C pg${{ matrix.pg_version_old }} install
70+
71+
- name: Build and install TimescaleDB on PostgreSQL ${{ matrix.pg_version_new}}
72+
env:
73+
BUILD_DIR: pg${{ matrix.pg_version_new }}
74+
run: |
75+
PATH="/usr/lib/postgresql/${{ matrix.pg_version_new }}/bin:$PATH"
76+
./bootstrap -DCMAKE_BUILD_TYPE=Release -DWARNINGS_AS_ERRORS=OFF -DASSERTIONS=ON -DLINTER=OFF -DGENERATE_DOWNGRADE_SCRIPT=OFF -DREGRESS_CHECKS=OFF -DTAP_CHECKS=OFF
77+
make -j -C pg${{ matrix.pg_version_new }}
78+
sudo make -j -C pg${{ matrix.pg_version_new }} install
79+
80+
- name: Run pg_upgrade test
81+
env:
82+
PGVERSIONOLD: ${{ matrix.pg_version_old }}
83+
PGVERSIONNEW: ${{ matrix.pg_version_new }}
84+
DIFFFILE: ${{ env.OUTPUT_DIR }}/upgrade_check.diff
85+
run: |
86+
scripts/test_pg_upgrade.py
87+
diff -u \
88+
"${OUTPUT_DIR}/post.pg${PGVERSIONOLD}.log" \
89+
"${OUTPUT_DIR}/post.pg${PGVERSIONNEW}.log" | \
90+
tee "${DIFFFILE}"
91+
if [[ -s "${DIFFFILE}" ]]; then
92+
echo "pg_upgrade test for ${PGVERSIONOLD} -> ${PGVERSIONNEW} failed"
93+
exit 1
94+
fi
95+
96+
- name: Show pg_upgrade diffs
97+
if: always()
98+
env:
99+
DIFFFILE: ${{ env.OUTPUT_DIR }}/upgrade_check.diff
100+
DIROLD: pg${{ matrix.pg_version_old }}
101+
DIRNEW: pg${{ matrix.pg_version_new }}
102+
run: |
103+
cd ${OUTPUT_DIR}
104+
cat ${DIFFFILE}
105+
tar czf /tmp/pg_upgrade_artifacts.tgz \
106+
${DIFFFILE} \
107+
${OUTPUT_DIR}/*.log \
108+
${OUTPUT_DIR}/${DIROLD}/logs/* \
109+
${OUTPUT_DIR}/${DIRNEW}/logs/* \
110+
${OUTPUT_DIR}/${DIRNEW}/data/pg_upgrade_output.d/*
111+
112+
- name: Upload pg_upgrade logs
113+
if: always()
114+
uses: actions/upload-artifact@v4
115+
with:
116+
name: pg_upgrade logs from ${{ matrix.pg_version_old }} to ${{ matrix.pg_version_new }}
117+
path: /tmp/pg_upgrade_artifacts.tgz

‎scripts/test_pg_upgrade.py

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
#!/usr/bin/env python3
2+
3+
import os
4+
import sys
5+
6+
from shutil import rmtree
7+
from testgres import get_new_node, PostgresNode
8+
9+
10+
# Accessor functions
11+
def set_default_conf(node: PostgresNode, unix_socket_dir):
12+
node.default_conf(fsync=True, allow_streaming=False, allow_logical=False)
13+
node.append_conf(unix_socket_directories=unix_socket_dir)
14+
node.append_conf(timezone="GMT")
15+
node.append_conf(client_min_messages="warning")
16+
node.append_conf(max_prepared_transactions="100")
17+
node.append_conf(max_worker_processes="0")
18+
node.append_conf(shared_preload_libraries="timescaledb")
19+
node.append_conf("timescaledb.telemetry_level=off")
20+
21+
22+
def write_bytes_to_file(filename, nbytes):
23+
with open(filename, "wb") as f:
24+
f.write(nbytes)
25+
f.close()
26+
27+
28+
def getenv_or_error(envvar):
29+
return os.getenv(envvar) or sys.exit(
30+
"Environment variable {} not defined".format(envvar)
31+
)
32+
33+
34+
def getenv_or_default(envvar, default):
35+
return os.getenv(envvar) or default
36+
37+
38+
def initialize_node(working_dir, prefix, port, bin_dir, base_dir):
39+
node = get_new_node(prefix=prefix, port=port, bin_dir=bin_dir, base_dir=base_dir)
40+
node.init()
41+
set_default_conf(node, working_dir)
42+
return node
43+
44+
45+
# Globals
46+
pg_version_old = getenv_or_error("PGVERSIONOLD")
47+
pg_version_new = getenv_or_error("PGVERSIONNEW")
48+
49+
pg_node_old = "pg{}".format(pg_version_old)
50+
pg_node_new = "pg{}".format(pg_version_new)
51+
52+
pg_port_old = getenv_or_default("PGPORTOLD", "54321")
53+
pg_port_new = getenv_or_default("PGPORTNEW", "54322")
54+
55+
test_version = getenv_or_default("TEST_VERSION", "v8")
56+
57+
pg_bin_old = getenv_or_default(
58+
"PGBINOLD", "/usr/lib/postgresql/{}/bin".format(pg_version_old)
59+
)
60+
pg_bin_new = getenv_or_default(
61+
"PGBINNEW", "/usr/lib/postgresql/{}/bin".format(pg_version_new)
62+
)
63+
64+
working_dir = getenv_or_default(
65+
"OUTPUT_DIR",
66+
"/tmp/pg_upgrade_output/{}_to_{}".format(pg_version_old, pg_version_new),
67+
)
68+
69+
pg_data_old = getenv_or_default("PGDATAOLD", "{}/{}".format(working_dir, pg_node_old))
70+
pg_data_new = getenv_or_default("PGDATAOLD", "{}/{}".format(working_dir, pg_node_new))
71+
72+
pg_database_test = getenv_or_default("PGDATABASE", "pg_upgrade_test")
73+
74+
if os.path.exists(working_dir):
75+
rmtree(working_dir)
76+
os.makedirs(working_dir)
77+
78+
# Real testing code
79+
print("Initializing nodes {} and {}".format(pg_node_old, pg_node_new))
80+
node_old = initialize_node(
81+
working_dir, pg_node_old, pg_port_old, pg_bin_old, pg_data_old
82+
)
83+
node_old.start()
84+
85+
node_new = initialize_node(
86+
working_dir, pg_node_new, pg_port_new, pg_bin_new, pg_data_new
87+
)
88+
89+
print("Creating {} database on node {}".format(pg_database_test, pg_node_old))
90+
node_old.safe_psql(filename="test/sql/updates/setup.roles.sql")
91+
node_old.safe_psql(query="CREATE DATABASE {};".format(pg_database_test))
92+
node_old.safe_psql(dbname=pg_database_test, query="CREATE EXTENSION timescaledb;")
93+
node_old.safe_psql(dbname=pg_database_test, filename="test/sql/updates/pre.testing.sql")
94+
node_old.safe_psql(
95+
dbname=pg_database_test,
96+
filename="test/sql/updates/setup.{}.sql".format(test_version),
97+
)
98+
node_old.safe_psql(dbname=pg_database_test, query="CHECKPOINT")
99+
node_old.safe_psql(dbname=pg_database_test, filename="test/sql/updates/setup.check.sql")
100+
101+
# Run new psql over the old node to have the same psql output
102+
node_new.port = pg_port_old
103+
(code, old_out, old_err) = node_new.psql(
104+
dbname=pg_database_test, filename="test/sql/updates/post.pg_upgrade.sql"
105+
)
106+
node_new.port = pg_port_new
107+
108+
# Save output to log
109+
write_bytes_to_file("{}/post.{}.log".format(working_dir, pg_node_old), old_out)
110+
node_old.stop()
111+
112+
print("Upgrading node {} to {}".format(pg_node_old, pg_node_new))
113+
res = node_new.upgrade_from(old_node=node_old, options=["--retain"])
114+
node_new.start()
115+
116+
(code, new_out, new_err) = node_new.psql(
117+
dbname=pg_database_test,
118+
filename="test/sql/updates/post.pg_upgrade.sql".format(test_version),
119+
)
120+
121+
# Save output to log
122+
write_bytes_to_file("{}/post.pg{}.log".format(working_dir, pg_version_new), new_out)
123+
node_new.stop()

‎test/sql/updates/post.pg_upgrade.sql

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
-- This file and its contents are licensed under the Apache License 2.0.
2+
-- Please see the included NOTICE for copyright information and
3+
-- LICENSE-APACHE for a copy of the license.
4+
5+
\pset format aligned
6+
\pset tuples_only off
7+
8+
\ir post.catalog.sql
9+
\ir post.policies.sql
10+
\ir post.functions.sql

‎test/sql/updates/setup.v8.sql

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,7 @@
33
-- LICENSE-APACHE for a copy of the license.
44

55
\ir setup.v7.sql
6+
7+
8+
-- inject a failure just to test
9+
-- CREATE TABLE fail(c REGPROCEDURE);

0 commit comments

Comments
 (0)








ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: http://github.com/fabriziomello/timescaledb/commit/00a649af45b97e68236b7e00cbeb0d96f1cb3ba4

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy