Skip to content

Commit 00a649a

Browse files
fabriziomellofabriziomello
committed
Add pg_upgrade test to CI
It's long we need to have binary upgrade tests in our CI. One example is timescale#6935 (and others) that can be prevented if we have such kind of test in our CI. To implement the `pg_upgrade` test we used the python Testing Framework for PostgreSQL (https://github.com/postgrespro/testgres). Unfortunately the testing framework don't have the ability to retain the pg_upgrade log files after a successful execution, then we created a PR to make it possible and we'll use this patched version until we get the code merged and released on upstream. postgrespro/testgres#125 Closes timescale#3868 timescale#4428
1 parent f5d466c commit 00a649a

File tree

5 files changed

+255
-1
lines changed

5 files changed

+255
-1
lines changed

‎.github/workflows/code_style.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ jobs:
104104
# sporadically when a new version becomes available.
105105
pip install black prospector==1.8.4 pylint==2.16.0 dodgy==0.2.1 \
106106
mccabe==0.7.0 pycodestyle==2.9.1 pyflakes==2.5.0 \
107-
psutil pygithub pglast
107+
psutil pygithub pglast testgres
108108
pip list
109109
pip list --user
110110
- name: Checkout source

‎.github/workflows/pg_upgrade-test.yaml

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
name: pg_upgrade test
2+
"on":
3+
push:
4+
branches:
5+
- main
6+
- prerelease_test
7+
pull_request:
8+
9+
jobs:
10+
pg_upgrade_test:
11+
name: pg_upgrade test from PG${{ matrix.pg_version_old }} to PG${{ matrix.pg_version_new }}
12+
runs-on: 'ubuntu-latest'
13+
strategy:
14+
matrix:
15+
include:
16+
- pg_version_old: 14 # 14 to 15
17+
pg_version_new: 15
18+
- pg_version_old: 14 # 14 to 16
19+
pg_version_new: 16
20+
- pg_version_old: 15 # 15 to 16
21+
pg_version_new: 16
22+
fail-fast: false
23+
env:
24+
OUTPUT_DIR: ${{ github.workspace }}/pg_upgrade_test
25+
26+
steps:
27+
- name: Install Linux Dependencies
28+
run: |
29+
sudo apt-get update
30+
sudo apt-get install pip postgresql-common libkrb5-dev
31+
32+
# Use custom testgres repo until upstream PR don't get merged and released
33+
# https://github.com/postgrespro/testgres/pull/125
34+
- name: Checkout testgres
35+
uses: actions/checkout@v4
36+
with:
37+
repository: 'fabriziomello/testgres'
38+
path: 'testgres'
39+
ref: 'add-options-pg-upgrade'
40+
41+
- name: Build and install testgres
42+
run: |
43+
cd testgres
44+
python setup.py install --user
45+
46+
- name: Install PostgreSQL ${{ matrix.pg_version_old}} and ${{ matrix.pg_version_new }}
47+
run: |
48+
yes | sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
49+
echo "deb https://packagecloud.io/timescale/timescaledb/ubuntu/ $(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/timescaledb.list
50+
wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo apt-key add -
51+
sudo apt-get update
52+
sudo apt-get install -y \
53+
postgresql-${{ matrix.pg_version_old }} postgresql-server-dev-${{ matrix.pg_version_old }} \
54+
postgresql-${{ matrix.pg_version_new }} postgresql-server-dev-${{ matrix.pg_version_new }}
55+
sudo apt-get install -y --no-install-recommends \
56+
timescaledb-2-postgresql-${{ matrix.pg_version_old }} \
57+
timescaledb-2-postgresql-${{ matrix.pg_version_new }}
58+
59+
- name: Checkout TimescaleDB
60+
uses: actions/checkout@v4
61+
62+
- name: Build and install TimescaleDB on PostgreSQL ${{ matrix.pg_version_old}}
63+
env:
64+
BUILD_DIR: pg${{ matrix.pg_version_old }}
65+
run: |
66+
PATH="/usr/lib/postgresql/${{ matrix.pg_version_old }}/bin:$PATH"
67+
./bootstrap -DCMAKE_BUILD_TYPE=Release -DWARNINGS_AS_ERRORS=OFF -DASSERTIONS=ON -DLINTER=OFF -DGENERATE_DOWNGRADE_SCRIPT=OFF -DREGRESS_CHECKS=OFF -DTAP_CHECKS=OFF
68+
make -j -C pg${{ matrix.pg_version_old }}
69+
sudo make -j -C pg${{ matrix.pg_version_old }} install
70+
71+
- name: Build and install TimescaleDB on PostgreSQL ${{ matrix.pg_version_new}}
72+
env:
73+
BUILD_DIR: pg${{ matrix.pg_version_new }}
74+
run: |
75+
PATH="/usr/lib/postgresql/${{ matrix.pg_version_new }}/bin:$PATH"
76+
./bootstrap -DCMAKE_BUILD_TYPE=Release -DWARNINGS_AS_ERRORS=OFF -DASSERTIONS=ON -DLINTER=OFF -DGENERATE_DOWNGRADE_SCRIPT=OFF -DREGRESS_CHECKS=OFF -DTAP_CHECKS=OFF
77+
make -j -C pg${{ matrix.pg_version_new }}
78+
sudo make -j -C pg${{ matrix.pg_version_new }} install
79+
80+
- name: Run pg_upgrade test
81+
env:
82+
PGVERSIONOLD: ${{ matrix.pg_version_old }}
83+
PGVERSIONNEW: ${{ matrix.pg_version_new }}
84+
DIFFFILE: ${{ env.OUTPUT_DIR }}/upgrade_check.diff
85+
run: |
86+
scripts/test_pg_upgrade.py
87+
diff -u \
88+
"${OUTPUT_DIR}/post.pg${PGVERSIONOLD}.log" \
89+
"${OUTPUT_DIR}/post.pg${PGVERSIONNEW}.log" | \
90+
tee "${DIFFFILE}"
91+
if [[ -s "${DIFFFILE}" ]]; then
92+
echo "pg_upgrade test for ${PGVERSIONOLD} -> ${PGVERSIONNEW} failed"
93+
exit 1
94+
fi
95+
96+
- name: Show pg_upgrade diffs
97+
if: always()
98+
env:
99+
DIFFFILE: ${{ env.OUTPUT_DIR }}/upgrade_check.diff
100+
DIROLD: pg${{ matrix.pg_version_old }}
101+
DIRNEW: pg${{ matrix.pg_version_new }}
102+
run: |
103+
cd ${OUTPUT_DIR}
104+
cat ${DIFFFILE}
105+
tar czf /tmp/pg_upgrade_artifacts.tgz \
106+
${DIFFFILE} \
107+
${OUTPUT_DIR}/*.log \
108+
${OUTPUT_DIR}/${DIROLD}/logs/* \
109+
${OUTPUT_DIR}/${DIRNEW}/logs/* \
110+
${OUTPUT_DIR}/${DIRNEW}/data/pg_upgrade_output.d/*
111+
112+
- name: Upload pg_upgrade logs
113+
if: always()
114+
uses: actions/upload-artifact@v4
115+
with:
116+
name: pg_upgrade logs from ${{ matrix.pg_version_old }} to ${{ matrix.pg_version_new }}
117+
path: /tmp/pg_upgrade_artifacts.tgz

‎scripts/test_pg_upgrade.py

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
#!/usr/bin/env python3
2+
3+
import os
4+
import sys
5+
6+
from shutil import rmtree
7+
from testgres import get_new_node, PostgresNode
8+
9+
10+
# Accessor functions
11+
def set_default_conf(node: PostgresNode, unix_socket_dir):
12+
node.default_conf(fsync=True, allow_streaming=False, allow_logical=False)
13+
node.append_conf(unix_socket_directories=unix_socket_dir)
14+
node.append_conf(timezone="GMT")
15+
node.append_conf(client_min_messages="warning")
16+
node.append_conf(max_prepared_transactions="100")
17+
node.append_conf(max_worker_processes="0")
18+
node.append_conf(shared_preload_libraries="timescaledb")
19+
node.append_conf("timescaledb.telemetry_level=off")
20+
21+
22+
def write_bytes_to_file(filename, nbytes):
23+
with open(filename, "wb") as f:
24+
f.write(nbytes)
25+
f.close()
26+
27+
28+
def getenv_or_error(envvar):
29+
return os.getenv(envvar) or sys.exit(
30+
"Environment variable {} not defined".format(envvar)
31+
)
32+
33+
34+
def getenv_or_default(envvar, default):
35+
return os.getenv(envvar) or default
36+
37+
38+
def initialize_node(working_dir, prefix, port, bin_dir, base_dir):
39+
node = get_new_node(prefix=prefix, port=port, bin_dir=bin_dir, base_dir=base_dir)
40+
node.init()
41+
set_default_conf(node, working_dir)
42+
return node
43+
44+
45+
# Globals
46+
pg_version_old = getenv_or_error("PGVERSIONOLD")
47+
pg_version_new = getenv_or_error("PGVERSIONNEW")
48+
49+
pg_node_old = "pg{}".format(pg_version_old)
50+
pg_node_new = "pg{}".format(pg_version_new)
51+
52+
pg_port_old = getenv_or_default("PGPORTOLD", "54321")
53+
pg_port_new = getenv_or_default("PGPORTNEW", "54322")
54+
55+
test_version = getenv_or_default("TEST_VERSION", "v8")
56+
57+
pg_bin_old = getenv_or_default(
58+
"PGBINOLD", "/usr/lib/postgresql/{}/bin".format(pg_version_old)
59+
)
60+
pg_bin_new = getenv_or_default(
61+
"PGBINNEW", "/usr/lib/postgresql/{}/bin".format(pg_version_new)
62+
)
63+
64+
working_dir = getenv_or_default(
65+
"OUTPUT_DIR",
66+
"/tmp/pg_upgrade_output/{}_to_{}".format(pg_version_old, pg_version_new),
67+
)
68+
69+
pg_data_old = getenv_or_default("PGDATAOLD", "{}/{}".format(working_dir, pg_node_old))
70+
pg_data_new = getenv_or_default("PGDATAOLD", "{}/{}".format(working_dir, pg_node_new))
71+
72+
pg_database_test = getenv_or_default("PGDATABASE", "pg_upgrade_test")
73+
74+
if os.path.exists(working_dir):
75+
rmtree(working_dir)
76+
os.makedirs(working_dir)
77+
78+
# Real testing code
79+
print("Initializing nodes {} and {}".format(pg_node_old, pg_node_new))
80+
node_old = initialize_node(
81+
working_dir, pg_node_old, pg_port_old, pg_bin_old, pg_data_old
82+
)
83+
node_old.start()
84+
85+
node_new = initialize_node(
86+
working_dir, pg_node_new, pg_port_new, pg_bin_new, pg_data_new
87+
)
88+
89+
print("Creating {} database on node {}".format(pg_database_test, pg_node_old))
90+
node_old.safe_psql(filename="test/sql/updates/setup.roles.sql")
91+
node_old.safe_psql(query="CREATE DATABASE {};".format(pg_database_test))
92+
node_old.safe_psql(dbname=pg_database_test, query="CREATE EXTENSION timescaledb;")
93+
node_old.safe_psql(dbname=pg_database_test, filename="test/sql/updates/pre.testing.sql")
94+
node_old.safe_psql(
95+
dbname=pg_database_test,
96+
filename="test/sql/updates/setup.{}.sql".format(test_version),
97+
)
98+
node_old.safe_psql(dbname=pg_database_test, query="CHECKPOINT")
99+
node_old.safe_psql(dbname=pg_database_test, filename="test/sql/updates/setup.check.sql")
100+
101+
# Run new psql over the old node to have the same psql output
102+
node_new.port = pg_port_old
103+
(code, old_out, old_err) = node_new.psql(
104+
dbname=pg_database_test, filename="test/sql/updates/post.pg_upgrade.sql"
105+
)
106+
node_new.port = pg_port_new
107+
108+
# Save output to log
109+
write_bytes_to_file("{}/post.{}.log".format(working_dir, pg_node_old), old_out)
110+
node_old.stop()
111+
112+
print("Upgrading node {} to {}".format(pg_node_old, pg_node_new))
113+
res = node_new.upgrade_from(old_node=node_old, options=["--retain"])
114+
node_new.start()
115+
116+
(code, new_out, new_err) = node_new.psql(
117+
dbname=pg_database_test,
118+
filename="test/sql/updates/post.pg_upgrade.sql".format(test_version),
119+
)
120+
121+
# Save output to log
122+
write_bytes_to_file("{}/post.pg{}.log".format(working_dir, pg_version_new), new_out)
123+
node_new.stop()

‎test/sql/updates/post.pg_upgrade.sql

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
-- This file and its contents are licensed under the Apache License 2.0.
2+
-- Please see the included NOTICE for copyright information and
3+
-- LICENSE-APACHE for a copy of the license.
4+
5+
\pset format aligned
6+
\pset tuples_only off
7+
8+
\ir post.catalog.sql
9+
\ir post.policies.sql
10+
\ir post.functions.sql

‎test/sql/updates/setup.v8.sql

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,7 @@
33
-- LICENSE-APACHE for a copy of the license.
44

55
\ir setup.v7.sql
6+
7+
8+
-- inject a failure just to test
9+
-- CREATE TABLE fail(c REGPROCEDURE);

0 commit comments

Comments
 (0)
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy