Skip to content

Commit a9448f4

Browse files
fabriziomellofabriziomello
committed
Add pg_upgrade test to CI
It's long we need to have binary upgrade tests in our CI. One example is timescale#6935 (and others) that can be prevented if we have such kind of test in our CI. To implement the `pg_upgrade` test we used the python Testing Framework for PostgreSQL (https://github.com/postgrespro/testgres). Unfortunately the testing framework don't have the ability to retain the pg_upgrade log files after a successful execution, then we created a PR to make it possible and we'll use this patched version until we get the code merged and released on upstream. postgrespro/testgres#125 Closes timescale#3868 timescale#4428
1 parent f5d466c commit a9448f4

File tree

4 files changed

+260
-0
lines changed

4 files changed

+260
-0
lines changed

‎.github/workflows/code_style.yaml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,8 +107,24 @@ jobs:
107107
psutil pygithub pglast
108108
pip list
109109
pip list --user
110+
111+
# Use custom testgres repo until upstream PR don't get merged and released
112+
# https://github.com/postgrespro/testgres/pull/125
113+
- name: Checkout testgres
114+
uses: actions/checkout@v4
115+
with:
116+
repository: 'fabriziomello/testgres'
117+
path: 'testgres'
118+
ref: 'add-options-pg-upgrade'
119+
120+
- name: Build and install testgres
121+
run: |
122+
cd testgres
123+
python setup.py install --user
124+
110125
- name: Checkout source
111126
uses: actions/checkout@v4
127+
112128
- name: Run prospector
113129
run: |
114130
find . -type f -name "*.py" -print -exec prospector {} + -exec black {} +

‎.github/workflows/pg_upgrade-test.yaml

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
name: pg_upgrade test
2+
"on":
3+
push:
4+
branches:
5+
- main
6+
- prerelease_test
7+
pull_request:
8+
9+
jobs:
10+
pg_upgrade_test:
11+
name: pg_upgrade test from PG${{ matrix.pg_version_old }} to PG${{ matrix.pg_version_new }}
12+
runs-on: 'ubuntu-latest'
13+
strategy:
14+
matrix:
15+
include:
16+
- pg_version_old: 14 # 14 to 15
17+
pg_version_new: 15
18+
- pg_version_old: 14 # 14 to 16
19+
pg_version_new: 16
20+
- pg_version_old: 15 # 15 to 16
21+
pg_version_new: 16
22+
fail-fast: false
23+
env:
24+
OUTPUT_DIR: ${{ github.workspace }}/pg_upgrade_test
25+
26+
steps:
27+
- name: Install Linux Dependencies
28+
run: |
29+
sudo apt-get update
30+
sudo apt-get install pip postgresql-common libkrb5-dev
31+
32+
# Use custom testgres repo until upstream PR don't get merged and released
33+
# https://github.com/postgrespro/testgres/pull/125
34+
- name: Checkout testgres
35+
uses: actions/checkout@v4
36+
with:
37+
repository: 'fabriziomello/testgres'
38+
path: 'testgres'
39+
ref: 'add-options-pg-upgrade'
40+
41+
- name: Build and install testgres
42+
run: |
43+
cd testgres
44+
python setup.py install --user
45+
46+
- name: Install PostgreSQL ${{ matrix.pg_version_old}} and ${{ matrix.pg_version_new }}
47+
run: |
48+
yes | sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
49+
echo "deb https://packagecloud.io/timescale/timescaledb/ubuntu/ $(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/timescaledb.list
50+
wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo apt-key add -
51+
sudo apt-get update
52+
sudo apt-get install -y \
53+
postgresql-${{ matrix.pg_version_old }} postgresql-server-dev-${{ matrix.pg_version_old }} \
54+
postgresql-${{ matrix.pg_version_new }} postgresql-server-dev-${{ matrix.pg_version_new }}
55+
sudo apt-get install -y --no-install-recommends \
56+
timescaledb-2-postgresql-${{ matrix.pg_version_old }} \
57+
timescaledb-2-postgresql-${{ matrix.pg_version_new }}
58+
59+
- name: Checkout TimescaleDB
60+
uses: actions/checkout@v4
61+
62+
- name: Build and install TimescaleDB on PostgreSQL ${{ matrix.pg_version_old}}
63+
env:
64+
BUILD_DIR: pg${{ matrix.pg_version_old }}
65+
run: |
66+
PATH="/usr/lib/postgresql/${{ matrix.pg_version_old }}/bin:$PATH"
67+
./bootstrap -DCMAKE_BUILD_TYPE=Release -DWARNINGS_AS_ERRORS=OFF -DASSERTIONS=ON -DLINTER=OFF -DGENERATE_DOWNGRADE_SCRIPT=OFF -DREGRESS_CHECKS=OFF -DTAP_CHECKS=OFF
68+
make -j -C pg${{ matrix.pg_version_old }}
69+
sudo make -j -C pg${{ matrix.pg_version_old }} install
70+
71+
- name: Build and install TimescaleDB on PostgreSQL ${{ matrix.pg_version_new}}
72+
env:
73+
BUILD_DIR: pg${{ matrix.pg_version_new }}
74+
run: |
75+
PATH="/usr/lib/postgresql/${{ matrix.pg_version_new }}/bin:$PATH"
76+
./bootstrap -DCMAKE_BUILD_TYPE=Release -DWARNINGS_AS_ERRORS=OFF -DASSERTIONS=ON -DLINTER=OFF -DGENERATE_DOWNGRADE_SCRIPT=OFF -DREGRESS_CHECKS=OFF -DTAP_CHECKS=OFF
77+
make -j -C pg${{ matrix.pg_version_new }}
78+
sudo make -j -C pg${{ matrix.pg_version_new }} install
79+
80+
- name: Run pg_upgrade test
81+
env:
82+
PGVERSIONOLD: ${{ matrix.pg_version_old }}
83+
PGVERSIONNEW: ${{ matrix.pg_version_new }}
84+
DIFFFILE: ${{ env.OUTPUT_DIR }}/upgrade_check.diff
85+
run: |
86+
scripts/test_pg_upgrade.py
87+
diff -u \
88+
"${OUTPUT_DIR}/post.pg${PGVERSIONOLD}.log" \
89+
"${OUTPUT_DIR}/post.pg${PGVERSIONNEW}.log" | \
90+
tee "${DIFFFILE}"
91+
if [[ -s "${DIFFFILE}" ]]; then
92+
echo "pg_upgrade test for ${PGVERSIONOLD} -> ${PGVERSIONNEW} failed"
93+
exit 1
94+
fi
95+
96+
- name: Show pg_upgrade diffs
97+
if: always()
98+
env:
99+
DIFFFILE: ${{ env.OUTPUT_DIR }}/upgrade_check.diff
100+
DIROLD: pg${{ matrix.pg_version_old }}
101+
DIRNEW: pg${{ matrix.pg_version_new }}
102+
run: |
103+
cd ${OUTPUT_DIR}
104+
cat ${DIFFFILE}
105+
tar czf /tmp/pg_upgrade_artifacts.tgz \
106+
${DIFFFILE} \
107+
${OUTPUT_DIR}/*.log \
108+
${OUTPUT_DIR}/${DIROLD}/logs/* \
109+
${OUTPUT_DIR}/${DIRNEW}/logs/* \
110+
${OUTPUT_DIR}/${DIRNEW}/data/pg_upgrade_output.d/*
111+
112+
- name: Upload pg_upgrade logs
113+
if: always()
114+
uses: actions/upload-artifact@v4
115+
with:
116+
name: pg_upgrade logs from ${{ matrix.pg_version_old }} to ${{ matrix.pg_version_new }}
117+
path: /tmp/pg_upgrade_artifacts.tgz

‎scripts/test_pg_upgrade.py

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
#!/usr/bin/env python3
2+
3+
import os
4+
import sys
5+
6+
from shutil import rmtree
7+
from testgres import get_new_node, PostgresNode
8+
9+
10+
# Accessor functions
11+
def set_default_conf(node: PostgresNode, unix_socket_dir):
12+
node.default_conf(fsync=True, allow_streaming=False, allow_logical=False)
13+
node.append_conf(unix_socket_directories=unix_socket_dir)
14+
node.append_conf(timezone="GMT")
15+
node.append_conf(client_min_messages="warning")
16+
node.append_conf(max_prepared_transactions="100")
17+
node.append_conf(max_worker_processes="0")
18+
node.append_conf(shared_preload_libraries="timescaledb")
19+
node.append_conf("timescaledb.telemetry_level=off")
20+
21+
22+
def write_bytes_to_file(filename, nbytes):
23+
with open(filename, "wb") as f:
24+
f.write(nbytes)
25+
f.close()
26+
27+
28+
def getenv_or_error(envvar):
29+
return os.getenv(envvar) or sys.exit(f"Environment variable {envvar} not defined")
30+
31+
32+
def getenv_or_default(envvar, default):
33+
return os.getenv(envvar) or default
34+
35+
36+
def initialize_node(working_dir, prefix, port, bin_dir, base_dir):
37+
node = get_new_node(prefix=prefix, port=port, bin_dir=bin_dir, base_dir=base_dir)
38+
node.init()
39+
set_default_conf(node, working_dir)
40+
return node
41+
42+
43+
# Globals
44+
pg_version_old = getenv_or_error("PGVERSIONOLD")
45+
pg_version_new = getenv_or_error("PGVERSIONNEW")
46+
47+
pg_node_old = f"pg{pg_version_old}"
48+
pg_node_new = f"pg{pg_version_new}"
49+
50+
pg_port_old = getenv_or_default("PGPORTOLD", "54321")
51+
pg_port_new = getenv_or_default("PGPORTNEW", "54322")
52+
53+
test_version = getenv_or_default("TEST_VERSION", "v8")
54+
55+
pg_bin_old = getenv_or_default("PGBINOLD", f"/usr/lib/postgresql/{pg_version_old}/bin")
56+
pg_bin_new = getenv_or_default("PGBINNEW", f"/usr/lib/postgresql/{pg_version_new}/bin")
57+
58+
working_dir = getenv_or_default(
59+
"OUTPUT_DIR",
60+
f"/tmp/pg_upgrade_output/{pg_version_old}_to_{pg_version_new}",
61+
)
62+
63+
pg_data_old = getenv_or_default("PGDATAOLD", f"{working_dir}/{pg_node_old}")
64+
pg_data_new = getenv_or_default("PGDATAOLD", f"{working_dir}/{pg_node_new}")
65+
66+
pg_database_test = getenv_or_default("PGDATABASE", "pg_upgrade_test")
67+
68+
if os.path.exists(working_dir):
69+
rmtree(working_dir)
70+
os.makedirs(working_dir)
71+
72+
# Real testing code
73+
print(f"Initializing nodes {pg_node_old} and {pg_node_new}")
74+
node_old = initialize_node(
75+
working_dir, pg_node_old, pg_port_old, pg_bin_old, pg_data_old
76+
)
77+
node_old.start()
78+
79+
node_new = initialize_node(
80+
working_dir, pg_node_new, pg_port_new, pg_bin_new, pg_data_new
81+
)
82+
83+
print(f"Creating {pg_database_test} database on node {pg_node_old}")
84+
node_old.safe_psql(filename="test/sql/updates/setup.roles.sql")
85+
node_old.safe_psql(query=f"CREATE DATABASE {pg_database_test};")
86+
node_old.safe_psql(dbname=pg_database_test, query="CREATE EXTENSION timescaledb;")
87+
node_old.safe_psql(dbname=pg_database_test, filename="test/sql/updates/pre.testing.sql")
88+
node_old.safe_psql(
89+
dbname=pg_database_test,
90+
filename=f"test/sql/updates/setup.{test_version}.sql",
91+
)
92+
node_old.safe_psql(dbname=pg_database_test, query="CHECKPOINT")
93+
node_old.safe_psql(dbname=pg_database_test, filename="test/sql/updates/setup.check.sql")
94+
95+
# Run new psql over the old node to have the same psql output
96+
node_new.port = pg_port_old
97+
(code, old_out, old_err) = node_new.psql(
98+
dbname=pg_database_test, filename="test/sql/updates/post.pg_upgrade.sql"
99+
)
100+
node_new.port = pg_port_new
101+
102+
# Save output to log
103+
write_bytes_to_file(f"{working_dir}/post.{pg_node_old}.log", old_out)
104+
node_old.stop()
105+
106+
print(f"Upgrading node {pg_node_old} to {pg_node_new}")
107+
res = node_new.upgrade_from(old_node=node_old, options=["--retain"])
108+
node_new.start()
109+
110+
(code, new_out, new_err) = node_new.psql(
111+
dbname=pg_database_test,
112+
filename="test/sql/updates/post.pg_upgrade.sql",
113+
)
114+
115+
# Save output to log
116+
write_bytes_to_file(f"{working_dir}/post.pg{pg_version_new}.log", new_out)
117+
node_new.stop()

‎test/sql/updates/post.pg_upgrade.sql

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
-- This file and its contents are licensed under the Apache License 2.0.
2+
-- Please see the included NOTICE for copyright information and
3+
-- LICENSE-APACHE for a copy of the license.
4+
5+
\pset format aligned
6+
\pset tuples_only off
7+
8+
\ir post.catalog.sql
9+
\ir post.policies.sql
10+
\ir post.functions.sql

0 commit comments

Comments
 (0)
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy