Skip to content

Commit a209184

Browse files
tglsfdctglsfdc
committed
Create common infrastructure for cross-version upgrade testing.
To test pg_upgrade across major PG versions, we have to be able to modify or drop any old objects with no-longer-supported properties, and we have to be able to deal with cosmetic changes in pg_dump output. Up to now, the buildfarm and pg_upgrade's own test infrastructure had separate implementations of the former, and we had nothing but very ad-hoc rules for the latter (including an arbitrary threshold on how many lines of unchecked diff were okay!). This patch creates a Perl module that can be shared by both those use-cases, and adds logic that deals with pg_dump output diffs in a much more tightly defined fashion. This largely supersedes previous efforts in commits 0df9641, 9814ff5, and 62be9e4, which developed a SQL-script-based solution for the task of dropping old objects. There was nothing fundamentally wrong with that work in itself, but it had no basis for solving the output-formatting problem. The most plausible way to deal with formatting is to build a Perl module that can perform editing on the dump files; and once we commit to that, it makes more sense for the same module to also embed the knowledge of what has to be done for dropping old objects. Back-patch versions of the helper module as far as 9.2, to support buildfarm animals that still test that far back. It's also necessary to back-patch PostgreSQL/Version.pm, because the new code depends on that. I fixed up pg_upgrade's 002_pg_upgrade.pl in v15, but did not look into back-patching it further than that. Tom Lane and Andrew Dunstan Discussion: https://postgr.es/m/891521.1673657296@sss.pgh.pa.us
1 parent c58c077 commit a209184

File tree

2 files changed

+382
-0
lines changed

2 files changed

+382
-0
lines changed

‎src/test/perl/PostgreSQL/Test/AdjustUpgrade.pm

Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
2+
# Copyright (c) 2023, PostgreSQL Global Development Group
3+
4+
=pod
5+
6+
=head1 NAME
7+
8+
PostgreSQL::Test::AdjustUpgrade - helper module for cross-version upgrade tests
9+
10+
=head1 SYNOPSIS
11+
12+
use PostgreSQL::Test::AdjustUpgrade;
13+
14+
# Build commands to adjust contents of old-version database before dumping
15+
$statements = adjust_database_contents($old_version, %dbnames);
16+
17+
# Adjust contents of old pg_dumpall output file to match newer version
18+
$dump = adjust_old_dumpfile($old_version, $dump);
19+
20+
# Adjust contents of new pg_dumpall output file to match older version
21+
$dump = adjust_new_dumpfile($old_version, $dump);
22+
23+
=head1 DESCRIPTION
24+
25+
C<PostgreSQL::Test::AdjustUpgrade> encapsulates various hacks needed to
26+
compare the results of cross-version upgrade tests.
27+
28+
=cut
29+
30+
package PostgreSQL::Test::AdjustUpgrade;
31+
32+
use strict;
33+
use warnings;
34+
35+
use Exporter 'import';
36+
use PostgreSQL::Version;
37+
38+
our @EXPORT = qw(
39+
adjust_database_contents
40+
adjust_old_dumpfile
41+
adjust_new_dumpfile
42+
);
43+
44+
=pod
45+
46+
=head1 ROUTINES
47+
48+
=over
49+
50+
=item $statements = adjust_database_contents($old_version, %dbnames)
51+
52+
Generate SQL commands to perform any changes to an old-version installation
53+
that are needed before we can pg_upgrade it into the current PostgreSQL
54+
version.
55+
56+
Typically this involves dropping or adjusting no-longer-supported objects.
57+
58+
Arguments:
59+
60+
=over
61+
62+
=item C<old_version>: Branch we are upgrading from, represented as a
63+
PostgreSQL::Version object.
64+
65+
=item C<dbnames>: Hash of database names present in the old installation.
66+
67+
=back
68+
69+
Returns a reference to a hash, wherein the keys are database names and the
70+
values are arrayrefs to lists of statements to be run in those databases.
71+
72+
=cut
73+
74+
sub adjust_database_contents
75+
{
76+
my ($old_version, %dbnames) = @_;
77+
my $result = {};
78+
79+
# remove dbs of modules known to cause pg_upgrade to fail
80+
# anything not builtin and incompatible should clean up its own db
81+
foreach my $bad_module ('test_ddl_deparse', 'tsearch2')
82+
{
83+
if ($dbnames{"contrib_regression_$bad_module"})
84+
{
85+
_add_st($result, 'postgres',
86+
"drop database contrib_regression_$bad_module");
87+
delete($dbnames{"contrib_regression_$bad_module"});
88+
}
89+
}
90+
91+
# avoid version number issues with test_ext7
92+
if ($dbnames{contrib_regression_test_extensions})
93+
{
94+
_add_st(
95+
$result,
96+
'contrib_regression_test_extensions',
97+
'drop extension if exists test_ext7');
98+
}
99+
100+
# get rid of dblink's dependencies on regress.so
101+
my $regrdb =
102+
$old_version le '9.4'
103+
? 'contrib_regression'
104+
: 'contrib_regression_dblink';
105+
106+
if ($dbnames{$regrdb})
107+
{
108+
_add_st(
109+
$result, $regrdb,
110+
'drop function if exists public.putenv(text)',
111+
'drop function if exists public.wait_pid(integer)');
112+
}
113+
114+
return $result;
115+
}
116+
117+
# Internal subroutine to add statement(s) to the list for the given db.
118+
sub _add_st
119+
{
120+
my ($result, $db, @st) = @_;
121+
122+
$result->{$db} ||= [];
123+
push(@{ $result->{$db} }, @st);
124+
}
125+
126+
=pod
127+
128+
=item adjust_old_dumpfile($old_version, $dump)
129+
130+
Edit a dump output file, taken from the adjusted old-version installation
131+
by current-version C<pg_dumpall -s>, so that it will match the results of
132+
C<pg_dumpall -s> on the pg_upgrade'd installation.
133+
134+
Typically this involves coping with cosmetic differences in the output
135+
of backend subroutines used by pg_dump.
136+
137+
Arguments:
138+
139+
=over
140+
141+
=item C<old_version>: Branch we are upgrading from, represented as a
142+
PostgreSQL::Version object.
143+
144+
=item C<dump>: Contents of dump file
145+
146+
=back
147+
148+
Returns the modified dump text.
149+
150+
=cut
151+
152+
sub adjust_old_dumpfile
153+
{
154+
my ($old_version, $dump) = @_;
155+
156+
# use Unix newlines
157+
$dump =~ s/\r\n/\n/g;
158+
159+
# Version comments will certainly not match.
160+
$dump =~ s/^-- Dumped from database version.*\n//mg;
161+
162+
# Suppress blank lines, as some places in pg_dump emit more or fewer.
163+
$dump =~ s/\n\n+/\n/g;
164+
165+
return $dump;
166+
}
167+
168+
=pod
169+
170+
=item adjust_new_dumpfile($old_version, $dump)
171+
172+
Edit a dump output file, taken from the pg_upgrade'd installation
173+
by current-version C<pg_dumpall -s>, so that it will match the old
174+
dump output file as adjusted by C<adjust_old_dumpfile>.
175+
176+
Typically this involves deleting data not present in the old installation.
177+
178+
Arguments:
179+
180+
=over
181+
182+
=item C<old_version>: Branch we are upgrading from, represented as a
183+
PostgreSQL::Version object.
184+
185+
=item C<dump>: Contents of dump file
186+
187+
=back
188+
189+
Returns the modified dump text.
190+
191+
=cut
192+
193+
sub adjust_new_dumpfile
194+
{
195+
my ($old_version, $dump) = @_;
196+
197+
# use Unix newlines
198+
$dump =~ s/\r\n/\n/g;
199+
200+
# Version comments will certainly not match.
201+
$dump =~ s/^-- Dumped from database version.*\n//mg;
202+
203+
# Suppress blank lines, as some places in pg_dump emit more or fewer.
204+
$dump =~ s/\n\n+/\n/g;
205+
206+
return $dump;
207+
}
208+
209+
=pod
210+
211+
=back
212+
213+
=cut
214+
215+
1;

‎src/test/perl/PostgreSQL/Version.pm

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
############################################################################
2+
#
3+
# PostgreSQL/Version.pm
4+
#
5+
# Module encapsulating Postgres Version numbers
6+
#
7+
# Copyright (c) 2021-2022, PostgreSQL Global Development Group
8+
#
9+
############################################################################
10+
11+
=pod
12+
13+
=head1 NAME
14+
15+
PostgreSQL::Version - class representing PostgreSQL version numbers
16+
17+
=head1 SYNOPSIS
18+
19+
use PostgreSQL::Version;
20+
21+
my $version = PostgreSQL::Version->new($version_arg);
22+
23+
# compare two versions
24+
my $bool = $version1 <= $version2;
25+
26+
# or compare with a number
27+
$bool = $version < 12;
28+
29+
# or with a string
30+
$bool = $version lt "13.1";
31+
32+
# interpolate in a string
33+
my $stringyval = "version: $version";
34+
35+
# get the major version
36+
my $maj = $version->major;
37+
38+
=head1 DESCRIPTION
39+
40+
PostgreSQL::Version encapsulates Postgres version numbers, providing parsing
41+
of common version formats and comparison operations.
42+
43+
=cut
44+
45+
package PostgreSQL::Version;
46+
47+
use strict;
48+
use warnings;
49+
50+
use Scalar::Util qw(blessed);
51+
52+
use overload
53+
'<=>' => \&_version_cmp,
54+
'cmp' => \&_version_cmp,
55+
'""' => \&_stringify;
56+
57+
=pod
58+
59+
=head1 METHODS
60+
61+
=over
62+
63+
=item PostgreSQL::Version->new($version)
64+
65+
Create a new PostgreSQL::Version instance.
66+
67+
The argument can be a number like 12, or a string like '12.2' or the output
68+
of a Postgres command like `psql --version` or `pg_config --version`;
69+
70+
=back
71+
72+
=cut
73+
74+
sub new
75+
{
76+
my $class = shift;
77+
my $arg = shift;
78+
79+
chomp $arg;
80+
81+
# Accept standard formats, in case caller has handed us the output of a
82+
# postgres command line tool
83+
my $devel;
84+
($arg, $devel) = ($1, $2)
85+
if (
86+
$arg =~ m!^ # beginning of line
87+
(?:\(?PostgreSQL\)?\s)? # ignore PostgreSQL marker
88+
(\d+(?:\.\d+)*) # version number, dotted notation
89+
(devel|(?:alpha|beta|rc)\d+)? # dev marker - see version_stamp.pl
90+
!x);
91+
92+
# Split into an array
93+
my @numbers = split(/\./, $arg);
94+
95+
# Treat development versions as having a minor/micro version one less than
96+
# the first released version of that branch.
97+
push @numbers, -1 if ($devel);
98+
99+
$devel ||= "";
100+
101+
return bless { str => "$arg$devel", num => \@numbers }, $class;
102+
}
103+
104+
# Routine which compares the _pg_version_array obtained for the two
105+
# arguments and returns -1, 0, or 1, allowing comparison between two
106+
# PostgreSQL::Version objects or a PostgreSQL::Version and a version string or number.
107+
#
108+
# If the second argument is not a blessed object we call the constructor
109+
# to make one.
110+
#
111+
# Because we're overloading '<=>' and 'cmp' this function supplies us with
112+
# all the comparison operators ('<' and friends, 'gt' and friends)
113+
#
114+
sub _version_cmp
115+
{
116+
my ($a, $b, $swapped) = @_;
117+
118+
$b = __PACKAGE__->new($b) unless blessed($b);
119+
120+
($a, $b) = ($b, $a) if $swapped;
121+
122+
my ($an, $bn) = ($a->{num}, $b->{num});
123+
124+
for (my $idx = 0;; $idx++)
125+
{
126+
return 0
127+
if ($idx >= @$an && $idx >= @$bn);
128+
# treat a missing number as 0
129+
my ($anum, $bnum) = ($an->[$idx] || 0, $bn->[$idx] || 0);
130+
return $anum <=> $bnum
131+
if ($anum <=> $bnum);
132+
}
133+
}
134+
135+
# Render the version number using the saved string.
136+
sub _stringify
137+
{
138+
my $self = shift;
139+
return $self->{str};
140+
}
141+
142+
=pod
143+
144+
=over
145+
146+
=item major([separator => 'char'])
147+
148+
Returns the major version. For versions before 10 the parts are separated by
149+
a dot unless the separator argument is given.
150+
151+
=back
152+
153+
=cut
154+
155+
sub major
156+
{
157+
my ($self, %params) = @_;
158+
my $result = $self->{num}->[0];
159+
if ($result + 0 < 10)
160+
{
161+
my $sep = $params{separator} || '.';
162+
$result .= "$sep$self->{num}->[1]";
163+
}
164+
return $result;
165+
}
166+
167+
1;

0 commit comments

Comments
 (0)
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy