diff --git a/README.md b/README.md index 8abb7164..fb7f20e0 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@
-
Deutsch | Português (BR) | Русский | Español | Українська +
Deutsch | Português (BR) | Русский | Español | Українська

@@ -16,7 +16,7 @@
:zap: Blazing-fast cloning of PostgreSQL databases :elephant:
Thin clones of PostgreSQL to build powerful development, test, QA, and staging environments.
- Available for any PostgreSQL, including AWS RDS, GCP CloudSQL, Heroku, Digital Ocean, and self-managed instances. + Available for any PostgreSQL, including AWS RDS*, GCP CloudSQL*, Heroku*, Digital Ocean*, and self-managed instances.

@@ -43,6 +43,9 @@ +--- + * For a managed PostgreSQL cloud service such as AWS RDS or Heroku, where physical connection and access to PGDATA are not available, DLE is supposed to be running on a separate VM in the same region, performing periodical automated full refresh of data and serving itself as a database-as-a-service solution providing thin database clones for development and testing purposes. + ## Why DLE? - Build dev/QA/staging environments based on full-size production-like databases. - Provide temporary full-size database clones for SQL query analysis and optimization (see also: [SQL optimization chatbot Joe](https://gitlab.com/postgres-ai/joe)). diff --git a/cloudformation/dle_cf_template.yaml b/cloudformation/dle_cf_template.yaml index 30d4ef7f..6b53e154 100644 --- a/cloudformation/dle_cf_template.yaml +++ b/cloudformation/dle_cf_template.yaml @@ -108,7 +108,7 @@ Parameters: InstanceType: Description: DLE EC2 instance type Type: String - Default: m5.4xlarge + Default: m5.2xlarge AllowedValues: - r5.large - r5.xlarge @@ -126,13 +126,6 @@ Parameters: - m5.12xlarge - m5.16xlarge - m5.24xlarge - - t3.nano - - t3.micro - - t3.small - - t3.medium - - t3.large - - t3.xlarge - - t3.2xlarge ConstraintDescription: must be a valid EC2 instance type. SSHLocation: Description: CIDR in format x.x.x.x/32 to allow one specific IP address access, 0.0.0.0/0 to allow all IP addresses access, or another CIDR range @@ -253,58 +246,46 @@ Mappings: Arch: HVM64 m5.24xlarge: Arch: HVM64 - t3.nano: - Arch: HVM64 - t3.micro: - Arch: HVM64 - t3.small: - Arch: HVM64 - t3.medium: - Arch: HVM64 - t3.large: - Arch: HVM64 - t3.xlarge: - Arch: HVM64 - t3.2xlarge: - Arch: HVM64 AWSRegionArch2AMI: eu-north-1: - HVM64: ami-0665ae2cfbd4e342d + HVM64: ami-0888261a1eacb636e ap-south-1: - HVM64: ami-0e374efc30e300f09 + HVM64: ami-00539bfa7a6926e1b eu-west-3: - HVM64: ami-0efda6ea87e5c4d96 + HVM64: ami-038d1f1d1ef71112b eu-west-2: - HVM64: ami-0687cbc11ebc16691 + HVM64: ami-07c2bca027887871b eu-west-1: - HVM64: ami-0d50368f3e8f1ccc0 + HVM64: ami-0e38f0f4f0acd49c2 ap-northeast-3: - HVM64: ami-0e65633c1b72de22f + HVM64: ami-01cd2976ef1688c25 ap-northeast-2: - HVM64: ami-02f4e02a76c68579d + HVM64: ami-049c608703690f99e ap-northeast-1: - HVM64: ami-04603eedf1f55b4cb + HVM64: ami-0cb59515cd67fdc93 sa-east-1: - HVM64: ami-05267d11294fbeb12 + HVM64: ami-0b3aeaa58412025de ca-central-1: - HVM64: ami-0504c9f745022749a + HVM64: ami-075d0aae6fdd356b1 ap-southeast-1: - HVM64: ami-0fdf327ea5e077df4 + HVM64: ami-054e735ba76985f92 ap-southeast-2: - HVM64: ami-01e5c77c1fbc46669 + HVM64: ami-06558ef4fedcf3c2f eu-central-1: - HVM64: ami-0793f98b004f79c42 + HVM64: ami-048a27a74e4c1239d us-east-1: - HVM64: ami-07ed8ca1867e9803a + HVM64: ami-0ed40b8023c788775 us-east-2: - HVM64: ami-042693a1c63d12800 + HVM64: ami-0d6a0bd053962b66f us-west-1: - HVM64: ami-0484ba45ecb22a99e + HVM64: ami-0ef7453c037b624ec us-west-2: - HVM64: ami-04859f68862a8bcfd + HVM64: ami-0bdf048f8e10f02eb Conditions: CreateSubDomain: !Not [!Equals [!Ref CertificateHostedZone, '']] + NotCreateSubDomain: + !Not [Condition: CreateSubDomain] Resources: LambdaExecutionRole: @@ -446,7 +427,7 @@ Resources: --volume $postgres_conf_path:/home/dblab/standard/postgres/control \ --env DOCKER_API_VERSION=1.39 \ --restart always \ - registry.gitlab.com/postgres-ai/database-lab/dblab-server:3.1.0 + registry.gitlab.com/postgres-ai/database-lab/dblab-server:3.1.1 if [ ! -z "${CertificateHostedZone}" ]; then export DOMAIN=${CertificateSubdomain}.${CertificateHostedZone} @@ -470,6 +451,18 @@ Resources: sudo systemctl enable envoy sudo systemctl start envoy fi + + while ! echo "UI started" | nc localhost 2346; do sleep 10; done + /opt/aws/bin/cfn-signal -e $? -d "DLE UI is available" -r "DLE Deploy Process Complete" '${WaitHandle}' + + WaitHandle: + Type: AWS::CloudFormation::WaitConditionHandle + WaitCondition: + Type: AWS::CloudFormation::WaitCondition + DependsOn: DLEInstance + Properties: + Handle: !Ref 'WaitHandle' + Timeout: '600' MountPoint: Type: AWS::EC2::VolumeAttachment @@ -539,49 +532,52 @@ Resources: VpcId: !Ref VPC Outputs: - VerificationToken: + 02VerificationToken: Description: 'DLE verification token' Value: !Ref DLEVerificationToken - DLE: + 08DLEInstance: Description: URL for newly created DLE instance Value: !Sub 'https://${CertificateSubdomain}.${CertificateHostedZone}' Condition: CreateSubDomain - UI: + 01WebUIUrl: Description: UI URL with a domain for newly created DLE instance Value: !Sub 'https://${CertificateSubdomain}.${CertificateHostedZone}:446' Condition: CreateSubDomain + 01WebUIUrl: + Description: UI URL with a domain for newly created DLE instance + Value: !Sub 'http://localhost:2346' + Condition: NotCreateSubDomain - EBSVolume: + 07EBSVolumeSize: Description: Size of provisioned EBS volume Value: !GetAtt SizeCalculate.Value - DNSName: + 03DNSName: Description: Public DNS name Value: !GetAtt DLEInstance.PublicDnsName - EC2SSH: + 06EC2SSH: Description: SSH connection to the EC2 instance with Database Lab Engine Value: !Sub - - 'ssh -i YOUR_PRIVATE_KEY ubuntu@${DNSName}' + - 'ssh ubuntu@${DNSName} -i YOUR_PRIVATE_KEY' - DNSName: !GetAtt DLEInstance.PublicDnsName - DLETunnel: + 05DLETunnel: Description: Create an SSH-tunnel to Database Lab Engine Value: !Sub - - 'ssh -N -L 2345:${DNSName}:2345 -i YOUR_PRIVATE_KEY ubuntu@${DNSName}' + - 'ssh -N -L 2345:${DNSName}:2345 ubuntu@${DNSName} -i YOUR_PRIVATE_KEY' - DNSName: !GetAtt DLEInstance.PublicDnsName - UITunnel: - Description: Create an SSH-tunnel to Database Lab UI + 00UITunnel: + Description: Use SSH port forwarding to be able to access DLE UI Value: !Sub - - 'ssh -N -L 2346:${DNSName}:2346 -i YOUR_PRIVATE_KEY ubuntu@${DNSName}' + - 'ssh -N -L 2346:${DNSName}:2346 ubuntu@${DNSName} -i YOUR_PRIVATE_KEY' - DNSName: !GetAtt DLEInstance.PublicDnsName - CloneTunnel: - Description: Create an SSH-tunnel to Database Lab clones + 04CloneTunnel: + Description: Use SSH port forwarding to be able to access a database clone Value: !Sub - - 'ssh -N -L CLONE_PORT:${DNSName}:CLONE_PORT -i YOUR_PRIVATE_KEY ubuntu@${DNSName}' + - 'ssh -N -L CLONE_PORT:${DNSName}:CLONE_PORT ubuntu@${DNSName} -i YOUR_PRIVATE_KEY' - DNSName: !GetAtt DLEInstance.PublicDnsName - diff --git a/engine/Dockerfile.dblab-server-debug b/engine/Dockerfile.dblab-server-debug new file mode 100644 index 00000000..84d31ef4 --- /dev/null +++ b/engine/Dockerfile.dblab-server-debug @@ -0,0 +1,42 @@ +# How to start a container: https://postgres.ai/docs/how-to-guides/administration/engine-manage + +# Compile stage +FROM golang:1.18 AS build-env + +# Build Delve +RUN go install github.com/go-delve/delve/cmd/dlv@latest + +# Build DLE (Uncomment if the binary doesn't compile locally). +# ADD . /dockerdev +# WORKDIR /dockerdev +# RUN GO111MODULE=on CGO_ENABLED=0 go build -gcflags="all=-N -l" -o /dblab-server-debug ./cmd/database-lab/main.go + +# Final stage +FROM docker:20.10.12 + +# Install dependencies +RUN apk update \ + && apk add zfs=2.1.4-r0 --no-cache --repository=https://dl-cdn.alpinelinux.org/alpine/edge/main \ + && apk add --no-cache lvm2 bash util-linux +RUN echo 'http://dl-cdn.alpinelinux.org/alpine/v3.13/main' >> /etc/apk/repositories \ + && echo 'http://dl-cdn.alpinelinux.org/alpine/v3.13/community' >> /etc/apk/repositories \ + && apk add bcc-tools=0.18.0-r0 bcc-doc=0.18.0-r0 && ln -s $(which python3) /usr/bin/python \ + # TODO: remove after release the PR: https://github.com/iovisor/bcc/pull/3286 (issue: https://github.com/iovisor/bcc/issues/3099) + && wget https://raw.githubusercontent.com/iovisor/bcc/master/tools/biosnoop.py -O /usr/share/bcc/tools/biosnoop + +ENV PATH="${PATH}:/usr/share/bcc/tools" + +WORKDIR /home/dblab + +EXPOSE 2345 40000 + +# Replace if the binary doesn't compile locally. +# COPY --from=build-env /dblab-server-debug ./bin/dblab-server-debug +COPY ./bin/dblab-server-debug ./bin/dblab-server-debug + +COPY --from=build-env /go/bin/dlv ./ +COPY ./configs/standard ./standard +COPY ./api ./api +COPY ./scripts ./scripts + +CMD ["./dlv", "--listen=:40000", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "./bin/dblab-server-debug"] diff --git a/engine/Makefile b/engine/Makefile index 50de429c..e76f7538 100644 --- a/engine/Makefile +++ b/engine/Makefile @@ -44,6 +44,11 @@ build: ## Build binary files of all Database Lab components (Engine, CI Checker, ${GOBUILD} -o bin/${RUN_CI_BINARY} ./cmd/runci/main.go ${GOBUILD} -o bin/${CLI_BINARY} ./cmd/cli/main.go +build-debug: ## Build the Database Lab Server binary for debugging + ${GOBUILD} -ldflags "-X gitlab.com/postgres-ai/database-lab/v3/version.version=${VERSION} \ + -X gitlab.com/postgres-ai/database-lab/v3/version.buildTime=${BUILD_TIME}" \ + -gcflags="all=-N -l" -o bin/${SERVER_BINARY}-debug ./cmd/database-lab/main.go + build-ci-checker: ## Build the Database Lab CI Checker binary ${GOBUILD} -o bin/${RUN_CI_BINARY} ./cmd/runci/main.go diff --git a/engine/cmd/database-lab/main.go b/engine/cmd/database-lab/main.go index adfd00a5..ed8bac56 100644 --- a/engine/cmd/database-lab/main.go +++ b/engine/cmd/database-lab/main.go @@ -126,10 +126,7 @@ func main() { emergencyShutdown := func() { cancel() - shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), shutdownTimeout) - defer shutdownCancel() - - shutdownDatabaseLabEngine(shutdownCtx, docker, engProps, pm.First()) + shutdownDatabaseLabEngine(context.Background(), docker, &cfg.Global.Database, engProps.InstanceID, pm.First()) } cloningSvc := cloning.NewBase(&cfg.Cloning, provisioner, tm, observingChan) @@ -185,16 +182,18 @@ func main() { <-shutdownCh cancel() - shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), shutdownTimeout) + ctxBackground := context.Background() + + shutdownCtx, shutdownCancel := context.WithTimeout(ctxBackground, shutdownTimeout) defer shutdownCancel() if err := server.Shutdown(shutdownCtx); err != nil { log.Msg(err) } - shutdownDatabaseLabEngine(shutdownCtx, docker, engProps, pm.First()) + shutdownDatabaseLabEngine(ctxBackground, docker, &cfg.Global.Database, engProps.InstanceID, pm.First()) cloningSvc.SaveClonesState() - tm.SendEvent(context.Background(), telemetry.EngineStoppedEvent, telemetry.EngineStopped{Uptime: server.Uptime()}) + tm.SendEvent(ctxBackground, telemetry.EngineStoppedEvent, telemetry.EngineStopped{Uptime: server.Uptime()}) } func getEngineProperties(ctx context.Context, dockerCLI *client.Client, cfg *config.Config) (global.EngineProps, error) { @@ -291,16 +290,14 @@ func setShutdownListener() chan os.Signal { return c } -func shutdownDatabaseLabEngine(ctx context.Context, dockerCLI *client.Client, engProps global.EngineProps, fsm pool.FSManager) { +func shutdownDatabaseLabEngine(ctx context.Context, docker *client.Client, dbCfg *global.Database, instanceID string, fsm pool.FSManager) { log.Msg("Stopping auxiliary containers") - if fsm != nil { - if err := cont.StopControlContainers(ctx, dockerCLI, engProps.InstanceID, fsm.Pool().DataDir()); err != nil { - log.Err("Failed to stop control containers", err) - } + if err := cont.StopControlContainers(ctx, docker, dbCfg, instanceID, fsm); err != nil { + log.Err("Failed to stop control containers", err) } - if err := cont.CleanUpSatelliteContainers(ctx, dockerCLI, engProps.InstanceID); err != nil { + if err := cont.CleanUpSatelliteContainers(ctx, docker, instanceID); err != nil { log.Err("Failed to stop satellite containers", err) } diff --git a/engine/cmd/runci/main.go b/engine/cmd/runci/main.go index 53ba3d49..9f089cdd 100644 --- a/engine/cmd/runci/main.go +++ b/engine/cmd/runci/main.go @@ -81,6 +81,18 @@ func main() { return } + if err := os.MkdirAll(source.RepoDir, 0666); err != nil { + log.Errf("failed to create a directory to download archives: %v", err) + return + } + + defer func() { + if err := os.RemoveAll(source.RepoDir); err != nil { + log.Errf("failed to remove the directory with source code archives: %v", err) + return + } + }() + codeProvider := source.NewCodeProvider(ctx, &cfg.Source) srv := runci.NewServer(cfg, dleClient, platformSvc, codeProvider, dockerCLI, networkID) @@ -114,7 +126,7 @@ func discoverNetwork(ctx context.Context, cfg *runci.Config, dockerCLI *client.C networkID := "" for networkLabel, endpointSettings := range inspection.NetworkSettings.Networks { - if strings.HasPrefix(networkLabel, "network_") { + if strings.HasPrefix(networkLabel, networks.NetworkPrefix) { networkResource, err := dockerCLI.NetworkInspect(ctx, endpointSettings.NetworkID, types.NetworkInspectOptions{}) if err != nil { log.Err(err) diff --git a/engine/configs/config.example.logical_generic.yml b/engine/configs/config.example.logical_generic.yml index a5bcc160..f726b816 100644 --- a/engine/configs/config.example.logical_generic.yml +++ b/engine/configs/config.example.logical_generic.yml @@ -219,9 +219,15 @@ retrieval: # Do not specify the databases section to take all databases. # databases: # database1: - # # Option for a partial dump. Do not specify the tables section to dump all available tables. + # Options for a partial dump. + # Do not specify the tables section to dump all available tables. + # Corresponds to the --table option of pg_dump. # tables: # - table1 + # Do not dump data for any of the tables matching pattern. + # Corresponds to the --exclude-table option of pg_dump. + # excludeTables: + # - table2 # database2: # databaseN: diff --git a/engine/configs/config.example.logical_rds_iam.yml b/engine/configs/config.example.logical_rds_iam.yml index 02d5655e..1337ae57 100644 --- a/engine/configs/config.example.logical_rds_iam.yml +++ b/engine/configs/config.example.logical_rds_iam.yml @@ -281,9 +281,15 @@ retrieval: # format: directory # # Compression (only for plain-text dumps): "gzip", "bzip2", or "no". Default: "no". # compression: no - # # Option for a partial restore. Do not specify the tables section to restore all available tables. + # Options for a partial dump. + # Do not specify the tables section to dump all available tables. + # Corresponds to the --table option of pg_dump. # tables: # - table1 + # Do not dump data for any of the tables matching pattern. + # Corresponds to the --exclude-table option of pg_dump. + # excludeTables: + # - table2 # database2: # databaseN: diff --git a/engine/configs/standard/postgres/default/9.6/postgresql.dblab.postgresql.conf b/engine/configs/standard/postgres/default/9.6/postgresql.dblab.postgresql.conf index 4469a013..cd8de14f 100644 --- a/engine/configs/standard/postgres/default/9.6/postgresql.dblab.postgresql.conf +++ b/engine/configs/standard/postgres/default/9.6/postgresql.dblab.postgresql.conf @@ -341,7 +341,7 @@ dynamic_shared_memory_type = posix # the default is the first option # (change requires restart) # These are only used if logging_collector is on: -#log_directory = 'pg_log' # directory where log files are written, +log_directory = 'log' # directory where log files are written, # can be absolute or relative to PGDATA #log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # log file name pattern, # can include strftime() escapes diff --git a/engine/go.mod b/engine/go.mod index 2f8cea5c..66b688ad 100644 --- a/engine/go.mod +++ b/engine/go.mod @@ -66,6 +66,7 @@ require ( github.com/shurcooL/sanitized_anchor_name v1.0.0 // indirect github.com/sirupsen/logrus v1.8.1 // indirect github.com/stretchr/objx v0.2.0 // indirect + golang.org/x/mod v0.5.1 // indirect golang.org/x/net v0.0.0-20211216030914-fe4d6282115f // indirect golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e // indirect golang.org/x/text v0.3.7 // indirect diff --git a/engine/go.sum b/engine/go.sum index e4029969..0b90296b 100644 --- a/engine/go.sum +++ b/engine/go.sum @@ -779,6 +779,8 @@ golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.5.1 h1:OJxoQ/rynoF0dcCdI7cLPktw/hR2cueqYfjm43oqK38= +golang.org/x/mod v0.5.1/go.mod h1:5OXOZSfqPIIbmVBIIKWRFfZjPR0E5r58TLhUjH0a2Ro= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= diff --git a/engine/internal/embeddedui/embedded_ui.go b/engine/internal/embeddedui/embedded_ui.go index 717c17b5..994a9c47 100644 --- a/engine/internal/embeddedui/embedded_ui.go +++ b/engine/internal/embeddedui/embedded_ui.go @@ -13,7 +13,6 @@ import ( "github.com/docker/docker/api/types" "github.com/docker/docker/api/types/container" - "github.com/docker/docker/api/types/network" "github.com/docker/docker/client" "github.com/docker/go-connections/nat" @@ -91,11 +90,11 @@ func (ui *UIManager) isConfigChanged(cfg Config) bool { // Run creates a new embedded UI container. func (ui *UIManager) Run(ctx context.Context) error { - if err := docker.PrepareImage(ui.runner, ui.cfg.DockerImage); err != nil { + if err := docker.PrepareImage(ctx, ui.docker, ui.cfg.DockerImage); err != nil { return fmt.Errorf("failed to prepare Docker image: %w", err) } - embeddedUI, err := ui.docker.ContainerCreate(ctx, + containerID, err := tools.CreateContainerIfMissing(ctx, ui.docker, getEmbeddedUIName(ui.engProps.InstanceID), &container.Config{ Labels: map[string]string{ cont.DBLabSatelliteLabel: cont.DBLabEmbeddedUILabel, @@ -122,22 +121,18 @@ func (ui *UIManager) Run(ctx context.Context) error { }, }, }, - }, - &network.NetworkingConfig{}, - nil, - getEmbeddedUIName(ui.engProps.InstanceID), - ) + }) if err != nil { return fmt.Errorf("failed to prepare Docker image for embedded UI: %w", err) } - if err := networks.Connect(ctx, ui.docker, ui.engProps.InstanceID, embeddedUI.ID); err != nil { + if err := networks.Connect(ctx, ui.docker, ui.engProps.InstanceID, containerID); err != nil { return fmt.Errorf("failed to connect UI container to the internal Docker network: %w", err) } - if err := ui.docker.ContainerStart(ctx, embeddedUI.ID, types.ContainerStartOptions{}); err != nil { - return fmt.Errorf("failed to start container %q: %w", embeddedUI.ID, err) + if err := ui.docker.ContainerStart(ctx, containerID, types.ContainerStartOptions{}); err != nil { + return fmt.Errorf("failed to start container %q: %w", containerID, err) } reportLaunching(ui.cfg) diff --git a/engine/internal/embeddedui/embedded_ui_integration_test.go b/engine/internal/embeddedui/embedded_ui_integration_test.go new file mode 100644 index 00000000..88535f5e --- /dev/null +++ b/engine/internal/embeddedui/embedded_ui_integration_test.go @@ -0,0 +1,80 @@ +//go:build integration +// +build integration + +/* +2021 © Postgres.ai +*/ + +package embeddedui + +import ( + "context" + "testing" + "time" + + "github.com/docker/docker/api/types" + "github.com/docker/docker/api/types/filters" + "github.com/docker/docker/client" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "gitlab.com/postgres-ai/database-lab/v3/internal/provision/runners" + "gitlab.com/postgres-ai/database-lab/v3/internal/retrieval/engine/postgres/tools" + "gitlab.com/postgres-ai/database-lab/v3/pkg/config/global" + "gitlab.com/postgres-ai/database-lab/v3/pkg/util/networks" +) + +func TestStartExistingContainer(t *testing.T) { + t.Parallel() + docker, err := client.NewClientWithOpts(client.FromEnv) + require.NoError(t, err) + + engProps := global.EngineProps{ + InstanceID: "testuistart", + } + + embeddedUI := New( + Config{ + // "mock" UI image + DockerImage: "gcr.io/google_containers/pause-amd64:3.0", + }, + engProps, + runners.NewLocalRunner(false), + docker, + ) + + ctx := context.TODO() + + networks.Setup(ctx, docker, engProps.InstanceID, getEmbeddedUIName(engProps.InstanceID)) + + // clean test UI container + defer tools.RemoveContainer(ctx, docker, getEmbeddedUIName(engProps.InstanceID), 30*time.Second) + + // start UI container + err = embeddedUI.Run(ctx) + require.NoError(t, err) + + // explicitly stop container + tools.StopContainer(ctx, docker, getEmbeddedUIName(engProps.InstanceID), 30*time.Second) + + // start UI container back + err = embeddedUI.Run(ctx) + require.NoError(t, err) + + // list containers + filterArgs := filters.NewArgs() + filterArgs.Add("name", getEmbeddedUIName(engProps.InstanceID)) + + list, err := docker.ContainerList( + ctx, + types.ContainerListOptions{ + All: true, + Filters: filterArgs, + }, + ) + + require.NoError(t, err) + assert.NotEmpty(t, list) + // initial container + assert.Equal(t, "running", list[0].State) +} diff --git a/engine/internal/observer/stats.go b/engine/internal/observer/stats.go index 87470cd9..aacbef55 100644 --- a/engine/internal/observer/stats.go +++ b/engine/internal/observer/stats.go @@ -129,7 +129,7 @@ func (c *ObservingClone) getMaxQueryTime(ctx context.Context, maxTime *float64) func (c *ObservingClone) countLogErrors(ctx context.Context, logErrors *LogErrors) error { row := c.superUserDB.QueryRow(ctx, `select coalesce(sum(count), 0), coalesce(string_agg(distinct message, ','), '') from pg_log_errors_stats() - where type in ('ERROR', 'FATAL') and database = current_database()`) + where type in ('ERROR', 'FATAL')`) return row.Scan(&logErrors.Count, &logErrors.Message) } diff --git a/engine/internal/provision/databases/postgres/postgres.go b/engine/internal/provision/databases/postgres/postgres.go index b1527b68..cc4e603f 100644 --- a/engine/internal/provision/databases/postgres/postgres.go +++ b/engine/internal/provision/databases/postgres/postgres.go @@ -193,3 +193,48 @@ func runSimpleSQL(command, connStr string) (string, error) { return result, err } + +// runSQLSelectQuery executes a select query and returns the result as a slice of strings. +func runSQLSelectQuery(selectQuery, connStr string) ([]string, error) { + result := make([]string, 0) + db, err := sql.Open("postgres", connStr) + + if err != nil { + return result, fmt.Errorf("cannot connect to database: %w", err) + } + + defer func() { + err := db.Close() + + if err != nil { + log.Err("cannot close database connection.") + } + }() + + rows, err := db.Query(selectQuery) + + if err != nil { + return result, fmt.Errorf("failed to execute query: %w", err) + } + + for rows.Next() { + var s string + + if e := rows.Scan(&s); e != nil { + log.Err("query execution error:", e) + return result, e + } + + result = append(result, s) + } + + if err := rows.Err(); err != nil { + return result, fmt.Errorf("query execution error: %w", err) + } + + if err := rows.Close(); err != nil { + return result, fmt.Errorf("cannot close database result: %w", err) + } + + return result, err +} diff --git a/engine/internal/provision/databases/postgres/postgres_mgmt.go b/engine/internal/provision/databases/postgres/postgres_mgmt.go index 6326dd95..b1ae0fe8 100644 --- a/engine/internal/provision/databases/postgres/postgres_mgmt.go +++ b/engine/internal/provision/databases/postgres/postgres_mgmt.go @@ -70,6 +70,9 @@ func ResetAllPasswords(c *resources.AppConfig, whitelistUsers []string) error { return nil } +// selectAllDatabases provides a query to list available databases. +const selectAllDatabases = "select datname from pg_catalog.pg_database where not datistemplate" + // CreateUser defines a method for creation of Postgres user. func CreateUser(c *resources.AppConfig, user resources.EphemeralUser) error { var query string @@ -80,17 +83,43 @@ func CreateUser(c *resources.AppConfig, user resources.EphemeralUser) error { } if user.Restricted { - query = restrictedUserQuery(user.Name, user.Password, dbName) + // create restricted user + query = restrictedUserQuery(user.Name, user.Password) + out, err := runSimpleSQL(query, getPgConnStr(c.Host, dbName, c.DB.Username, c.Port)) + + if err != nil { + return fmt.Errorf("failed to create restricted user: %w", err) + } + + log.Dbg("Restricted user has been created: ", out) + + // set restricted user as owner for database objects + databaseList, err := runSQLSelectQuery(selectAllDatabases, getPgConnStr(c.Host, dbName, c.DB.Username, c.Port)) + + if err != nil { + return fmt.Errorf("failed list all databases: %w", err) + } + + for _, database := range databaseList { + query = restrictedObjectsQuery(user.Name) + out, err = runSimpleSQL(query, getPgConnStr(c.Host, database, c.DB.Username, c.Port)) + + if err != nil { + return fmt.Errorf("failed to run objects restrict query: %w", err) + } + + log.Dbg("Objects restriction applied", database, out) + } } else { query = superuserQuery(user.Name, user.Password) - } - out, err := runSimpleSQL(query, getPgConnStr(c.Host, dbName, c.DB.Username, c.Port)) - if err != nil { - return errors.Wrap(err, "failed to run psql") - } + out, err := runSimpleSQL(query, getPgConnStr(c.Host, dbName, c.DB.Username, c.Port)) + if err != nil { + return fmt.Errorf("failed to create superuser: %w", err) + } - log.Dbg("AddUser:", out) + log.Dbg("Super user has been created: ", out) + } return nil } @@ -99,13 +128,31 @@ func superuserQuery(username, password string) string { return fmt.Sprintf(`create user %s with password %s login superuser;`, pq.QuoteIdentifier(username), pq.QuoteLiteral(password)) } -const restrictionTemplate = ` +const restrictionUserCreationTemplate = ` -- create a new user create user @username with password @password login; +do $$ +declare + new_owner text; + object_type record; + r record; +begin + new_owner := @usernameStr; --- change a database owner -alter database @database owner to @username; + -- Changing owner of all databases + for r in select datname from pg_catalog.pg_database where not datistemplate loop + raise debug 'Changing owner of %', r.datname; + execute format( + 'alter database %s owner to %s;', + r.datname, + new_owner + ); + end loop; +end +$$; +` +const restrictionTemplate = ` do $$ declare new_owner text; @@ -260,12 +307,20 @@ end $$; ` -func restrictedUserQuery(username, password, database string) string { +func restrictedUserQuery(username, password string) string { repl := strings.NewReplacer( "@usernameStr", pq.QuoteLiteral(username), "@username", pq.QuoteIdentifier(username), "@password", pq.QuoteLiteral(password), - "@database", pq.QuoteIdentifier(database), + ) + + return repl.Replace(restrictionUserCreationTemplate) +} + +func restrictedObjectsQuery(username string) string { + repl := strings.NewReplacer( + "@usernameStr", pq.QuoteLiteral(username), + "@username", pq.QuoteIdentifier(username), ) return repl.Replace(restrictionTemplate) diff --git a/engine/internal/provision/databases/postgres/postgres_mgmt_test.go b/engine/internal/provision/databases/postgres/postgres_mgmt_test.go index 6679777e..e510484f 100644 --- a/engine/internal/provision/databases/postgres/postgres_mgmt_test.go +++ b/engine/internal/provision/databases/postgres/postgres_mgmt_test.go @@ -28,8 +28,7 @@ func TestRestrictedUserQuery(t *testing.T) { t.Run("username and password must be quoted", func(t *testing.T) { user := "user1" pwd := "pwd" - db := "postgres" - query := restrictedUserQuery(user, pwd, db) + query := restrictedUserQuery(user, pwd) assert.Contains(t, query, `create user "user1" with password 'pwd' login;`) assert.Contains(t, query, `new_owner := 'user1'`) @@ -39,10 +38,18 @@ func TestRestrictedUserQuery(t *testing.T) { t.Run("special chars must be quoted", func(t *testing.T) { user := "user.test\"" pwd := "pwd\\'--" - db := "postgres" - query := restrictedUserQuery(user, pwd, db) + query := restrictedUserQuery(user, pwd) assert.Contains(t, query, `create user "user.test""" with password E'pwd\\''--' login;`) assert.Contains(t, query, `new_owner := 'user.test"'`) }) + + t.Run("change owner of all databases", func(t *testing.T) { + user := "user.test" + pwd := "pwd" + query := restrictedUserQuery(user, pwd) + + assert.Contains(t, query, `select datname from pg_catalog.pg_database where not datistemplat`) + }) + } diff --git a/engine/internal/provision/docker/docker.go b/engine/internal/provision/docker/docker.go index f9a3a86b..820a3513 100644 --- a/engine/internal/provision/docker/docker.go +++ b/engine/internal/provision/docker/docker.go @@ -9,12 +9,14 @@ import ( "context" "encoding/json" "fmt" + "io" "os" "path" "strconv" "strings" "github.com/docker/docker/api/types" + "github.com/docker/docker/api/types/filters" "github.com/docker/docker/client" "github.com/pkg/errors" "github.com/shirou/gopsutil/host" @@ -22,14 +24,24 @@ import ( "gitlab.com/postgres-ai/database-lab/v3/internal/provision/resources" "gitlab.com/postgres-ai/database-lab/v3/internal/provision/runners" "gitlab.com/postgres-ai/database-lab/v3/internal/retrieval/engine/postgres/tools" + "gitlab.com/postgres-ai/database-lab/v3/pkg/log" ) const ( labelClone = "dblab_clone" + + // referenceKey uses as a filtering key to identify image tag. + referenceKey = "reference" ) var systemVolumes = []string{"/sys", "/lib", "/proc"} +// imagePullProgress describes the progress of pulling the container image. +type imagePullProgress struct { + Status string `json:"status"` + Progress string `json:"progress"` +} + // RunContainer runs specified container. func RunContainer(r runners.Runner, c *resources.AppConfig) error { hostInfo, err := host.Info() @@ -221,8 +233,8 @@ func Exec(r runners.Runner, c *resources.AppConfig, cmd string) (string, error) } // PrepareImage prepares a Docker image to use. -func PrepareImage(runner runners.Runner, dockerImage string) error { - imageExists, err := ImageExists(runner, dockerImage) +func PrepareImage(ctx context.Context, docker *client.Client, dockerImage string) error { + imageExists, err := ImageExists(ctx, docker, dockerImage) if err != nil { return fmt.Errorf("cannot check docker image existence: %w", err) } @@ -231,7 +243,7 @@ func PrepareImage(runner runners.Runner, dockerImage string) error { return nil } - if err := PullImage(runner, dockerImage); err != nil { + if err := PullImage(ctx, docker, dockerImage); err != nil { return fmt.Errorf("cannot pull docker image: %w", err) } @@ -239,23 +251,50 @@ func PrepareImage(runner runners.Runner, dockerImage string) error { } // ImageExists checks existence of Docker image. -func ImageExists(r runners.Runner, dockerImage string) (bool, error) { - dockerListImagesCmd := "docker images " + dockerImage + " --quiet" +func ImageExists(ctx context.Context, docker *client.Client, dockerImage string) (bool, error) { + filterArgs := filters.NewArgs() + filterArgs.Add(referenceKey, dockerImage) + + list, err := docker.ImageList(ctx, types.ImageListOptions{ + All: false, + Filters: filterArgs, + }) - out, err := r.Run(dockerListImagesCmd, true) if err != nil { return false, fmt.Errorf("failed to list images: %w", err) } - return len(strings.TrimSpace(out)) > 0, nil + return len(list) > 0, nil } // PullImage pulls Docker image from DockerHub registry. -func PullImage(r runners.Runner, dockerImage string) error { - dockerPullImageCmd := "docker pull " + dockerImage +func PullImage(ctx context.Context, docker *client.Client, dockerImage string) error { + pullResponse, err := docker.ImagePull(ctx, dockerImage, types.ImagePullOptions{}) + + if err != nil { + return fmt.Errorf("failed to pull image: %w", err) + } + + // reading output of image pulling, without reading pull will not be performed + decoder := json.NewDecoder(pullResponse) - if _, err := r.Run(dockerPullImageCmd, true); err != nil { - return fmt.Errorf("failed to pull images: %w", err) + for { + var pullResult imagePullProgress + if err := decoder.Decode(&pullResult); err != nil { + if errors.Is(err, io.EOF) { + break + } + + return fmt.Errorf("failed to pull image: %w", err) + } + + log.Dbg("Image pulling progress", pullResult.Status, pullResult.Progress) + } + + err = pullResponse.Close() + + if err != nil { + return fmt.Errorf("failed to pull image: %w", err) } return nil diff --git a/engine/internal/provision/mode_local.go b/engine/internal/provision/mode_local.go index dcdf1a02..9bd0342b 100644 --- a/engine/internal/provision/mode_local.go +++ b/engine/internal/provision/mode_local.go @@ -125,7 +125,7 @@ func (p *Provisioner) Init() error { return fmt.Errorf("failed to revise port pool: %w", err) } - if err := docker.PrepareImage(p.runner, p.config.DockerImage); err != nil { + if err := docker.PrepareImage(p.ctx, p.dockerClient, p.config.DockerImage); err != nil { return fmt.Errorf("cannot prepare docker image %s: %w", p.config.DockerImage, err) } diff --git a/engine/internal/provision/pool/manager.go b/engine/internal/provision/pool/manager.go index dd517351..74c41171 100644 --- a/engine/internal/provision/pool/manager.go +++ b/engine/internal/provision/pool/manager.go @@ -69,16 +69,12 @@ func NewManager(runner runners.Runner, config ManagerConfig) (FSManager, error) switch config.Pool.Mode { case zfs.PoolMode: - osUser, err := user.Current() + zfsConfig, err := buildZFSConfig(config) if err != nil { - return nil, errors.Wrap(err, "failed to get current user") + return nil, err } - manager = zfs.NewFSManager(runner, zfs.Config{ - Pool: config.Pool, - PreSnapshotSuffix: config.PreSnapshotSuffix, - OSUsername: osUser.Username, - }) + manager = zfs.NewFSManager(runner, zfsConfig) case lvm.PoolMode: if manager, err = lvm.NewFSManager(runner, config.Pool); err != nil { @@ -93,3 +89,41 @@ func NewManager(runner runners.Runner, config ManagerConfig) (FSManager, error) return manager, nil } + +// BuildFromExistingManager prepares FSManager from an existing one. +func BuildFromExistingManager(fsm FSManager, config ManagerConfig) (FSManager, error) { + switch manager := fsm.(type) { + case *zfs.Manager: + zfsConfig, err := buildZFSConfig(config) + if err != nil { + return nil, err + } + + manager.UpdateConfig(zfsConfig) + + fsm = manager + + case *lvm.LVManager: + manager.UpdateConfig(config.Pool) + + fsm = manager + + default: + return nil, fmt.Errorf(`unsupported thin-clone manager: %T`, manager) + } + + return fsm, nil +} + +func buildZFSConfig(config ManagerConfig) (zfs.Config, error) { + osUser, err := user.Current() + if err != nil { + return zfs.Config{}, fmt.Errorf("failed to get current user: %w", err) + } + + return zfs.Config{ + Pool: config.Pool, + PreSnapshotSuffix: config.PreSnapshotSuffix, + OSUsername: osUser.Username, + }, nil +} diff --git a/engine/internal/provision/pool/pool_manager.go b/engine/internal/provision/pool/pool_manager.go index 75948504..e58c0c8b 100644 --- a/engine/internal/provision/pool/pool_manager.go +++ b/engine/internal/provision/pool/pool_manager.go @@ -259,6 +259,10 @@ func (pm *Manager) examineEntries(entries []os.DirEntry) (map[string]FSManager, poolMappings := make(map[string]string) + pm.mu.Lock() + originalPools := pm.fsManagerPool + pm.mu.Unlock() + for _, entry := range entries { if !entry.IsDir() { continue @@ -324,10 +328,19 @@ func (pm *Manager) examineEntries(entries []os.DirEntry) (map[string]FSManager, } } - fsm, err := NewManager(pm.runner, ManagerConfig{ + var fsm FSManager + + managerConfig := ManagerConfig{ Pool: pool, PreSnapshotSuffix: pm.cfg.PreSnapshotSuffix, - }) + } + + if originalFSM, ok := originalPools[pool.Name]; ok { + fsm, err = BuildFromExistingManager(originalFSM, managerConfig) + } else { + fsm, err = NewManager(pm.runner, managerConfig) + } + if err != nil { log.Msg("failed to create clone manager:", err.Error()) continue diff --git a/engine/internal/provision/thinclones/lvm/lvmanager.go b/engine/internal/provision/thinclones/lvm/lvmanager.go index f683f893..35da7082 100644 --- a/engine/internal/provision/thinclones/lvm/lvmanager.go +++ b/engine/internal/provision/thinclones/lvm/lvmanager.go @@ -47,6 +47,11 @@ func (m *LVManager) Pool() *resources.Pool { return m.pool } +// UpdateConfig updates the manager's pool. +func (m *LVManager) UpdateConfig(pool *resources.Pool) { + m.pool = pool +} + // CreateClone creates a new volume. func (m *LVManager) CreateClone(name, _ string) error { return CreateVolume(m.runner, m.volumeGroup, m.logicalVolume, name, m.pool.ClonesDir()) diff --git a/engine/internal/provision/thinclones/zfs/zfs.go b/engine/internal/provision/thinclones/zfs/zfs.go index 5d9a5d6a..bcd6254f 100644 --- a/engine/internal/provision/thinclones/zfs/zfs.go +++ b/engine/internal/provision/thinclones/zfs/zfs.go @@ -159,9 +159,10 @@ type Config struct { // NewFSManager creates a new Manager instance for ZFS. func NewFSManager(runner runners.Runner, config Config) *Manager { m := Manager{ - runner: runner, - config: config, - mu: &sync.Mutex{}, + runner: runner, + config: config, + mu: &sync.Mutex{}, + snapshots: make([]resources.Snapshot, 0), } return &m @@ -172,6 +173,11 @@ func (m *Manager) Pool() *resources.Pool { return m.config.Pool } +// UpdateConfig updates the manager's configuration. +func (m *Manager) UpdateConfig(cfg Config) { + m.config = cfg +} + // CreateClone creates a new ZFS clone. func (m *Manager) CreateClone(cloneName, snapshotID string) error { exists, err := m.cloneExists(cloneName) @@ -316,14 +322,20 @@ func (m *Manager) CreateSnapshot(poolSuffix, dataStateAt string) (string, error) return "", fmt.Errorf("failed to parse dataStateAt: %w", err) } - m.addSnapshotToList(resources.Snapshot{ + newSnapshot := resources.Snapshot{ ID: snapshotName, CreatedAt: time.Now(), DataStateAt: dataStateTime, Pool: m.config.Pool.Name, - }) + } + + if !strings.HasSuffix(snapshotName, m.config.PreSnapshotSuffix) { + m.addSnapshotToList(newSnapshot) - go m.RefreshSnapshotList() + log.Dbg("New snapshot:", newSnapshot) + + m.RefreshSnapshotList() + } return snapshotName, nil } @@ -511,8 +523,11 @@ func (m *Manager) GetFilesystemState() (models.FileSystem, error) { // SnapshotList returns a list of snapshots. func (m *Manager) SnapshotList() []resources.Snapshot { - snapshotList := m.snapshots - return snapshotList + m.mu.Lock() + snapshots := m.snapshots + m.mu.Unlock() + + return snapshots } // RefreshSnapshotList updates the list of snapshots. @@ -559,20 +574,22 @@ func (m *Manager) getSnapshots() ([]resources.Snapshot, error) { func (m *Manager) addSnapshotToList(snapshot resources.Snapshot) { m.mu.Lock() - m.snapshots = append(m.snapshots, snapshot) + m.snapshots = append([]resources.Snapshot{snapshot}, m.snapshots...) m.mu.Unlock() } func (m *Manager) removeSnapshotFromList(snapshotName string) { + m.mu.Lock() + for i, snapshot := range m.snapshots { if snapshot.ID == snapshotName { - m.mu.Lock() - m.snapshots = append(m.snapshots[:i], m.snapshots[i+1:]...) - m.mu.Unlock() + m.snapshots = append((m.snapshots)[:i], (m.snapshots)[i+1:]...) break } } + + m.mu.Unlock() } // ListFilesystems lists ZFS file systems (clones, pools). diff --git a/engine/internal/provision/thinclones/zfs/zfs_test.go b/engine/internal/provision/thinclones/zfs/zfs_test.go index 1b2611f6..db2acecd 100644 --- a/engine/internal/provision/thinclones/zfs/zfs_test.go +++ b/engine/internal/provision/thinclones/zfs/zfs_test.go @@ -199,7 +199,7 @@ func TestBuildingCommands(t *testing.T) { func TestSnapshotList(t *testing.T) { t.Run("Snapshot list", func(t *testing.T) { - fsManager := NewFSManager(runnerMock{}, Config{}) + fsManager := NewFSManager(runnerMock{}, Config{Pool: &resources.Pool{Name: "testPool"}}) require.Equal(t, 0, len(fsManager.SnapshotList())) @@ -216,11 +216,11 @@ func TestSnapshotList(t *testing.T) { fsManager.addSnapshotToList(snapshot3) require.Equal(t, 3, len(fsManager.SnapshotList())) - require.Equal(t, []resources.Snapshot{{ID: "test1"}, {ID: "test2"}, {ID: "test3"}}, fsManager.SnapshotList()) + require.Equal(t, []resources.Snapshot{{ID: "test3"}, {ID: "test2"}, {ID: "test1"}}, fsManager.SnapshotList()) fsManager.removeSnapshotFromList("test2") require.Equal(t, 2, len(fsManager.SnapshotList())) - require.Equal(t, []resources.Snapshot{{ID: "test1"}, {ID: "test3"}}, fsManager.SnapshotList()) + require.Equal(t, []resources.Snapshot{{ID: "test3"}, {ID: "test1"}}, fsManager.SnapshotList()) }) } diff --git a/engine/internal/retrieval/engine/postgres/logical/dump.go b/engine/internal/retrieval/engine/postgres/logical/dump.go index 9ca3616a..e787c3e1 100644 --- a/engine/internal/retrieval/engine/postgres/logical/dump.go +++ b/engine/internal/retrieval/engine/postgres/logical/dump.go @@ -15,7 +15,6 @@ import ( "github.com/docker/docker/api/types" "github.com/docker/docker/api/types/container" - "github.com/docker/docker/api/types/network" "github.com/docker/docker/client" "github.com/jackc/pgx/v4" "github.com/pkg/errors" @@ -30,7 +29,6 @@ import ( "gitlab.com/postgres-ai/database-lab/v3/internal/retrieval/engine/postgres/tools/defaults" "gitlab.com/postgres-ai/database-lab/v3/internal/retrieval/engine/postgres/tools/health" "gitlab.com/postgres-ai/database-lab/v3/internal/retrieval/options" - "gitlab.com/postgres-ai/database-lab/v3/pkg/config/global" "gitlab.com/postgres-ai/database-lab/v3/pkg/log" ) @@ -98,10 +96,11 @@ type Source struct { // DumpDefinition describes a database for dumping. type DumpDefinition struct { - Tables []string `yaml:"tables"` - Format string `yaml:"format"` - Compression compressionType `yaml:"compression"` - dbName string + Tables []string `yaml:"tables"` + ExcludeTables []string `yaml:"excludeTables"` + Format string `yaml:"format"` + Compression compressionType `yaml:"compression"` + dbName string } type dumpJobConfig struct { @@ -269,16 +268,13 @@ func (d *DumpJob) Run(ctx context.Context) (err error) { return errors.Wrap(err, "failed to generate PostgreSQL password") } - dumpCont, err := d.dockerClient.ContainerCreate(ctx, d.buildContainerConfig(pwd), hostConfig, &network.NetworkingConfig{}, - nil, d.dumpContainerName(), - ) - if err != nil { - log.Err(err) + containerID, err := tools.CreateContainerIfMissing(ctx, d.dockerClient, d.dumpContainerName(), d.buildContainerConfig(pwd), hostConfig) - return errors.Wrapf(err, "failed to create container %q", d.dumpContainerName()) + if err != nil { + return fmt.Errorf("failed to create container %q %w", d.dumpContainerName(), err) } - defer tools.RemoveContainer(ctx, d.dockerClient, dumpCont.ID, cont.StopTimeout) + defer tools.RemoveContainer(ctx, d.dockerClient, containerID, cont.StopTimeout) defer func() { if err != nil { @@ -286,9 +282,9 @@ func (d *DumpJob) Run(ctx context.Context) (err error) { } }() - log.Msg(fmt.Sprintf("Running container: %s. ID: %v", d.dumpContainerName(), dumpCont.ID)) + log.Msg(fmt.Sprintf("Running container: %s. ID: %v", d.dumpContainerName(), containerID)) - if err := d.dockerClient.ContainerStart(ctx, dumpCont.ID, types.ContainerStartOptions{}); err != nil { + if err := d.dockerClient.ContainerStart(ctx, containerID, types.ContainerStartOptions{}); err != nil { return errors.Wrapf(err, "failed to start container %q", d.dumpContainerName()) } @@ -298,13 +294,13 @@ func (d *DumpJob) Run(ctx context.Context) (err error) { log.Msg("Waiting for container readiness") - if err := tools.MakeDir(ctx, d.dockerClient, dumpCont.ID, tmpDBLabPGDataDir); err != nil { + if err := tools.MakeDir(ctx, d.dockerClient, containerID, tmpDBLabPGDataDir); err != nil { return err } dataDir := d.fsPool.DataDir() - if err := tools.CheckContainerReadiness(ctx, d.dockerClient, dumpCont.ID); err != nil { + if err := tools.CheckContainerReadiness(ctx, d.dockerClient, containerID); err != nil { var errHealthCheck *tools.ErrHealthCheck if !errors.As(err, &errHealthCheck) { return errors.Wrap(err, "failed to readiness check") @@ -315,13 +311,13 @@ func (d *DumpJob) Run(ctx context.Context) (err error) { pgDataDir = dataDir } - if err := setupPGData(ctx, d.dockerClient, pgDataDir, dumpCont.ID); err != nil { + if err := setupPGData(ctx, d.dockerClient, pgDataDir, containerID); err != nil { return errors.Wrap(err, "failed to set up Postgres data") } } if d.DumpOptions.Restore.Enabled && len(d.DumpOptions.Restore.Configs) > 0 { - if err := updateConfigs(ctx, d.dockerClient, dataDir, dumpCont.ID, d.DumpOptions.Restore.Configs); err != nil { + if err := updateConfigs(ctx, d.dockerClient, dataDir, containerID, d.DumpOptions.Restore.Configs); err != nil { return errors.Wrap(err, "failed to update configs") } } @@ -335,12 +331,12 @@ func (d *DumpJob) Run(ctx context.Context) (err error) { } } - if err := d.cleanupDumpLocation(ctx, dumpCont.ID, dbList); err != nil { + if err := d.cleanupDumpLocation(ctx, containerID, dbList); err != nil { return err } for dbName, dbDetails := range dbList { - if err := d.dumpDatabase(ctx, dumpCont.ID, dbName, dbDetails); err != nil { + if err := d.dumpDatabase(ctx, containerID, dbName, dbDetails); err != nil { return errors.Wrapf(err, "failed to dump the database %s", dbName) } } @@ -357,11 +353,15 @@ func (d *DumpJob) Run(ctx context.Context) (err error) { log.Msg("Running analyze command: ", analyzeCmd) - if err := tools.ExecCommand(ctx, d.dockerClient, dumpCont.ID, types.ExecConfig{Cmd: analyzeCmd}); err != nil { + if err := tools.ExecCommand(ctx, d.dockerClient, containerID, types.ExecConfig{Cmd: analyzeCmd}); err != nil { return errors.Wrap(err, "failed to recalculate statistics after restore") } - if err := tools.StopPostgres(ctx, d.dockerClient, dumpCont.ID, dataDir, tools.DefaultStopTimeout); err != nil { + if err := tools.RunCheckpoint(ctx, d.dockerClient, containerID, d.globalCfg.Database.User(), d.globalCfg.Database.DBName); err != nil { + return errors.Wrap(err, "failed to run checkpoint before stop") + } + + if err := tools.StopPostgres(ctx, d.dockerClient, containerID, dataDir, tools.DefaultStopTimeout); err != nil { return errors.Wrap(err, "failed to stop Postgres instance") } } @@ -435,13 +435,23 @@ func (d *DumpJob) cleanupDumpLocation(ctx context.Context, dumpContID string, db } func (d *DumpJob) dumpDatabase(ctx context.Context, dumpContID, dbName string, dumpDefinition DumpDefinition) error { - dumpCommand := d.buildLogicalDumpCommand(dbName, dumpDefinition.Tables) - log.Msg("Running dump command: ", dumpCommand) + dumpCommand := d.buildLogicalDumpCommand(dbName, dumpDefinition) + + if len(dumpDefinition.Tables) > 0 || + len(dumpDefinition.ExcludeTables) > 0 { + log.Msg("Partial dump") - if len(dumpDefinition.Tables) > 0 { - log.Msg("Partial dump will be run. Tables for dumping: ", strings.Join(dumpDefinition.Tables, ", ")) + if len(dumpDefinition.Tables) > 0 { + log.Msg("Including tables: ", strings.Join(dumpDefinition.Tables, ", ")) + } + + if len(dumpDefinition.ExcludeTables) > 0 { + log.Msg("Excluding tables: ", strings.Join(dumpDefinition.ExcludeTables, ", ")) + } } + log.Msg("Running dump command: ", dumpCommand) + if output, err := d.performDumpCommand(ctx, dumpContID, types.ExecConfig{ Tty: true, Cmd: dumpCommand, @@ -488,7 +498,12 @@ func setupPGData(ctx context.Context, dockerClient *client.Client, dataDir strin return nil } -func updateConfigs(ctx context.Context, dockerClient *client.Client, dataDir, contID string, configs map[string]string) error { +func updateConfigs( + ctx context.Context, + dockerClient *client.Client, + dataDir, contID string, + configs map[string]string, +) error { log.Dbg("Stopping container to update configuration") tools.StopContainer(ctx, dockerClient, contID, cont.StopTimeout) @@ -605,21 +620,38 @@ func (d *DumpJob) getExecEnvironmentVariables() []string { return execEnvs } -func (d *DumpJob) buildLogicalDumpCommand(dbName string, tables []string) []string { - optionalArgs := map[string]string{ - "--host": d.config.db.Host, - "--port": strconv.Itoa(d.config.db.Port), - "--username": d.config.db.Username, - "--dbname": dbName, - "--jobs": strconv.Itoa(d.DumpOptions.ParallelJobs), +func (d *DumpJob) buildLogicalDumpCommand(dbName string, dump DumpDefinition) []string { + // don't use map here, it creates inconsistency in the order of arguments + dumpCmd := []string{"pg_dump", "--create"} + + if d.config.db.Host != "" { + dumpCmd = append(dumpCmd, "--host", d.config.db.Host) + } + + if d.config.db.Port > 0 { + dumpCmd = append(dumpCmd, "--port", strconv.Itoa(d.config.db.Port)) + } + + if d.config.db.Username != "" { + dumpCmd = append(dumpCmd, "--username", d.config.db.Username) } - dumpCmd := append([]string{"pg_dump", "--create"}, prepareCmdOptions(optionalArgs)...) + if dbName != "" { + dumpCmd = append(dumpCmd, "--dbname", dbName) + } - for _, table := range tables { + if d.DumpOptions.ParallelJobs > 0 { + dumpCmd = append(dumpCmd, "--jobs", strconv.Itoa(d.DumpOptions.ParallelJobs)) + } + + for _, table := range dump.Tables { dumpCmd = append(dumpCmd, "--table", table) } + for _, table := range dump.ExcludeTables { + dumpCmd = append(dumpCmd, "--exclude-table", table) + } + // Define if restore directly or export to dump location. if d.DumpOptions.Restore.Enabled { dumpCmd = append(dumpCmd, "--format", customFormat) @@ -652,18 +684,6 @@ func (d *DumpJob) buildLogicalRestoreCommand(dbName string) []string { return restoreCmd } -func prepareCmdOptions(options map[string]string) []string { - cmdOptions := []string{} - - for optionKey, optionValue := range options { - if optionValue != "" { - cmdOptions = append(cmdOptions, optionKey, optionValue) - } - } - - return cmdOptions -} - func (d *DumpJob) markDatabaseData() error { if err := d.dbMarker.CreateConfig(); err != nil { return errors.Wrap(err, "failed to create a DBMarker config of the database") diff --git a/engine/internal/retrieval/engine/postgres/logical/dump_integration_test.go b/engine/internal/retrieval/engine/postgres/logical/dump_integration_test.go new file mode 100644 index 00000000..7306f03a --- /dev/null +++ b/engine/internal/retrieval/engine/postgres/logical/dump_integration_test.go @@ -0,0 +1,91 @@ +//go:build integration +// +build integration + +/* +2021 © Postgres.ai +*/ + +package logical + +import ( + "context" + "fmt" + "math/rand" + "testing" + "time" + + "github.com/docker/docker/api/types" + "github.com/docker/docker/api/types/filters" + "github.com/docker/docker/api/types/network" + "github.com/docker/docker/client" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + dockerutils "gitlab.com/postgres-ai/database-lab/v3/internal/provision/docker" + "gitlab.com/postgres-ai/database-lab/v3/internal/provision/resources" + "gitlab.com/postgres-ai/database-lab/v3/internal/retrieval/config" + "gitlab.com/postgres-ai/database-lab/v3/internal/retrieval/engine/postgres/tools" + "gitlab.com/postgres-ai/database-lab/v3/pkg/config/global" +) + +func TestStartExisingDumpContainer(t *testing.T) { + t.Parallel() + ctx := context.Background() + + docker, err := client.NewClientWithOpts(client.FromEnv) + require.NoError(t, err) + + // create dump job + + source := rand.NewSource(time.Now().UnixNano()) + random := rand.New(source) + + engProps := global.EngineProps{ + InstanceID: fmt.Sprintf("dumpjob-%d", random.Intn(10000)), + } + + job, err := NewDumpJob( + config.JobConfig{ + Spec: config.JobSpec{Name: "test"}, + FSPool: &resources.Pool{ + DataSubDir: t.TempDir(), + }, + Docker: docker, + }, + &global.Config{}, + engProps, + ) + assert.NoError(t, err) + job.DockerImage = "postgresai/extended-postgres:14" + job.DumpOptions.DumpLocation = t.TempDir() + + err = dockerutils.PrepareImage(ctx, docker, job.DockerImage) + assert.NoError(t, err) + + // create dump container and stop it + container, err := docker.ContainerCreate(ctx, job.buildContainerConfig(""), nil, &network.NetworkingConfig{}, + nil, job.dumpContainerName(), + ) + assert.NoError(t, err) + + // clean container in case of any error + defer tools.RemoveContainer(ctx, docker, container.ID, 10*time.Second) + + job.Run(ctx) + + // list containers and check that container job container got processed + filterArgs := filters.NewArgs() + filterArgs.Add("name", job.dumpContainerName()) + + list, err := docker.ContainerList( + ctx, + types.ContainerListOptions{ + All: false, + Filters: filterArgs, + }, + ) + + require.NoError(t, err) + assert.Empty(t, list) + +} diff --git a/engine/internal/retrieval/engine/postgres/logical/restore.go b/engine/internal/retrieval/engine/postgres/logical/restore.go index 3caa4ab8..145881d1 100644 --- a/engine/internal/retrieval/engine/postgres/logical/restore.go +++ b/engine/internal/retrieval/engine/postgres/logical/restore.go @@ -20,7 +20,6 @@ import ( "github.com/docker/docker/api/types" "github.com/docker/docker/api/types/container" - "github.com/docker/docker/api/types/network" "github.com/docker/docker/client" "github.com/docker/docker/pkg/archive" "github.com/pkg/errors" @@ -195,18 +194,13 @@ func (r *RestoreJob) Run(ctx context.Context) (err error) { return errors.Wrap(err, "failed to generate PostgreSQL password") } - restoreCont, err := r.dockerClient.ContainerCreate(ctx, - r.buildContainerConfig(pwd), - hostConfig, - &network.NetworkingConfig{}, - nil, - r.restoreContainerName(), - ) + containerID, err := tools.CreateContainerIfMissing(ctx, r.dockerClient, r.restoreContainerName(), r.buildContainerConfig(pwd), hostConfig) + if err != nil { - return errors.Wrapf(err, "failed to create container %q", r.restoreContainerName()) + return fmt.Errorf("failed to create container %q %w", r.restoreContainerName(), err) } - defer tools.RemoveContainer(ctx, r.dockerClient, restoreCont.ID, cont.StopTimeout) + defer tools.RemoveContainer(ctx, r.dockerClient, containerID, cont.StopTimeout) defer func() { if err != nil { @@ -214,9 +208,9 @@ func (r *RestoreJob) Run(ctx context.Context) (err error) { } }() - log.Msg(fmt.Sprintf("Running container: %s. ID: %v", r.restoreContainerName(), restoreCont.ID)) + log.Msg(fmt.Sprintf("Running container: %s. ID: %v", r.restoreContainerName(), containerID)) - if err := r.dockerClient.ContainerStart(ctx, restoreCont.ID, types.ContainerStartOptions{}); err != nil { + if err := r.dockerClient.ContainerStart(ctx, containerID, types.ContainerStartOptions{}); err != nil { return errors.Wrapf(err, "failed to start container %q", r.restoreContainerName()) } @@ -224,24 +218,24 @@ func (r *RestoreJob) Run(ctx context.Context) (err error) { log.Msg("Waiting for container readiness") - if err := tools.CheckContainerReadiness(ctx, r.dockerClient, restoreCont.ID); err != nil { + if err := tools.CheckContainerReadiness(ctx, r.dockerClient, containerID); err != nil { var errHealthCheck *tools.ErrHealthCheck if !errors.As(err, &errHealthCheck) { return errors.Wrap(err, "failed to readiness check") } - if err := setupPGData(ctx, r.dockerClient, dataDir, restoreCont.ID); err != nil { + if err := setupPGData(ctx, r.dockerClient, dataDir, containerID); err != nil { return errors.Wrap(err, "failed to set up Postgres data") } } if len(r.RestoreOptions.Configs) > 0 { - if err := updateConfigs(ctx, r.dockerClient, dataDir, restoreCont.ID, r.RestoreOptions.Configs); err != nil { + if err := updateConfigs(ctx, r.dockerClient, dataDir, containerID, r.RestoreOptions.Configs); err != nil { return errors.Wrap(err, "failed to update configs") } } - dbList, err := r.getDBList(ctx, restoreCont.ID) + dbList, err := r.getDBList(ctx, containerID) if err != nil { return err } @@ -249,7 +243,7 @@ func (r *RestoreJob) Run(ctx context.Context) (err error) { log.Dbg("Database List to restore: ", dbList) for dbName, dbDefinition := range dbList { - if err := r.restoreDB(ctx, restoreCont.ID, dbName, dbDefinition); err != nil { + if err := r.restoreDB(ctx, containerID, dbName, dbDefinition); err != nil { return errors.Wrap(err, "failed to restore a database") } } @@ -261,11 +255,15 @@ func (r *RestoreJob) Run(ctx context.Context) (err error) { log.Msg("Running analyze command: ", analyzeCmd) - if err := tools.ExecCommand(ctx, r.dockerClient, restoreCont.ID, types.ExecConfig{Cmd: analyzeCmd}); err != nil { + if err := tools.ExecCommand(ctx, r.dockerClient, containerID, types.ExecConfig{Cmd: analyzeCmd}); err != nil { return errors.Wrap(err, "failed to recalculate statistics after restore") } - if err := tools.StopPostgres(ctx, r.dockerClient, restoreCont.ID, dataDir, tools.DefaultStopTimeout); err != nil { + if err := tools.RunCheckpoint(ctx, r.dockerClient, containerID, r.globalCfg.Database.User(), r.globalCfg.Database.DBName); err != nil { + return errors.Wrap(err, "failed to run checkpoint before stop") + } + + if err := tools.StopPostgres(ctx, r.dockerClient, containerID, dataDir, tools.DefaultStopTimeout); err != nil { return errors.Wrap(err, "failed to stop Postgres instance") } diff --git a/engine/internal/retrieval/engine/postgres/logical/restore_test.go b/engine/internal/retrieval/engine/postgres/logical/restore_test.go index de7c95ef..4ec172e6 100644 --- a/engine/internal/retrieval/engine/postgres/logical/restore_test.go +++ b/engine/internal/retrieval/engine/postgres/logical/restore_test.go @@ -122,6 +122,50 @@ func TestRestoreCommandBuilding(t *testing.T) { } } +func TestDumpCommandBuilding(t *testing.T) { + logicalJob := &DumpJob{ + config: dumpJobConfig{ + db: Connection{ + Host: "localhost", + Port: 5432, + DBName: "postgres", + Username: "john", + Password: "secret", + }, + }, + } + + testCases := []struct { + copyOptions DumpOptions + command []string + }{ + { + copyOptions: DumpOptions{ + ParallelJobs: 1, + DumpLocation: "/tmp/db.dump", + Databases: map[string]DumpDefinition{ + "testDB": { + Tables: []string{"test", "users"}, + ExcludeTables: []string{ + "test2", + "users2", + }, + }, + }, + }, + command: []string{"pg_dump", "--create", "--host", "localhost", "--port", "5432", "--username", "john", "--dbname", "testDB", "--jobs", "1", "--table", "test", "--table", "users", "--exclude-table", "test2", "--exclude-table", "users2", "--format", "directory", "--file", "/tmp/db.dump/testDB"}, + }, + } + + for _, tc := range testCases { + logicalJob.DumpOptions = tc.copyOptions + for dbName, definition := range tc.copyOptions.Databases { + dumpCommand := logicalJob.buildLogicalDumpCommand(dbName, definition) + assert.Equal(t, tc.command, dumpCommand) + } + } +} + func TestDiscoverDumpDirectories(t *testing.T) { t.Skip("docker client is required") diff --git a/engine/internal/retrieval/engine/postgres/physical/custom.go b/engine/internal/retrieval/engine/postgres/physical/custom.go index 567e45d3..edc940b0 100644 --- a/engine/internal/retrieval/engine/postgres/physical/custom.go +++ b/engine/internal/retrieval/engine/postgres/physical/custom.go @@ -5,6 +5,8 @@ package physical import ( + "context" + "gitlab.com/postgres-ai/database-lab/v3/internal/retrieval/engine/postgres/tools/defaults" ) @@ -46,3 +48,8 @@ func (c *custom) GetRecoveryConfig(pgVersion float64) map[string]string { return recoveryCfg } + +// Init initialize custom recovery tool to work in provided container. +func (c *custom) Init(ctx context.Context, containerID string) error { + return nil +} diff --git a/engine/internal/retrieval/engine/postgres/physical/pgbackrest.go b/engine/internal/retrieval/engine/postgres/physical/pgbackrest.go index 4e1afcc9..a47db505 100644 --- a/engine/internal/retrieval/engine/postgres/physical/pgbackrest.go +++ b/engine/internal/retrieval/engine/postgres/physical/pgbackrest.go @@ -5,6 +5,7 @@ package physical import ( + "context" "fmt" "gitlab.com/postgres-ai/database-lab/v3/internal/retrieval/engine/postgres/tools/defaults" @@ -54,3 +55,8 @@ func (p *pgbackrest) GetRecoveryConfig(pgVersion float64) map[string]string { return recoveryCfg } + +// Init initialize pgbackrest tool. +func (p *pgbackrest) Init(ctx context.Context, containerID string) error { + return nil +} diff --git a/engine/internal/retrieval/engine/postgres/physical/physical.go b/engine/internal/retrieval/engine/postgres/physical/physical.go index bd03e261..d10f6f52 100644 --- a/engine/internal/retrieval/engine/postgres/physical/physical.go +++ b/engine/internal/retrieval/engine/postgres/physical/physical.go @@ -16,7 +16,6 @@ import ( "github.com/docker/docker/api/types" "github.com/docker/docker/api/types/container" - "github.com/docker/docker/api/types/network" "github.com/docker/docker/client" "github.com/pkg/errors" @@ -93,6 +92,10 @@ type HealthCheck struct { // restorer describes the interface of tools for physical restore. type restorer interface { + + // Init initialize restore tooling inside of container. + Init(ctx context.Context, containerID string) error + // GetRestoreCommand returns a command to restore data. GetRestoreCommand() string @@ -115,7 +118,7 @@ func NewJob(cfg config.JobConfig, global *global.Config, engineProps global.Engi return nil, errors.Wrap(err, "failed to unmarshal configuration options") } - restorer, err := physicalJob.getRestorer(physicalJob.Tool) + restorer, err := physicalJob.getRestorer(cfg.Docker, physicalJob.Tool) if err != nil { return nil, errors.Wrap(err, "failed to init restorer") } @@ -126,10 +129,10 @@ func NewJob(cfg config.JobConfig, global *global.Config, engineProps global.Engi } // getRestorer builds a tool to perform physical restoring. -func (r *RestoreJob) getRestorer(tool string) (restorer, error) { +func (r *RestoreJob) getRestorer(client *client.Client, tool string) (restorer, error) { switch tool { case walgTool: - return newWALG(r.fsPool.DataDir(), r.WALG), nil + return newWALG(client, r.fsPool.DataDir(), r.WALG), nil case pgbackrestTool: return newPgBackRest(r.PgBackRest), nil @@ -215,6 +218,10 @@ func (r *RestoreJob) Run(ctx context.Context) (err error) { return errors.Wrapf(err, "failed to start container: %v", contID) } + if err := r.restorer.Init(ctx, contID); err != nil { + return fmt.Errorf("failed to initialize restorer: %w", err) + } + log.Msg("Running restore command: ", r.restorer.GetRestoreCommand()) log.Msg(fmt.Sprintf("View logs using the command: %s %s", tools.ViewLogsCmd, r.restoreContainerName())) @@ -283,17 +290,17 @@ func (r *RestoreJob) startContainer(ctx context.Context, containerName string, c return "", err } - newContainer, err := r.dockerClient.ContainerCreate(ctx, containerConfig, hostConfig, &network.NetworkingConfig{}, nil, - containerName) + containerID, err := tools.CreateContainerIfMissing(ctx, r.dockerClient, containerName, containerConfig, hostConfig) + if err != nil { - return "", errors.Wrapf(err, "failed to create container %s", containerName) + return "", fmt.Errorf("failed to create container %q %w", containerName, err) } - if err = r.dockerClient.ContainerStart(ctx, newContainer.ID, types.ContainerStartOptions{}); err != nil { + if err = r.dockerClient.ContainerStart(ctx, containerID, types.ContainerStartOptions{}); err != nil { return "", errors.Wrapf(err, "failed to start container %s", containerName) } - return newContainer.ID, nil + return containerID, nil } func (r *RestoreJob) syncInstanceName() string { diff --git a/engine/internal/retrieval/engine/postgres/physical/wal_g.go b/engine/internal/retrieval/engine/postgres/physical/wal_g.go index bb06bbcd..cdb934b8 100644 --- a/engine/internal/retrieval/engine/postgres/physical/wal_g.go +++ b/engine/internal/retrieval/engine/postgres/physical/wal_g.go @@ -5,35 +5,52 @@ package physical import ( + "context" "fmt" + "strings" + "github.com/docker/docker/api/types" + "github.com/docker/docker/client" + "golang.org/x/mod/semver" + + "gitlab.com/postgres-ai/database-lab/v3/internal/retrieval/engine/postgres/tools" "gitlab.com/postgres-ai/database-lab/v3/internal/retrieval/engine/postgres/tools/defaults" + "gitlab.com/postgres-ai/database-lab/v3/pkg/log" ) const ( - walgTool = "walg" + walgTool = "walg" + walgSplitCount = 3 + walg11Version = "v1.1" + latestBackup = "LATEST" ) // walg defines a WAL-G as an archival restoration tool. type walg struct { - pgDataDir string - options walgOptions + dockerClient *client.Client + pgDataDir string + parsedBackupName string + options walgOptions } type walgOptions struct { BackupName string `yaml:"backupName"` } -func newWALG(pgDataDir string, options walgOptions) *walg { - return &walg{ - pgDataDir: pgDataDir, - options: options, +func newWALG(dockerClient *client.Client, pgDataDir string, options walgOptions) *walg { + walg := &walg{ + dockerClient: dockerClient, + pgDataDir: pgDataDir, + options: options, + parsedBackupName: options.BackupName, } + + return walg } // GetRestoreCommand returns a command to restore data. func (w *walg) GetRestoreCommand() string { - return fmt.Sprintf("wal-g backup-fetch %s %s", w.pgDataDir, w.options.BackupName) + return fmt.Sprintf("wal-g backup-fetch %s %s", w.pgDataDir, w.parsedBackupName) } // GetRecoveryConfig returns a recovery config to restore data. @@ -48,3 +65,104 @@ func (w *walg) GetRecoveryConfig(pgVersion float64) map[string]string { return recoveryCfg } + +// Init initializes the wal-g tool to run in the provided container. +func (w *walg) Init(ctx context.Context, containerID string) error { + if strings.ToUpper(w.options.BackupName) != latestBackup { + return nil + } + // workaround for issue with identification of last backup + // https://gitlab.com/postgres-ai/database-lab/-/issues/365 + name, err := getLastBackupName(ctx, w.dockerClient, containerID) + + if err != nil { + return err + } + + if name != "" { + w.parsedBackupName = name + return nil + } + + return fmt.Errorf("failed to fetch last backup name from wal-g") +} + +// getLastBackupName returns the name of the latest backup from the wal-g backup list. +func getLastBackupName(ctx context.Context, dockerClient *client.Client, containerID string) (string, error) { + walgVersion, err := getWalgVersion(ctx, dockerClient, containerID) + + if err != nil { + return "", err + } + + result := semver.Compare(walgVersion, walg11Version) + + // Try to fetch the latest backup with the details command for WAL-G 1.1 and higher + if result >= 0 { + output, err := parseLastBackupFromDetails(ctx, dockerClient, containerID) + + if err == nil { + return output, err + } + + // fallback to fetching last backup from list + log.Err("Failed to parse last backup from wal-g details", err) + } + + return parseLastBackupFromList(ctx, dockerClient, containerID) +} + +// parseLastBackupFromList parses the name of the latest backup from "wal-g backup-list" output. +func parseLastBackupFromList(ctx context.Context, dockerClient *client.Client, containerID string) (string, error) { + output, err := tools.ExecCommandWithOutput(ctx, dockerClient, containerID, types.ExecConfig{ + Cmd: []string{"bash", "-c", "wal-g backup-list | grep base | sort -nk1 | tail -1 | awk '{print $1}'"}, + }) + if err != nil { + return "", err + } + + log.Dbg("The latest WAL-G backup from the list", output) + + return output, nil +} + +// parseLastBackupFromDetails parses the name of the latest backup from "wal-g backup-list --detail" output. +func parseLastBackupFromDetails(ctx context.Context, dockerClient *client.Client, containerID string) (string, error) { + output, err := tools.ExecCommandWithOutput(ctx, dockerClient, containerID, types.ExecConfig{ + Cmd: []string{"bash", "-c", "wal-g backup-list --detail | tail -1 | awk '{print $1}'"}, + }) + if err != nil { + return "", err + } + + log.Dbg("The latest WAL-G backup from list details", output) + + return output, nil +} + +// getWalgVersion fetches the WAL-G version installed in the provided container. +func getWalgVersion(ctx context.Context, dockerClient *client.Client, containerID string) (string, error) { + output, err := tools.ExecCommandWithOutput(ctx, dockerClient, containerID, types.ExecConfig{ + Cmd: []string{"bash", "-c", "wal-g --version"}, + }) + if err != nil { + return "", err + } + + log.Dbg(output) + + return parseWalGVersion(output) +} + +// parseWalGVersion extracts the version from the 'wal-g --version' output. +// For example, "wal-g version v2.0.0 1eb88a5 2022.05.20_10:45:57 PostgreSQL". +func parseWalGVersion(output string) (string, error) { + walgVersion := strings.Split(output, "\t") + versionParts := strings.Split(walgVersion[0], " ") + + if len(versionParts) < walgSplitCount { + return "", fmt.Errorf("failed to extract wal-g version number") + } + + return versionParts[walgSplitCount-1], nil +} diff --git a/engine/internal/retrieval/engine/postgres/physical/wal_g_test.go b/engine/internal/retrieval/engine/postgres/physical/wal_g_test.go index 040b4f25..270b6c23 100644 --- a/engine/internal/retrieval/engine/postgres/physical/wal_g_test.go +++ b/engine/internal/retrieval/engine/postgres/physical/wal_g_test.go @@ -7,7 +7,7 @@ import ( ) func TestWALGRecoveryConfig(t *testing.T) { - walg := newWALG("dataDir", walgOptions{}) + walg := newWALG(nil, "dataDir", walgOptions{}) recoveryConfig := walg.GetRecoveryConfig(11.7) expectedResponse11 := map[string]string{ @@ -22,3 +22,10 @@ func TestWALGRecoveryConfig(t *testing.T) { } assert.Equal(t, expectedResponse12, recoveryConfig) } + +func TestWALGVersionParse(t *testing.T) { + version, err := parseWalGVersion("wal-g version v2.0.0\t1eb88a5\t2022.05.20_10:45:57\tPostgreSQL") + assert.NoError(t, err) + assert.NotEmpty(t, version) + assert.Equal(t, "v2.0.0", version) +} diff --git a/engine/internal/retrieval/engine/postgres/snapshot/logical.go b/engine/internal/retrieval/engine/postgres/snapshot/logical.go index 8f812429..88e7a909 100644 --- a/engine/internal/retrieval/engine/postgres/snapshot/logical.go +++ b/engine/internal/retrieval/engine/postgres/snapshot/logical.go @@ -13,7 +13,6 @@ import ( "github.com/docker/docker/api/types" "github.com/docker/docker/api/types/container" - "github.com/docker/docker/api/types/network" "github.com/docker/docker/client" "github.com/pkg/errors" @@ -214,18 +213,14 @@ func (s *LogicalInitial) runPreprocessingQueries(ctx context.Context, dataDir st } // Run patch container. - patchCont, err := s.dockerClient.ContainerCreate(ctx, - s.buildContainerConfig(dataDir, patchImage, pwd), - hostConfig, - &network.NetworkingConfig{}, - nil, - s.patchContainerName(), - ) + containerID, err := tools.CreateContainerIfMissing(ctx, s.dockerClient, s.patchContainerName(), + s.buildContainerConfig(dataDir, patchImage, pwd), hostConfig) + if err != nil { - return errors.Wrap(err, "failed to create container") + return fmt.Errorf("failed to create container %w", err) } - defer tools.RemoveContainer(ctx, s.dockerClient, patchCont.ID, cont.StopPhysicalTimeout) + defer tools.RemoveContainer(ctx, s.dockerClient, containerID, cont.StopPhysicalTimeout) defer func() { if err != nil { @@ -234,20 +229,20 @@ func (s *LogicalInitial) runPreprocessingQueries(ctx context.Context, dataDir st } }() - log.Msg(fmt.Sprintf("Running container: %s. ID: %v", s.patchContainerName(), patchCont.ID)) + log.Msg(fmt.Sprintf("Running container: %s. ID: %v", s.patchContainerName(), containerID)) - if err := s.dockerClient.ContainerStart(ctx, patchCont.ID, types.ContainerStartOptions{}); err != nil { + if err := s.dockerClient.ContainerStart(ctx, containerID, types.ContainerStartOptions{}); err != nil { return errors.Wrap(err, "failed to start container") } log.Msg("Starting PostgreSQL and waiting for readiness") log.Msg(fmt.Sprintf("View logs using the command: %s %s", tools.ViewLogsCmd, s.patchContainerName())) - if err := tools.CheckContainerReadiness(ctx, s.dockerClient, patchCont.ID); err != nil { + if err := tools.CheckContainerReadiness(ctx, s.dockerClient, containerID); err != nil { return errors.Wrap(err, "failed to readiness check") } - if err := s.queryProcessor.applyPreprocessingQueries(ctx, patchCont.ID); err != nil { + if err := s.queryProcessor.applyPreprocessingQueries(ctx, containerID); err != nil { return errors.Wrap(err, "failed to run preprocessing queries") } diff --git a/engine/internal/retrieval/engine/postgres/snapshot/physical.go b/engine/internal/retrieval/engine/postgres/snapshot/physical.go index 42c84e1f..353788ae 100644 --- a/engine/internal/retrieval/engine/postgres/snapshot/physical.go +++ b/engine/internal/retrieval/engine/postgres/snapshot/physical.go @@ -19,7 +19,6 @@ import ( "github.com/araddon/dateparse" "github.com/docker/docker/api/types" "github.com/docker/docker/api/types/container" - "github.com/docker/docker/api/types/network" "github.com/docker/docker/client" "github.com/pkg/errors" "github.com/robfig/cron/v3" @@ -150,8 +149,10 @@ type syncState struct { } // NewPhysicalInitialJob creates a new physical initial job. -func NewPhysicalInitialJob(cfg config.JobConfig, global *global.Config, engineProps global.EngineProps, cloneManager pool.FSManager, - tm *telemetry.Agent) (*PhysicalInitial, error) { +func NewPhysicalInitialJob( + cfg config.JobConfig, global *global.Config, engineProps global.EngineProps, cloneManager pool.FSManager, + tm *telemetry.Agent, +) (*PhysicalInitial, error) { p := &PhysicalInitial{ name: cfg.Spec.Name, cloneManager: cloneManager, @@ -398,7 +399,13 @@ func (p *PhysicalInitial) checkSyncInstance(ctx context.Context) (string, error) log.Msg("Sync instance has been checked. It is running") - if err := p.checkpoint(ctx, syncContainer.ID); err != nil { + if err := tools.RunCheckpoint( + ctx, + p.dockerClient, + syncContainer.ID, + p.globalCfg.Database.User(), + p.globalCfg.Database.Name(), + ); err != nil { return "", errors.Wrap(err, "failed to make a checkpoint for sync instance") } @@ -444,7 +451,9 @@ func (p *PhysicalInitial) startScheduler(ctx context.Context) { } func (p *PhysicalInitial) waitToStopScheduler() { - <-p.schedulerCtx.Done() + if p.schedulerCtx != nil { + <-p.schedulerCtx.Done() + } if p.scheduler != nil { log.Msg("Stop snapshot scheduler") @@ -539,19 +548,14 @@ func (p *PhysicalInitial) promoteInstance(ctx context.Context, clonePath string, } // Run promotion container. - promoteCont, err := p.dockerClient.ContainerCreate(ctx, - p.buildContainerConfig(clonePath, promoteImage, pwd, recoveryConfig[targetActionOption]), - hostConfig, - &network.NetworkingConfig{}, - nil, - p.promoteContainerName(), - ) + containerID, err := tools.CreateContainerIfMissing(ctx, p.dockerClient, p.promoteContainerName(), + p.buildContainerConfig(clonePath, promoteImage, pwd, recoveryConfig[targetActionOption]), hostConfig) if err != nil { - return errors.Wrap(err, "failed to create container") + return fmt.Errorf("failed to create container %w", err) } - defer tools.RemoveContainer(ctx, p.dockerClient, promoteCont.ID, cont.StopPhysicalTimeout) + defer tools.RemoveContainer(ctx, p.dockerClient, containerID, cont.StopPhysicalTimeout) defer func() { if err != nil { @@ -560,14 +564,14 @@ func (p *PhysicalInitial) promoteInstance(ctx context.Context, clonePath string, } }() - log.Msg(fmt.Sprintf("Running container: %s. ID: %v", p.promoteContainerName(), promoteCont.ID)) + log.Msg(fmt.Sprintf("Running container: %s. ID: %v", p.promoteContainerName(), containerID)) - if err := p.dockerClient.ContainerStart(ctx, promoteCont.ID, types.ContainerStartOptions{}); err != nil { + if err := p.dockerClient.ContainerStart(ctx, containerID, types.ContainerStartOptions{}); err != nil { return errors.Wrap(err, "failed to start container") } if syState.DSA == "" { - dsa, err := p.getDSAFromWAL(ctx, cfgManager.GetPgVersion(), promoteCont.ID, clonePath) + dsa, err := p.getDSAFromWAL(ctx, cfgManager.GetPgVersion(), containerID, clonePath) if err != nil { log.Dbg("cannot extract DSA form WAL files: ", err) } @@ -582,11 +586,11 @@ func (p *PhysicalInitial) promoteInstance(ctx context.Context, clonePath string, log.Msg("Starting PostgreSQL and waiting for readiness") log.Msg(fmt.Sprintf("View logs using the command: %s %s", tools.ViewLogsCmd, p.promoteContainerName())) - if err := tools.CheckContainerReadiness(ctx, p.dockerClient, promoteCont.ID); err != nil { + if err := tools.CheckContainerReadiness(ctx, p.dockerClient, containerID); err != nil { return errors.Wrap(err, "failed to readiness check") } - shouldBePromoted, err := p.checkRecovery(ctx, promoteCont.ID) + shouldBePromoted, err := p.checkRecovery(ctx, containerID) if err != nil { return errors.Wrap(err, "failed to check recovery mode") } @@ -596,11 +600,11 @@ func (p *PhysicalInitial) promoteInstance(ctx context.Context, clonePath string, // Detect dataStateAt. if shouldBePromoted == "t" { // Promote PGDATA. - if err := p.runPromoteCommand(ctx, promoteCont.ID, clonePath); err != nil { + if err := p.runPromoteCommand(ctx, containerID, clonePath); err != nil { return errors.Wrapf(err, "failed to promote PGDATA: %s", clonePath) } - isInRecovery, err := p.checkRecovery(ctx, promoteCont.ID) + isInRecovery, err := p.checkRecovery(ctx, containerID) if err != nil { return errors.Wrap(err, "failed to check recovery mode after promotion") } @@ -610,19 +614,18 @@ func (p *PhysicalInitial) promoteInstance(ctx context.Context, clonePath string, } } - if err := p.markDSA(ctx, syState.DSA, promoteCont.ID, clonePath, cfgManager.GetPgVersion()); err != nil { + if err := p.markDSA(ctx, syState.DSA, containerID, clonePath, cfgManager.GetPgVersion()); err != nil { return errors.Wrap(err, "failed to mark dataStateAt") } if p.queryProcessor != nil { - if err := p.queryProcessor.applyPreprocessingQueries(ctx, promoteCont.ID); err != nil { + if err := p.queryProcessor.applyPreprocessingQueries(ctx, containerID); err != nil { return errors.Wrap(err, "failed to run preprocessing queries") } } - // Checkpoint. - if err := p.checkpoint(ctx, promoteCont.ID); err != nil { - return err + if err := tools.RunCheckpoint(ctx, p.dockerClient, containerID, p.globalCfg.Database.User(), p.globalCfg.Database.Name()); err != nil { + return errors.Wrap(err, "failed to run checkpoint") } if err := cfgManager.RemoveRecoveryConfig(); err != nil { @@ -642,15 +645,18 @@ func (p *PhysicalInitial) promoteInstance(ctx context.Context, clonePath string, return errors.Wrap(err, "failed to store prepared configuration") } - if err := tools.StopPostgres(ctx, p.dockerClient, promoteCont.ID, clonePath, tools.DefaultStopTimeout); err != nil { + if err := tools.StopPostgres(ctx, p.dockerClient, containerID, clonePath, tools.DefaultStopTimeout); err != nil { log.Msg("Failed to stop Postgres", err) - tools.PrintContainerLogs(ctx, p.dockerClient, promoteCont.ID) + tools.PrintContainerLogs(ctx, p.dockerClient, containerID) } return nil } -func (p *PhysicalInitial) getDSAFromWAL(ctx context.Context, pgVersion float64, containerID, cloneDir string) (string, error) { +func (p *PhysicalInitial) getDSAFromWAL(ctx context.Context, pgVersion float64, containerID, cloneDir string) ( + string, + error, +) { log.Dbg(cloneDir) walDirectory := walDir(cloneDir, pgVersion) @@ -696,14 +702,18 @@ func walDir(cloneDir string, pgVersion float64) string { return path.Join(cloneDir, dir) } -func (p *PhysicalInitial) parseWAL(ctx context.Context, containerID string, pgVersion float64, walFilePath string) string { +func (p *PhysicalInitial) parseWAL( + ctx context.Context, + containerID string, + pgVersion float64, + walFilePath string, +) string { cmd := walCommand(pgVersion, walFilePath) output, err := tools.ExecCommandWithOutput(ctx, p.dockerClient, containerID, types.ExecConfig{ Cmd: []string{"sh", "-c", cmd}, }) if err != nil { - log.Dbg("failed to parse WAL: ", err) return "" } @@ -773,7 +783,11 @@ func buildRecoveryConfig(fileConfig, userRecoveryConfig map[string]string) map[s return recoveryConf } -func (p *PhysicalInitial) markDSA(ctx context.Context, defaultDSA, containerID, dataDir string, pgVersion float64) error { +func (p *PhysicalInitial) markDSA( + ctx context.Context, + defaultDSA, containerID, dataDir string, + pgVersion float64, +) error { extractedDataStateAt, err := p.extractDataStateAt(ctx, containerID, dataDir, pgVersion, defaultDSA) if err != nil { if defaultDSA == "" { @@ -900,8 +914,10 @@ and the source doesn't have enough activity. Step 3. Use the timestamp of the latest checkpoint. This is extracted from PGDATA using the pg_controldata utility. Note that this is not an exact value of the latest activity in the source before we took a copy of PGDATA, but we suppose it is not far from it. */ -func (p *PhysicalInitial) extractDataStateAt(ctx context.Context, containerID, dataDir string, pgVersion float64, - defaultDSA string) (string, error) { +func (p *PhysicalInitial) extractDataStateAt( + ctx context.Context, containerID, dataDir string, pgVersion float64, + defaultDSA string, +) (string, error) { output, err := p.getLastXActReplayTimestamp(ctx, containerID) if err != nil { log.Dbg("unable to get last replay timestamp from the promotion container: ", err) @@ -1007,20 +1023,6 @@ func (p *PhysicalInitial) runPromoteCommand(ctx context.Context, containerID, cl return nil } -func (p *PhysicalInitial) checkpoint(ctx context.Context, containerID string) error { - commandCheckpoint := []string{"psql", "-U", p.globalCfg.Database.User(), "-d", p.globalCfg.Database.Name(), "-XAtc", "checkpoint"} - log.Msg("Run checkpoint command", commandCheckpoint) - - output, err := tools.ExecCommandWithOutput(ctx, p.dockerClient, containerID, types.ExecConfig{Cmd: commandCheckpoint}) - if err != nil { - return errors.Wrap(err, "failed to make checkpoint") - } - - log.Msg("Checkpoint result: ", output) - - return nil -} - func (p *PhysicalInitial) markDatabaseData() error { if err := p.dbMarker.CreateConfig(); err != nil { return errors.Wrap(err, "failed to create a DBMarker config of the database") diff --git a/engine/internal/retrieval/engine/postgres/tools/cont/container.go b/engine/internal/retrieval/engine/postgres/tools/cont/container.go index 4dcd289f..92b1c054 100644 --- a/engine/internal/retrieval/engine/postgres/tools/cont/container.go +++ b/engine/internal/retrieval/engine/postgres/tools/cont/container.go @@ -17,8 +17,10 @@ import ( "github.com/docker/go-units" "github.com/pkg/errors" + "gitlab.com/postgres-ai/database-lab/v3/internal/provision/pool" "gitlab.com/postgres-ai/database-lab/v3/internal/retrieval/engine/postgres/tools" "gitlab.com/postgres-ai/database-lab/v3/internal/retrieval/options" + "gitlab.com/postgres-ai/database-lab/v3/pkg/config/global" "gitlab.com/postgres-ai/database-lab/v3/pkg/log" ) @@ -63,7 +65,8 @@ const ( // TODO(akartasov): Control container manager. // StopControlContainers stops control containers run by Database Lab Engine. -func StopControlContainers(ctx context.Context, dockerClient *client.Client, instanceID, dataDir string) error { +func StopControlContainers(ctx context.Context, dockerClient *client.Client, dbCfg *global.Database, instanceID string, + fsm pool.FSManager) error { log.Msg("Stop control containers") list, err := getContainerList(ctx, dockerClient, instanceID, getControlContainerFilters()) @@ -80,14 +83,17 @@ func StopControlContainers(ctx context.Context, dockerClient *client.Client, ins continue } - if shouldStopInternalProcess(controlLabel) { + if shouldStopInternalProcess(controlLabel) && fsm != nil { log.Msg("Stopping control container: ", containerName) - if err := tools.StopPostgres(ctx, dockerClient, controlCont.ID, dataDir, tools.DefaultStopTimeout); err != nil { - log.Msg("Failed to stop Postgres", err) + if err := tools.RunCheckpoint(ctx, dockerClient, controlCont.ID, dbCfg.User(), dbCfg.Name()); err != nil { + log.Msg("Failed to make a checkpoint:", err) tools.PrintContainerLogs(ctx, dockerClient, controlCont.ID) + } - continue + if err := tools.StopPostgres(ctx, dockerClient, controlCont.ID, fsm.Pool().DataDir(), tools.DefaultStopTimeout); err != nil { + log.Msg("Failed to stop Postgres", err) + tools.PrintContainerLogs(ctx, dockerClient, controlCont.ID) } } @@ -113,7 +119,11 @@ func CleanUpControlContainers(ctx context.Context, dockerClient *client.Client, // CleanUpSatelliteContainers removes satellite containers run by Database Lab Engine. func CleanUpSatelliteContainers(ctx context.Context, dockerClient *client.Client, instanceID string) error { log.Msg("Clean up satellite containers") - return cleanUpContainers(ctx, dockerClient, instanceID, getSatelliteContainerFilters()) + + shutdownCtx, shutdownCancel := context.WithTimeout(ctx, StopTimeout) + defer shutdownCancel() + + return cleanUpContainers(shutdownCtx, dockerClient, instanceID, getSatelliteContainerFilters()) } // cleanUpContainers removes containers run by Database Lab Engine. diff --git a/engine/internal/retrieval/engine/postgres/tools/tools.go b/engine/internal/retrieval/engine/postgres/tools/tools.go index a02bc43e..824c3fca 100644 --- a/engine/internal/retrieval/engine/postgres/tools/tools.go +++ b/engine/internal/retrieval/engine/postgres/tools/tools.go @@ -24,6 +24,7 @@ import ( "github.com/docker/docker/api/types" "github.com/docker/docker/api/types/container" "github.com/docker/docker/api/types/mount" + "github.com/docker/docker/api/types/network" "github.com/docker/docker/client" "github.com/docker/docker/errdefs" "github.com/docker/docker/pkg/jsonmessage" @@ -257,6 +258,40 @@ func StartPostgres(ctx context.Context, dockerClient *client.Client, containerID return nil } +// RunCheckpoint runs checkpoint, usually before the postgres stop +func RunCheckpoint( + ctx context.Context, + dockerClient *client.Client, + containerID string, + user string, + database string, +) error { + commandCheckpoint := []string{ + "psql", + "-U", + user, + "-d", + database, + "-XAtc", + "checkpoint", + } + log.Msg("Run checkpoint command", commandCheckpoint) + + output, err := ExecCommandWithOutput( + ctx, + dockerClient, + containerID, + types.ExecConfig{Cmd: commandCheckpoint}, + ) + if err != nil { + return errors.Wrap(err, "failed to make checkpoint") + } + + log.Msg("Checkpoint result: ", output) + + return nil +} + // StopPostgres stops Postgres inside container. func StopPostgres(ctx context.Context, dockerClient *client.Client, containerID, dataDir string, timeout int) error { pgVersion, err := DetectPGVersion(dataDir) @@ -521,3 +556,23 @@ func processAttachResponse(ctx context.Context, reader io.Reader) ([]byte, error return bytes.TrimSpace(outBuf.Bytes()), nil } + +// CreateContainerIfMissing create a new container if there is no other container with the same name, if the container +// exits returns existing container id. +func CreateContainerIfMissing(ctx context.Context, docker *client.Client, containerName string, + config *container.Config, hostConfig *container.HostConfig) (string, error) { + containerData, err := docker.ContainerInspect(ctx, containerName) + + if err == nil { + return containerData.ID, nil + } + + createdContainer, err := docker.ContainerCreate(ctx, config, hostConfig, &network.NetworkingConfig{}, + nil, containerName, + ) + if err != nil { + return "", err + } + + return createdContainer.ID, nil +} diff --git a/engine/internal/runci/source/github.go b/engine/internal/runci/source/github.go index af79cbcf..7354403e 100644 --- a/engine/internal/runci/source/github.go +++ b/engine/internal/runci/source/github.go @@ -44,7 +44,7 @@ func (cp *GHProvider) Download(ctx context.Context, opts Opts, outputFile string log.Dbg(fmt.Sprintf("Download options: %#v", opts)) archiveLink, _, err := cp.client.Repositories.GetArchiveLink(ctx, opts.Owner, opts.Repo, - github.Zipball, &github.RepositoryContentGetOptions{Ref: opts.Ref}, true) + github.Zipball, &github.RepositoryContentGetOptions{Ref: getRunRef(opts)}, true) if err != nil { return errors.Wrap(err, "failed to download content") } @@ -72,6 +72,16 @@ func (cp *GHProvider) Download(ctx context.Context, opts Opts, outputFile string return nil } +func getRunRef(opts Opts) string { + ref := opts.Commit + + if ref == "" { + ref = opts.Ref + } + + return ref +} + // Extract extracts downloaded repository archive. func (cp *GHProvider) Extract(file string) (string, error) { extractDirNameCmd := fmt.Sprintf("unzip -qql %s | head -n1 | tr -s ' ' | cut -d' ' -f5-", file) @@ -85,14 +95,19 @@ func (cp *GHProvider) Extract(file string) (string, error) { log.Dbg("Archive directory: ", string(bytes.TrimSpace(dirName))) - resp, err := exec.Command("unzip", "-d", RepoDir, file).CombinedOutput() + archiveDir, err := os.MkdirTemp(RepoDir, "*_extract") + if err != nil { + return "", err + } + + resp, err := exec.Command("unzip", "-d", archiveDir, file).CombinedOutput() log.Dbg("Response: ", string(resp)) if err != nil { return "", err } - source := path.Join(RepoDir, string(bytes.TrimSpace(dirName))) + source := path.Join(archiveDir, string(bytes.TrimSpace(dirName))) log.Dbg("Source: ", source) return source, nil diff --git a/engine/pkg/config/config.go b/engine/pkg/config/config.go index 62d623fa..6569bdae 100644 --- a/engine/pkg/config/config.go +++ b/engine/pkg/config/config.go @@ -75,11 +75,11 @@ func LoadInstanceID() (string, error) { instanceID = xid.New().String() log.Dbg("no instance_id file was found, generate new instance ID", instanceID) - if err := os.MkdirAll(path.Dir(idFilepath), 0755); err != nil { + if err := os.MkdirAll(path.Dir(idFilepath), 0744); err != nil { return "", fmt.Errorf("failed to make directory meta: %w", err) } - return instanceID, os.WriteFile(idFilepath, []byte(instanceID), 0544) + return instanceID, os.WriteFile(idFilepath, []byte(instanceID), 0644) } return instanceID, fmt.Errorf("failed to load instanceid, %w", err) diff --git a/engine/pkg/util/networks/networks.go b/engine/pkg/util/networks/networks.go index 261a41de..311db071 100644 --- a/engine/pkg/util/networks/networks.go +++ b/engine/pkg/util/networks/networks.go @@ -23,8 +23,8 @@ const ( // InternalType contains name of the internal network type. InternalType = "internal" - // networkPrefix defines a distinctive prefix for internal DLE networks. - networkPrefix = "dle_network_" + // NetworkPrefix defines a distinctive prefix for internal DLE networks. + NetworkPrefix = "dle_network_" ) // Setup creates a new internal Docker network and connects container to it. @@ -164,5 +164,5 @@ func hasContainerConnected(networkResource types.NetworkResource, containerID st } func getNetworkName(instanceID string) string { - return networkPrefix + instanceID + return NetworkPrefix + instanceID } diff --git a/engine/test/1.synthetic.sh b/engine/test/1.synthetic.sh index 49a187a9..aa7d274f 100644 --- a/engine/test/1.synthetic.sh +++ b/engine/test/1.synthetic.sh @@ -10,7 +10,7 @@ export POSTGRES_VERSION="${POSTGRES_VERSION:-13}" export DLE_SERVER_PORT=${DLE_SERVER_PORT:-12345} export DLE_PORT_POOL_FROM=${DLE_PORT_POOL_FROM:-9000} export DLE_PORT_POOL_TO=${DLE_PORT_POOL_TO:-9100} -export DLE_TEST_MOUNT_DIR="/var/lib/test/dblab" +export DLE_TEST_MOUNT_DIR="/var/lib/test/dblab_mount" export DLE_TEST_POOL_NAME="test_dblab_pool" DIR=${0%/*} @@ -91,9 +91,12 @@ yq eval -i ' .databaseContainer.dockerImage = "postgresai/extended-postgres:" + strenv(POSTGRES_VERSION) ' "${configDir}/server.yml" -# logerrors is not supported in PostgreSQL 9.6 +# Edit the following options for PostgreSQL 9.6 if [ "${POSTGRES_VERSION}" = "9.6" ]; then - yq eval -i '.databaseConfigs.configs.shared_preload_libraries = "pg_stat_statements, auto_explain"' "${configDir}/server.yml" + yq eval -i ' + .databaseConfigs.configs.shared_preload_libraries = "pg_stat_statements, auto_explain" | + .databaseConfigs.configs.log_directory = "log" + ' "${configDir}/server.yml" fi ## Launch Database Lab server @@ -161,6 +164,21 @@ dblab clone create \ --password secret_password \ --id testclone +### Check that database system was properly shut down (clone data dir) +CLONE_LOG_DIR="${DLE_TEST_MOUNT_DIR}"/"${DLE_TEST_POOL_NAME}"/clones/dblab_clone_"${DLE_PORT_POOL_FROM}"/data/log +LOG_FILE_CSV=$(sudo ls -t "$CLONE_LOG_DIR" | grep .csv | head -n 1) +if sudo test -d "$CLONE_LOG_DIR" +then + if sudo grep -q 'database system was not properly shut down; automatic recovery in progress' "$CLONE_LOG_DIR"/"$LOG_FILE_CSV" + then + echo "ERROR: database system was not properly shut down" && exit 1 + else + echo "INFO: database system was properly shut down - OK" + fi +else + echo "ERROR: the log directory \"$CLONE_LOG_DIR\" does not exist" && exit 1 +fi + # Connect to a clone and check the available table PGPASSWORD=secret_password psql \ "host=localhost port=${DLE_PORT_POOL_FROM} user=dblab_user_1 dbname=test" -c '\dt+' diff --git a/engine/test/2.logical_generic.sh b/engine/test/2.logical_generic.sh index c5aebe49..ac7c1dfa 100644 --- a/engine/test/2.logical_generic.sh +++ b/engine/test/2.logical_generic.sh @@ -105,9 +105,12 @@ yq eval -i ' .databaseContainer.dockerImage = "postgresai/extended-postgres:" + strenv(POSTGRES_VERSION) ' "${configDir}/server.yml" -# logerrors is not supported in PostgreSQL 9.6 +# Edit the following options for PostgreSQL 9.6 if [ "${POSTGRES_VERSION}" = "9.6" ]; then - yq eval -i '.databaseConfigs.configs.shared_preload_libraries = "pg_stat_statements, auto_explain"' "${configDir}/server.yml" + yq eval -i ' + .databaseConfigs.configs.shared_preload_libraries = "pg_stat_statements, auto_explain" | + .databaseConfigs.configs.log_directory = "log" + ' "${configDir}/server.yml" fi ## Launch Database Lab server @@ -171,6 +174,21 @@ dblab clone create \ --password secret_password \ --id testclone +### Check that database system was properly shut down (clone data dir) +CLONE_LOG_DIR="${DLE_TEST_MOUNT_DIR}"/"${DLE_TEST_POOL_NAME}"/clones/dblab_clone_"${DLE_PORT_POOL_FROM}"/data/log +LOG_FILE_CSV=$(sudo ls -t "$CLONE_LOG_DIR" | grep .csv | head -n 1) +if sudo test -d "$CLONE_LOG_DIR" +then + if sudo grep -q 'database system was not properly shut down; automatic recovery in progress' "$CLONE_LOG_DIR"/"$LOG_FILE_CSV" + then + echo "ERROR: database system was not properly shut down" && exit 1 + else + echo "INFO: database system was properly shut down - OK" + fi +else + echo "ERROR: the log directory \"$CLONE_LOG_DIR\" does not exist" && exit 1 +fi + # Connect to a clone and check the available table PGPASSWORD=secret_password psql \ "host=localhost port=${DLE_PORT_POOL_FROM} user=dblab_user_1 dbname=${SOURCE_DBNAME}" -c '\dt+' @@ -199,5 +217,20 @@ dblab clone list ## Stop DLE. sudo docker stop ${DLE_SERVER_NAME} +### Check that database system was properly shut down (main data dir) +LOG_DIR="${DLE_TEST_MOUNT_DIR}"/"${DLE_TEST_POOL_NAME}"/data/log +LOG_FILE_CSV=$(sudo ls -t "$LOG_DIR" | grep .csv | head -n 1) +if sudo test -d "$LOG_DIR" +then + if [[ $(sudo tail -n 10 "$LOG_DIR"/"$LOG_FILE_CSV" | grep -c 'received fast shutdown request\|database system is shut down') = 2 ]] + then + echo "INFO: database system was properly shut down - OK" + else + echo "ERROR: database system was not properly shut down" && exit 1 + fi +else + echo "ERROR: the log directory \"$LOG_DIR\" does not exist" && exit 1 +fi + ### Finish. clean up source "${DIR}/_cleanup.sh" diff --git a/engine/test/3.physical_walg.sh b/engine/test/3.physical_walg.sh index b1dc932a..5e04fd0a 100644 --- a/engine/test/3.physical_walg.sh +++ b/engine/test/3.physical_walg.sh @@ -8,7 +8,7 @@ DLE_SERVER_NAME="dblab_server_test" # Environment variables for replacement rules export POSTGRES_VERSION="${POSTGRES_VERSION:-13}" export WALG_BACKUP_NAME="${WALG_BACKUP_NAME:-"LATEST"}" -export DLE_TEST_MOUNT_DIR="/var/lib/test/dblab" +export DLE_TEST_MOUNT_DIR="/var/lib/test/dblab_mount" export DLE_TEST_POOL_NAME="test_dblab_pool" export DLE_SERVER_PORT=${DLE_SERVER_PORT:-12345} export DLE_PORT_POOL_FROM=${DLE_PORT_POOL_FROM:-9000} @@ -58,9 +58,14 @@ yq eval -i ' .retrieval.spec.physicalSnapshot.options.skipStartSnapshot = true ' "${configDir}/server.yml" -# logerrors is not supported in PostgreSQL 9.6 +# Edit the following options for PostgreSQL 9.6 if [ "${POSTGRES_VERSION}" = "9.6" ]; then - yq eval -i '.databaseConfigs.configs.shared_preload_libraries = "pg_stat_statements, auto_explain"' "${configDir}/server.yml" + yq eval -i ' + .databaseConfigs.configs.shared_preload_libraries = "pg_stat_statements, auto_explain" | + .databaseConfigs.configs.log_directory = "log" | + .retrieval.spec.physicalRestore.options.sync.configs.log_directory = "log" | + .retrieval.spec.physicalSnapshot.options.promotion.configs.log_directory = "log" + ' "${configDir}/server.yml" fi set +euxo pipefail # ---- do not display secrets @@ -177,6 +182,21 @@ dblab clone create \ --password secret_password \ --id testclone +### Check that database system was properly shut down (clone data dir) +CLONE_LOG_DIR="${DLE_TEST_MOUNT_DIR}"/"${DLE_TEST_POOL_NAME}"/clones/dblab_clone_"${DLE_PORT_POOL_FROM}"/data/log +LOG_FILE_CSV=$(sudo ls -t "$CLONE_LOG_DIR" | grep .csv | head -n 1) +if sudo test -d "$CLONE_LOG_DIR" +then + if sudo grep -q 'database system was not properly shut down; automatic recovery in progress' "$CLONE_LOG_DIR"/"$LOG_FILE_CSV" + then + echo "ERROR: database system was not properly shut down" && exit 1 + else + echo "INFO: database system was properly shut down - OK" + fi +else + echo "ERROR: the log directory \"$CLONE_LOG_DIR\" does not exist" && exit 1 +fi + PGPASSWORD=secret_password psql \ "host=localhost port=${DLE_PORT_POOL_FROM} user=dblab_user_1 dbname=test" -c 'show max_wal_senders' @@ -208,6 +228,21 @@ dblab clone list ## Stop DLE. sudo docker stop ${DLE_SERVER_NAME} +### Check that database system was properly shut down (main data dir) +LOG_DIR="${DLE_TEST_MOUNT_DIR}"/"${DLE_TEST_POOL_NAME}"/data/log +LOG_FILE_CSV=$(sudo ls -t "$LOG_DIR" | grep .csv | head -n 1) +if sudo test -d "$LOG_DIR" +then + if [[ $(sudo tail -n 10 "$LOG_DIR"/"$LOG_FILE_CSV" | grep -c 'received fast shutdown request\|database system is shut down') = 2 ]] + then + echo "INFO: database system was properly shut down - OK" + else + echo "ERROR: database system was not properly shut down" && exit 1 + fi +else + echo "ERROR: the log directory \"$LOG_DIR\" does not exist" && exit 1 +fi + ## Stop control containers. cleanup_service_containers diff --git a/engine/test/4.physical_basebackup.sh b/engine/test/4.physical_basebackup.sh index b14560d7..cc5dbd29 100644 --- a/engine/test/4.physical_basebackup.sh +++ b/engine/test/4.physical_basebackup.sh @@ -122,9 +122,14 @@ yq eval -i ' .retrieval.spec.physicalRestore.options.customTool.command = "pg_basebackup -X stream -D " + strenv(DLE_TEST_MOUNT_DIR) + "/" + strenv(DLE_TEST_POOL_NAME) + "/data" ' "${configDir}/server.yml" -# logerrors is not supported in PostgreSQL 9.6 +# Edit the following options for PostgreSQL 9.6 if [ "${POSTGRES_VERSION}" = "9.6" ]; then - yq eval -i '.databaseConfigs.configs.shared_preload_libraries = "pg_stat_statements, auto_explain"' "${configDir}/server.yml" + yq eval -i ' + .databaseConfigs.configs.shared_preload_libraries = "pg_stat_statements, auto_explain" | + .databaseConfigs.configs.log_directory = "log" | + .retrieval.spec.physicalRestore.options.sync.configs.log_directory = "log" | + .retrieval.spec.physicalSnapshot.options.promotion.configs.log_directory = "log" + ' "${configDir}/server.yml" fi ## Launch Database Lab server @@ -193,6 +198,21 @@ dblab clone create \ --password secret_password \ --id testclone +### Check that database system was properly shut down (clone data dir) +CLONE_LOG_DIR="${DLE_TEST_MOUNT_DIR}"/"${DLE_TEST_POOL_NAME}"/clones/dblab_clone_"${DLE_PORT_POOL_FROM}"/data/log +LOG_FILE_CSV=$(sudo ls -t "$CLONE_LOG_DIR" | grep .csv | head -n 1) +if sudo test -d "$CLONE_LOG_DIR" +then + if sudo grep -q 'database system was not properly shut down; automatic recovery in progress' "$CLONE_LOG_DIR"/"$LOG_FILE_CSV" + then + echo "ERROR: database system was not properly shut down" && exit 1 + else + echo "INFO: database system was properly shut down - OK" + fi +else + echo "ERROR: the log directory \"$CLONE_LOG_DIR\" does not exist" && exit 1 +fi + # Connect to a clone and check the available table PGPASSWORD=secret_password psql \ "host=localhost port=${DLE_PORT_POOL_FROM} user=dblab_user_1 dbname=test" -c '\dt+' @@ -221,6 +241,21 @@ dblab clone list ## Stop DLE. sudo docker stop ${DLE_SERVER_NAME} +### Check that database system was properly shut down (main data dir) +LOG_DIR="${DLE_TEST_MOUNT_DIR}"/"${DLE_TEST_POOL_NAME}"/data/log +LOG_FILE_CSV=$(sudo ls -t "$LOG_DIR" | grep .csv | head -n 1) +if sudo test -d "$LOG_DIR" +then + if [[ $(sudo tail -n 10 "$LOG_DIR"/"$LOG_FILE_CSV" | grep -c 'received fast shutdown request\|database system is shut down') = 2 ]] + then + echo "INFO: database system was properly shut down - OK" + else + echo "ERROR: database system was not properly shut down" && exit 1 + fi +else + echo "ERROR: the log directory \"$LOG_DIR\" does not exist" && exit 1 +fi + ## Stop control containers. cleanup_service_containers diff --git a/engine/test/5.logical_rds.sh b/engine/test/5.logical_rds.sh index a78a8620..eb3f7b8e 100644 --- a/engine/test/5.logical_rds.sh +++ b/engine/test/5.logical_rds.sh @@ -6,7 +6,7 @@ IMAGE2TEST="registry.gitlab.com/postgres-ai/database-lab/dblab-server:${TAG}" DLE_SERVER_NAME="dblab_server_test" # Environment variables for replacement rules -export DLE_TEST_MOUNT_DIR="/var/lib/test/dblab" +export DLE_TEST_MOUNT_DIR="/var/lib/test/dblab_mount" export DLE_TEST_POOL_NAME="test_dblab_pool" export DLE_SERVER_PORT=${DLE_SERVER_PORT:-12345} export DLE_PORT_POOL_FROM=${DLE_PORT_POOL_FROM:-9000} @@ -57,9 +57,12 @@ yq eval -i ' .retrieval.spec.logicalRestore.options.dumpLocation = env(DLE_TEST_MOUNT_DIR) + "/" + env(DLE_TEST_POOL_NAME) + "/dump" ' "${configDir}/server.yml" -# logerrors is not supported in PostgreSQL 9.6 +# Edit the following options for PostgreSQL 9.6 if [ "${POSTGRES_VERSION}" = "9.6" ]; then - yq eval -i '.databaseConfigs.configs.shared_preload_libraries = "pg_stat_statements, auto_explain"' "${configDir}/server.yml" + yq eval -i ' + .databaseConfigs.configs.shared_preload_libraries = "pg_stat_statements, auto_explain" | + .databaseConfigs.configs.log_directory = "log" + ' "${configDir}/server.yml" fi # Download AWS RDS certificate @@ -130,6 +133,21 @@ dblab clone create \ --password secret_password \ --id testclone +### Check that database system was properly shut down (clone data dir) +CLONE_LOG_DIR="${DLE_TEST_MOUNT_DIR}"/"${DLE_TEST_POOL_NAME}"/clones/dblab_clone_"${DLE_PORT_POOL_FROM}"/data/log +LOG_FILE_CSV=$(sudo ls -t "$CLONE_LOG_DIR" | grep .csv | head -n 1) +if sudo test -d "$CLONE_LOG_DIR" +then + if sudo grep -q 'database system was not properly shut down; automatic recovery in progress' "$CLONE_LOG_DIR"/"$LOG_FILE_CSV" + then + echo "ERROR: database system was not properly shut down" && exit 1 + else + echo "INFO: database system was properly shut down - OK" + fi +else + echo "ERROR: the log directory \"$CLONE_LOG_DIR\" does not exist" && exit 1 +fi + # Connect to a clone and check the available table PGPASSWORD=secret_password psql \ "host=localhost port=${DLE_PORT_POOL_FROM} user=dblab_user_1 dbname=${SOURCE_DBNAME}" -c '\dt+' @@ -158,5 +176,20 @@ dblab clone list ## Stop DLE. sudo docker stop ${DLE_SERVER_NAME} +### Check that database system was properly shut down (main data dir) +LOG_DIR="${DLE_TEST_MOUNT_DIR}"/"${DLE_TEST_POOL_NAME}"/data/log +LOG_FILE_CSV=$(sudo ls -t "$LOG_DIR" | grep .csv | head -n 1) +if sudo test -d "$LOG_DIR" +then + if [[ $(sudo tail -n 10 "$LOG_DIR"/"$LOG_FILE_CSV" | grep -c 'received fast shutdown request\|database system is shut down') = 2 ]] + then + echo "INFO: database system was properly shut down - OK" + else + echo "ERROR: database system was not properly shut down" && exit 1 + fi +else + echo "ERROR: the log directory \"$LOG_DIR\" does not exist" && exit 1 +fi + ### Finish. clean up source "${DIR}/_cleanup.sh" diff --git a/packer/install-prereqs.sh b/packer/install-prereqs.sh index 7fadff84..0badab21 100644 --- a/packer/install-prereqs.sh +++ b/packer/install-prereqs.sh @@ -38,7 +38,13 @@ sudo apt-get install -y \ postgresql-client-14 \ s3fs \ yq \ - jq + jq + +# Install cfn-signal helper script +sudo mkdir -p /opt/aws/bin +wget https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz +sudo python3 -m easy_install --script-dir /opt/aws/bin aws-cfn-bootstrap-py3-latest.tar.gz +rm aws-cfn-bootstrap-py3-latest.tar.gz # Install certbot sudo snap install certbot --classic @@ -51,3 +57,10 @@ sudo /usr/local/bin/func-e use 1.19.1 # https://www.envoyproxy.io/docs/envoy/lat # Pull DLE image image_version=$(echo ${dle_version} | sed 's/v*//') sudo docker pull registry.gitlab.com/postgres-ai/database-lab/dblab-server:$image_version +sudo docker pull postgresai/ce-ui:latest +sudo docker pull postgresai/extended-postgres:10 +sudo docker pull postgresai/extended-postgres:11 +sudo docker pull postgresai/extended-postgres:12 +sudo docker pull postgresai/extended-postgres:13 +sudo docker pull postgresai/extended-postgres:14 + diff --git a/ui/packages/platform/src/components/ContentLayout/Footer/index.tsx b/ui/packages/platform/src/components/ContentLayout/Footer/index.tsx index 02eb1007..807b46e2 100644 --- a/ui/packages/platform/src/components/ContentLayout/Footer/index.tsx +++ b/ui/packages/platform/src/components/ContentLayout/Footer/index.tsx @@ -59,7 +59,7 @@ export const Footer = () => { return (
-
2021 © Postgres.ai
+
{new Date().getFullYear()} © Postgres.ai
Documentation @@ -85,12 +85,9 @@ export const Footer = () => {
|
- window.Intercom && window.Intercom('show')} - style={{ cursor: 'pointer' }} - > + Ask support - +
) diff --git a/ui/packages/platform/src/components/IndexPage.js b/ui/packages/platform/src/components/IndexPage.js index 18dcef2f..7c3d91b2 100644 --- a/ui/packages/platform/src/components/IndexPage.js +++ b/ui/packages/platform/src/components/IndexPage.js @@ -954,16 +954,17 @@ function SupportMenu(props) { - window.Intercom && window.Intercom('show')} + activeClassName={props.classes.menuSectionHeaderLink} + target='_blank' + href={settings.rootUrl + '/contact'} > {icons.supportIcon} Ask support - + ); pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy