feat: auto-downgrading revision #4678

korolenkowork · 2025-05-01T21:22:44Z

Closes #4665

📑 Description

This PR introduces automatic database downgrading support when rolling back KeepHQ to a previous version.

✅ Checks

ℹ Additional Information

From this commit onward, all migration files will be copied to the SECRET_MANAGER_DIRECTORY to enable safe database downgrades during rollback scenarios.
A new environment variable ALLOW_DB_DOWNGRADE has been added to explicitly allow or prevent database downgrades, reducing the risk of accidental downgrades.

vercel · 2025-05-01T21:22:48Z

@EnotShow is attempting to deploy a commit to the KeepHQ Team on Vercel.

A member of the Team first needs to authorize it.

shahargl · 2025-05-02T08:13:43Z

hey @EnotShow - thanks for this PR!

as this PR introduces some major changes to how Keep works, I thought I would add some e2e tests for it (mainly install Keep, do some migration, then downgrade). wdyt?

shahargl

see comments

codecov · 2025-05-03T07:03:22Z

Codecov Report

Attention: Patch coverage is 8.45070% with 65 lines in your changes missing coverage. Please review.

Project coverage is 46.03%. Comparing base (1ae49bf) to head (f1328ef).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
keep/api/core/db_on_start.py	8.45%	65 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4678      +/-   ##
==========================================
- Coverage   46.19%   46.03%   -0.16%     
==========================================
  Files         165      165              
  Lines       17495    17564      +69     
==========================================
+ Hits         8081     8086       +5     
- Misses       9414     9478      +64

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…EnotShow/keep into feature/auto-downgrade-revision

korolenkowork · 2025-05-19T13:37:40Z

Hey @shahargl! Thx for your review! I made all the required changes, so it's ready for re-review :)

Signed-off-by: Ihor <88674695+EnotShow@users.noreply.github.com>

shahargl · 2025-05-25T13:09:41Z

@EnotShow let tests pass and if they will I'll review

korolenkowork · 2025-05-29T17:42:10Z

Hey @shahargl, it's resolved!

I also added a separate Docker Compose file for E2E migrations testing, which attaches a dedicated volume. Because it seems that accessing the same volume from different workflows was causing concurrent map writes, like this.

Signed-off-by: Ihor Korolenko <88674695+korolenkowork@users.noreply.github.com>

cursor · 2025-06-14T20:39:55Z

🚨 BugBot couldn't run

Pull requests from forked repositories are not yet supported (requestId: serverGenReqId_48ddba26-49ed-4dcf-a98c-b6051afed68d).

tuantran0910 · 2025-06-17T09:13:01Z

Hi @shahargl, @talboren, can we get this PR merged ? I think it would be a good feature and we are planning to deploy Keep in our production environment.

korolenkowork · 2025-07-03T13:21:48Z

@shahargl @talboren any updates on this?

cursor

Bug: Migration Function Fails with Incomplete Cleanup

The copy_migrations function has insufficient error handling and an incomplete cleanup mechanism. If os.makedirs fails, the function continues execution, causing subsequent file operations to fail. Furthermore, the cleanup logic only removes files and symlinks, leaving subdirectories. This incomplete cleanup, coupled with clearing the destination before verifying the source, risks data loss of previous backups if the subsequent copy operation fails or conflicts arise.

keep/api/core/db_on_start.py#L183-L204

keep/keep/api/core/db_on_start.py

Lines 183 to 204 in 71e81be

    
           # Ensure destination exists 
        
           try: 
        
               os.makedirs(local_migrations_path, exist_ok=True) 
        
           except Exception as e: 
        
               logger.error(f"Failed to create local migrations folder with error: {e}") 
        
           # Clear previous versioned migrations to ensure only migrations relevant to the current version are present 
        
           for filename in os.listdir(local_migrations_path): 
        
               file_path = os.path.join(local_migrations_path, filename) 
        
               if os.path.isfile(file_path) or os.path.islink(file_path): 
        
                   os.remove(file_path) 
        
           # Alembic needs the full migration history to safely perform a downgrade to earlier versions 
        
           # Copy new migrations 
        
           for item in os.listdir(source_versions_path): 
        
               src = os.path.join(source_versions_path, item) 
        
               dst = os.path.join(local_migrations_path, item) 
        
               if os.path.isdir(src): 
        
                   shutil.copytree(src, dst, dirs_exist_ok=True) 
        
               else: 
        
                   shutil.copy(src, dst)

Fix in Cursor

Bug: Environment Variable Name Discrepancy

Environment variable name mismatch: The docker-compose files set MIGRATION_PATH (singular), but the Python code in keep/api/core/db_on_start.py expects MIGRATIONS_PATH (plural). This causes the application to ignore the intended /tmp/migrations path and fall back to the default /tmp/keep/migrations for database migrations.

tests/e2e_tests/docker-compose-e2e-postgres.yml#L41-L42

keep/tests/e2e_tests/docker-compose-e2e-postgres.yml

Lines 41 to 42 in 71e81be

    
           - SECRET_MANAGER_DIRECTORY=/app 
        
           - MIGRATION_PATH=/tmp/migrations

tests/e2e_tests/docker-compose-e2e-postgres-migrations.yml#L40-L41

keep/tests/e2e_tests/docker-compose-e2e-postgres-migrations.yml

Lines 40 to 41 in 71e81be

    
           - SECRET_MANAGER_DIRECTORY=/app 
        
           - MIGRATION_PATH=/tmp/migrations

Fix in Cursor

Bug: Database Migration Error Handling Flaws

The migrate_db function incorrectly assumes any exception during alembic.command.upgrade() indicates the database needs a downgrade, potentially triggering inappropriate downgrade attempts for unrelated issues (e.g., connectivity, permissions, syntax errors). Additionally, copy_migrations is unconditionally called at the end of migrate_db, which can corrupt the local migration backup if a downgrade operation failed, preventing future successful downgrades.

keep/api/core/db_on_start.py#L275-L289

keep/keep/api/core/db_on_start.py

Lines 275 to 289 in 71e81be

    
           logger.info("Running migrations...") 
        
           try: 
        
               alembic.command.upgrade(config, "head") 
        
           except Exception as e: 
        
               logger.error(f"{e} it's seems like Keep was rolled back to a previous version") 
        
               if not os.getenv("ALLOW_DB_DOWNGRADE", "false") == "true": 
        
                   logger.error(f"ALLOW_DB_DOWNGRADE is not set to true, but the database schema ({current_revision}) doesn't match application version ({expected_revision})") 
        
                   raise RuntimeError("Database downgrade is not allowed") 
        
               logger.info("Downgrading database schema...") 
        
               downgrade_db(config, expected_revision, local_migrations_path, app_migrations_path) 
        
           # Copy migrations to local folder for safe downgrade purposes 
        
           copy_migrations(app_migrations_path, local_migrations_path)

Fix in Cursor

Bug: Missing Step ID Causes Empty Revision Output

The workflow references steps.get_revision.outputs.revision to set the WORKFLOW_REVISION environment variable. However, no step in the workflow has the ID get_revision. This prevents the revision output from being captured, resulting in an empty WORKFLOW_REVISION and breaking the intended revision comparison logic.

.github/workflows/run-migrations-e2e-tests.yml#L335-L336

keep/.github/workflows/run-migrations-e2e-tests.yml

Lines 335 to 336 in 71e81be

    
                   env: 
        
                     WORKFLOW_REVISION: ${{ steps.get_revision.outputs.revision }}

.github/workflows/run-migrations-e2e-tests.yml#L479-L480

keep/.github/workflows/run-migrations-e2e-tests.yml

Lines 479 to 480 in 71e81be

    
                   env: 
        
                     WORKFLOW_REVISION: ${{ steps.get_revision.outputs.revision }}

Fix in Cursor

Was this report helpful? Give feedback by reacting with 👍 or 👎

shahargl · 2025-07-03T19:33:23Z

Hey @korolenkowork, honestly it looks good but it requires some big effort for me to review it’s ready. This one touches the main flow so I’m bit worried.

korolenkowork added 4 commits May 1, 2025 21:14

first version

408e7fc

fix

dce47b0

Main logic completed

6127d86

minor

8cb9b81

dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. Feature A new feature labels May 1, 2025

korolenkowork marked this pull request as draft May 1, 2025 21:23

korolenkowork added 5 commits May 2, 2025 00:03

Update configuration.mdx

9cc98a8

lint fix

2758d9f

Add test

e1ee04b

fix path

51787ba

fix

db174f9

shahargl requested changes May 2, 2025

View reviewed changes

korolenkowork added 7 commits May 2, 2025 14:58

migrations-e2e-prepare-for-coding

9b2f67f

migrations-e2e

3fb2996

Done!

08db877

fix pull old version e2e

4f635f3

fix

f140de4

Update run-migrations-e2e-tests.yml

995fd83

Update run-migrations-e2e-tests.yml

1d1e462

korolenkowork and others added 7 commits May 3, 2025 09:33

fix

5dd3bfa

Update test-pr-e2e.yml

cb46278

Update run-migrations-e2e-tests.yml

bf1b653

Merge branch 'main' into feature/auto-downgrade-revision

f469bab

Check revision between runs

3e3975c

Merge branch 'feature/auto-downgrade-revision' of https://github.com/…

4671d62

…EnotShow/keep into feature/auto-downgrade-revision

fix execute container name

c55103a

dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels May 19, 2025

korolenkowork added 3 commits May 19, 2025 14:05

fix

49df674

Merge branch 'feature/auto-downgrade-revision' of https://github.com/…

077ba69

…EnotShow/keep into feature/auto-downgrade-revision

Add migration path to dockerfiles

ed91ff4

korolenkowork requested a review from shahargl May 19, 2025 13:37

korolenkowork and others added 2 commits May 23, 2025 00:07

Merge branch 'main' into feature/auto-downgrade-revision

f1328ef

Signed-off-by: Ihor <88674695+EnotShow@users.noreply.github.com>

Merge branch 'main' into feature/auto-downgrade-revision

bc5ebce

Merge branch 'main' into feature/auto-downgrade-revision

02ca8bf

dosubot bot added size:XS This PR changes 0-9 lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels May 26, 2025

Merge branch 'main' into feature/auto-downgrade-revision

db76ae7

dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels May 29, 2025

Make separate docker compose for migrations e2e

c05e35e

korolenkowork added 2 commits June 9, 2025 07:56

Merge branch 'main' into feature/auto-downgrade-revision

aea9b81

Merge branch 'main' into feature/auto-downgrade-revision

356ce18

Signed-off-by: Ihor Korolenko <88674695+korolenkowork@users.noreply.github.com>

Update poetry.lock

13fe761

This comment was marked as outdated.

Sign in to view

Merge branch 'main' into feature/auto-downgrade-revision

96d9479

This comment was marked as outdated.

Sign in to view

Merge branch 'main' into feature/auto-downgrade-revision

71e81be

cursor bot reviewed Jul 3, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: auto-downgrading revision #4678

feat: auto-downgrading revision #4678

Uh oh!

korolenkowork commented May 1, 2025 •

edited

Loading

Uh oh!

vercel bot commented May 1, 2025

Uh oh!

shahargl commented May 2, 2025

Uh oh!

shahargl left a comment

Uh oh!

codecov bot commented May 3, 2025 •

edited

Loading

Uh oh!

korolenkowork commented May 19, 2025

Uh oh!

shahargl commented May 25, 2025

Uh oh!

korolenkowork commented May 29, 2025

Uh oh!

cursor bot commented Jun 14, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

tuantran0910 commented Jun 17, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

korolenkowork commented Jul 3, 2025

Uh oh!

cursor bot left a comment

Uh oh!

shahargl commented Jul 3, 2025

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

	# Ensure destination exists
	try:
	os.makedirs(local_migrations_path, exist_ok=True)
	except Exception as e:
	logger.error(f"Failed to create local migrations folder with error: {e}")


	# Clear previous versioned migrations to ensure only migrations relevant to the current version are present
	for filename in os.listdir(local_migrations_path):
	file_path = os.path.join(local_migrations_path, filename)
	if os.path.isfile(file_path) or os.path.islink(file_path):
	os.remove(file_path)

	# Alembic needs the full migration history to safely perform a downgrade to earlier versions
	# Copy new migrations
	for item in os.listdir(source_versions_path):
	src = os.path.join(source_versions_path, item)
	dst = os.path.join(local_migrations_path, item)
	if os.path.isdir(src):
	shutil.copytree(src, dst, dirs_exist_ok=True)
	else:
	shutil.copy(src, dst)

	- SECRET_MANAGER_DIRECTORY=/app
	- MIGRATION_PATH=/tmp/migrations

	logger.info("Running migrations...")
	try:
	alembic.command.upgrade(config, "head")
	except Exception as e:
	logger.error(f"{e} it's seems like Keep was rolled back to a previous version")

	if not os.getenv("ALLOW_DB_DOWNGRADE", "false") == "true":
	logger.error(f"ALLOW_DB_DOWNGRADE is not set to true, but the database schema ({current_revision}) doesn't match application version ({expected_revision})")
	raise RuntimeError("Database downgrade is not allowed")

	logger.info("Downgrading database schema...")
	downgrade_db(config, expected_revision, local_migrations_path, app_migrations_path)

	# Copy migrations to local folder for safe downgrade purposes
	copy_migrations(app_migrations_path, local_migrations_path)

	env:
	WORKFLOW_REVISION: ${{ steps.get_revision.outputs.revision }}

feat: auto-downgrading revision #4678

Are you sure you want to change the base?

feat: auto-downgrading revision #4678

Uh oh!

Conversation

korolenkowork commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📑 Description

✅ Checks

ℹ Additional Information

Uh oh!

vercel bot commented May 1, 2025

Uh oh!

shahargl commented May 2, 2025

Uh oh!

shahargl left a comment

Choose a reason for hiding this comment

Uh oh!

codecov bot commented May 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

korolenkowork commented May 19, 2025

Uh oh!

shahargl commented May 25, 2025

Uh oh!

korolenkowork commented May 29, 2025

Uh oh!

cursor bot commented Jun 14, 2025

🚨 BugBot couldn't run

Uh oh!

This comment was marked as outdated.

Uh oh!

tuantran0910 commented Jun 17, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

korolenkowork commented Jul 3, 2025

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Bug: Migration Function Fails with Incomplete Cleanup

Bug: Environment Variable Name Discrepancy

Bug: Database Migration Error Handling Flaws

Bug: Missing Step ID Causes Empty Revision Output

Uh oh!

shahargl commented Jul 3, 2025

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

korolenkowork commented May 1, 2025 •

edited

Loading

codecov bot commented May 3, 2025 •

edited

Loading