Skip to content

feat: Decouple workflow dal from bl #5023

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 168 commits into from
Closed

feat: Decouple workflow dal from bl #5023

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
168 commits
Select commit Hold shift + click to select a range
914b9b1
add workflow repository
skynetigor Jun 10, 2025
9ffc97d
add one more function to repository
skynetigor Jun 10, 2025
cda4598
introduce WorkflowDalModel to decouple from DB model WorkflowModel
skynetigor Jun 10, 2025
a13faa8
add types
skynetigor Jun 10, 2025
8aa9a86
add more types
skynetigor Jun 10, 2025
612342a
fixes
skynetigor Jun 10, 2025
037008c
move
skynetigor Jun 10, 2025
597e458
refactor: clean up imports in workflow repository files
skynetigor Jun 11, 2025
6e8d493
Move handling of sqlalchemy exceptions from WorkflowStore to SQL DAL
skynetigor Jun 11, 2025
14afce3
feat: add create_workflow_execution method to WorkflowRepository and …
skynetigor Jun 12, 2025
715b324
feat: implement update_workflow_execution method in WorkflowRepositor…
skynetigor Jun 12, 2025
494c98e
feat: add update_workflow_execution method and refactor SqlWorkflowRe…
skynetigor Jun 12, 2025
791b96f
refactor: update WorkflowStore calls to use instance method for provi…
skynetigor Jun 12, 2025
7bc1a07
make workflow repository injectable to WorkflowStore and WorkflowSche…
skynetigor Jun 12, 2025
c614982
test: mock WorkflowRepository in workflow scheduler tests
skynetigor Jun 12, 2025
d6fd56d
refactor: make workflow_repository parameter optional in WorkflowStor…
skynetigor Jun 12, 2025
e8a81ec
refactor: replace direct database call with repository method in Work…
skynetigor Jun 12, 2025
f502b63
feat: add method to retrieve previous workflow execution in WorkflowR…
skynetigor Jun 12, 2025
75b8ad0
feat: add conflict handling for workflow execution creation and intro…
skynetigor Jun 13, 2025
6d45118
refactor: make all fields in WorkflowExecutionDalModel optional
skynetigor Jun 13, 2025
4460579
feat: implement workflow result serialization and update workflow exe…
skynetigor Jun 13, 2025
cd97aa2
move get_workflows_that_should_run to scheduler
skynetigor Jun 13, 2025
3c451a7
reorganize functions
skynetigor Jun 13, 2025
139221e
Merge branch 'main' into decouple-workflow-dal-from-bl
skynetigor Jun 13, 2025
3775a10
fix: remove exception variable from ConflictError handling
skynetigor Jun 13, 2025
ab4e71d
refactor: simplify event handling in workflow execution status
skynetigor Jun 16, 2025
c0b53fa
feat: move mappers for converting workflow and execution data between…
skynetigor Jun 16, 2025
7a14501
refactor: update workflow execution handling to use DAL model for dat…
skynetigor Jun 16, 2025
907fcef
refactor: update workflow execution update method to accept a patch d…
skynetigor Jun 16, 2025
591e8ba
refactor: remove unused import of WorkflowExecutionDalModel from db.py
skynetigor Jun 16, 2025
762483b
fix: update workflow execution status assignment to use status value
skynetigor Jun 16, 2025
a3abe8b
Merge branch 'main' into decouple-workflow-dal-from-bl
skynetigor Jun 16, 2025
2264140
fix: correct event_id assignment in workflow execution mapping from i…
skynetigor Jun 16, 2025
e56b2b5
Merge branch 'decouple-workflow-dal-from-bl' of https://github.com/ke…
skynetigor Jun 16, 2025
e32f086
feat: add timeout status to workflow execution and update repository …
skynetigor Jun 16, 2025
855c26e
feat: refactor workflow manager to use factory for workflow repositor…
skynetigor Jun 17, 2025
befd953
refactor: update workflow manager instantiation to use singleton pattern
skynetigor Jun 17, 2025
026fdfa
feat: add workflow mapping tests and update workflow_from_db_to_dto f…
skynetigor Jun 17, 2025
758413c
refactor: update WorkflowManager to use create_instance method for in…
skynetigor Jun 17, 2025
9e50f38
Merge branch 'main' into decouple-workflow-dal-from-bl
skynetigor Jun 17, 2025
5d868f7
feat: add end-to-end test for interval workflow and corresponding YAM…
skynetigor Jun 17, 2025
23c4b10
Merge branch 'main' into add_e2e_test_for_interval_workflow
skynetigor Jun 17, 2025
76fcc16
Merge branch 'add_e2e_test_for_interval_workflow' into decouple-workf…
skynetigor Jun 17, 2025
aba7ef6
feat: enhance get_last_completed_execution to include related workflo…
skynetigor Jun 18, 2025
c6cf301
Merge branch 'main' into decouple-workflow-dal-from-bl
skynetigor Jun 18, 2025
3cecf13
refactor: consolidate workflow execution creation into a single method
skynetigor Jun 20, 2025
5db9f78
feat: add create_elastic_client function for flexible Elasticsearch c…
skynetigor Jun 20, 2025
9624c87
feat: implement Elasticsearch workflow repository creation based on e…
skynetigor Jun 20, 2025
a4ba926
feat: implement ElasticSearchWorkflowRepository for managing workflow…
skynetigor Jun 20, 2025
68c3cd7
feat: enhance SQL provider classes with unified literal processing an…
skynetigor Jun 20, 2025
24a8400
feat: enhance workflow data models and queries to include last execut…
skynetigor Jun 20, 2025
06ba693
feat: update SqlWorkflowRepository to use WorkflowWithLastExecutionsD…
skynetigor Jun 20, 2025
ef8994a
feat: add CEL field configurations for Elasticsearch workflow mapping
skynetigor Jun 20, 2025
62f61f3
feat: enhance ElasticSearchWorkflowRepository with improved indexing …
skynetigor Jun 20, 2025
227fa7a
feat: implement workflow execution updates and retrieval in ElasticSe…
skynetigor Jun 20, 2025
546ab2d
move add_or_update workflow logic to WorkflowStore
skynetigor Jun 23, 2025
5037d8f
feat: refactor get_workflow_by_id to use WorkflowRepository for data …
skynetigor Jun 23, 2025
2253df5
feat: rename add_or_update_workflow to add_workflow and introduce get…
skynetigor Jun 23, 2025
f382938
test: mock workflows in session for improved isolation in tests
skynetigor Jun 23, 2025
27c2998
feat: initialize workflow provisioned and is_test attributes in Workf…
skynetigor Jun 23, 2025
d0ef338
feat: implement update_workflow and add_workflow_version methods in S…
skynetigor Jun 23, 2025
df902d2
feat: add lookup_by_name parameter to workflow provisioning and updat…
skynetigor Jun 23, 2025
e103aaa
Merge branch 'main' into decouple-workflow-dal-from-bl
skynetigor Jun 23, 2025
3c1c1ca
refactor: remove unused imports and clean up code in various modules
skynetigor Jun 23, 2025
5cf4b47
refactor: update assertions in test_get_all_workflows_with_last_execu…
skynetigor Jun 23, 2025
d41b0c3
refactor: rename get_workflows_with_last_executions_v2 to get_workflo…
skynetigor Jun 24, 2025
7746de5
feat: add workflow version management with add_workflow_version metho…
skynetigor Jun 24, 2025
19b026b
refactor: update get_workflow_runs_by_id to use workflow_repository f…
skynetigor Jun 24, 2025
c8e1835
feat: add get_workflow_executions method to WorkflowRepository for re…
skynetigor Jun 24, 2025
82cb9b3
refactor: remove unused get_workflow_versions and get_workflow_versio…
skynetigor Jun 24, 2025
fab33d4
decouple workflow routes from direct DB calls
skynetigor Jun 24, 2025
a9d9a47
feat: add update_workflow method to WorkflowStore for updating existi…
skynetigor Jun 24, 2025
1a033e6
feat: add WorkflowStatsDalModel and get_workflow_stats method to Work…
skynetigor Jun 24, 2025
19d0aa8
refactor: remove unused imports from workflows and elasticsearch work…
skynetigor Jun 24, 2025
fff7c23
Merge branch 'main' into decouple-workflow-dal-from-bl
skynetigor Jun 24, 2025
756f463
fix bugs
skynetigor Jun 24, 2025
39e636e
Merge branch 'decouple-workflow-dal-from-bl' of https://github.com/ke…
skynetigor Jun 24, 2025
032b7c8
feat: add workflow statistics retrieval methods and enhance workflow …
skynetigor Jun 24, 2025
72edfbe
refactor: remove unused workflow execution functions for cleaner code
skynetigor Jun 25, 2025
3342b8b
feat: introduce WorkflowStatus enum and update workflow execution ret…
skynetigor Jun 25, 2025
4cc6821
refactor: rename variable for clarity in workflow execution retrieval…
skynetigor Jun 25, 2025
6670609
Merge branch 'main' into decouple-workflow-dal-from-bl
skynetigor Jun 25, 2025
19b3ebe
feat: implement Elasticsearch models for workflow management and logging
skynetigor Jun 25, 2025
f08a689
feat: enhance logging functionality and workflow execution status man…
skynetigor Jun 26, 2025
92adc2c
refactor: remove unused Elasticsearch client and update workflow exec…
skynetigor Jun 26, 2025
0e7f6bd
refactor: remove unused update_workflow_execution function from SQL w…
skynetigor Jun 26, 2025
249010d
fix: handle optional status value in SQL workflow repository
skynetigor Jun 26, 2025
f1514fe
fix: correct return type in workflow_execution_log_from_db_to_dto fun…
skynetigor Jun 26, 2025
0dbfc9d
Merge branch 'main' into decouple-workflow-dal-from-bl
skynetigor Jun 26, 2025
3229d81
refactor: remove unused imports and print statements from logging and…
skynetigor Jun 26, 2025
666c18d
refactor: remove unused Integer import from workflow_execution_log.py
skynetigor Jun 27, 2025
abd1a7a
fix: update WorkflowStatus.ERROR to use the correct value in SQL work…
skynetigor Jun 27, 2025
328603d
Merge branch 'main' into decouple-workflow-dal-from-bl
skynetigor Jun 27, 2025
f366bf5
feat: enhance workflow statistics aggregation in Elasticsearch reposi…
skynetigor Jun 27, 2025
0a88a8a
refactor: remove index_suffix parameter from ElasticSearchWorkflowRep…
skynetigor Jun 27, 2025
62b1cfe
Merge branch 'main' into decouple-workflow-dal-from-bl
skynetigor Jun 27, 2025
61adb39
fix: ensure workflow name is correctly assigned when adding or updati…
skynetigor Jun 27, 2025
245e0bc
Merge branch 'decouple-workflow-dal-from-bl' of https://github.com/ke…
skynetigor Jun 27, 2025
bd8962a
feat: add E2E tests running when ElasticSearch is WF Engine storage
skynetigor Jun 27, 2025
035aa72
fix: ensure elasticsearch-wf-engine service dependency is healthy in …
skynetigor Jun 27, 2025
a0cbf7d
fix: add healthcheck for elasticsearch service in E2E tests
skynetigor Jun 27, 2025
78f091f
fix: remove healthcheck from elasticsearch service in E2E tests and s…
skynetigor Jun 27, 2025
d475f04
fix: add healthcheck for elasticsearch service in E2E tests
skynetigor Jun 27, 2025
01b02f5
fix: reduce Java memory options for Elasticsearch in E2E tests
skynetigor Jun 27, 2025
edced8f
fx
skynetigor Jun 27, 2025
d9a5448
fix: update E2E test workflow to check Elasticsearch logs instead of …
skynetigor Jun 27, 2025
d4aab23
fix: add Elasticsearch workflow engine service to docker-compose and …
skynetigor Jun 27, 2025
ca354a5
fix: update E2E test workflows to use 'docker compose' command and co…
skynetigor Jun 27, 2025
0513865
fix: remove redundant container log checks and ensure service depende…
skynetigor Jun 27, 2025
26e190f
fix: add health check for Elasticsearch client and simplify service d…
skynetigor Jun 27, 2025
5ccc904
fix: improve Elasticsearch client initialization with retry logic for…
skynetigor Jun 27, 2025
41aeac5
Merge branch 'main' into decouple-workflow-dal-from-bl
skynetigor Jun 29, 2025
35cd9c1
fix: enhance Elasticsearch client initialization with improved error …
skynetigor Jun 29, 2025
6fe8421
Merge branch 'decouple-workflow-dal-from-bl' of https://github.com/ke…
skynetigor Jun 29, 2025
8678252
fix: simplify status checks in get_all_workflows method
skynetigor Jun 30, 2025
b866f61
fix: update condition to use 'is False' for clarity in get_all_workfl…
skynetigor Jun 30, 2025
a82213f
fix: update current time retrieval to use timezone-aware datetime in …
skynetigor Jun 30, 2025
6eb4578
fix: bug in getting last completed workflow executions
skynetigor Jun 30, 2025
8ae5244
fix: refactor index usage to directly reference WorkflowDoc.Index.nam…
skynetigor Jun 30, 2025
d55f839
fix: restore and configure E2E test jobs for MySQL, PostgreSQL, and S…
skynetigor Jun 30, 2025
b96cc61
fix: rename create_workflow_execution method to add_workflow_executio…
skynetigor Jun 30, 2025
6457ae4
revert
skynetigor Jun 30, 2025
87d63c7
fix: update condition to use equality for is_test_run and set timezon…
skynetigor Jun 30, 2025
f8bcdf3
fix: enable refresh on save operations and improve is_test_run filter…
skynetigor Jun 30, 2025
3b7ce02
fix: update query to filter enabled workflows correctly
skynetigor Jun 30, 2025
bcf477c
fix: enable refresh on save in ElasticSearchWorkflowRepository and im…
skynetigor Jun 30, 2025
6cf4ef8
refactor add_workflow_execution function
skynetigor Jun 30, 2025
8308d57
fix: remove unused WorkflowStatus import and update test_run to is_te…
skynetigor Jun 30, 2025
e30a5c3
Merge branch 'main' into decouple-workflow-dal-from-bl
skynetigor Jun 30, 2025
365c9d5
refactor: remove unused get_all_workflows_yamls method and update wor…
skynetigor Jun 30, 2025
0511b14
refactor: remove unused get_all_workflows method and update workflow …
skynetigor Jun 30, 2025
70eb46a
Merge branch 'decouple-workflow-dal-from-bl' of https://github.com/ke…
skynetigor Jun 30, 2025
515950d
refactor: remove unused get_all_provisioned_workflows method and upda…
skynetigor Jun 30, 2025
89fc5d8
refactor: update workflow provisioned filter in base query
skynetigor Jun 30, 2025
018a539
Merge branch 'main' into decouple-workflow-dal-from-bl
skynetigor Jun 30, 2025
92b1756
refactor: simplify provisioned workflows retrieval in WorkflowStore
skynetigor Jun 30, 2025
77df62f
Merge branch 'decouple-workflow-dal-from-bl' of https://github.com/ke…
skynetigor Jun 30, 2025
008cebf
refactor: standardize provisioned filter naming in queries
skynetigor Jul 1, 2025
0e2c04a
refactor: update filter defaults to None in workflow repository methods
skynetigor Jul 1, 2025
4e77d95
refactor: improve argument formatting in workflow execution methods a…
skynetigor Jul 1, 2025
d0d235a
refactor: streamline base query construction in __build_base_query fu…
skynetigor Jul 1, 2025
ca4df51
refactor: simplify latest executions subquery construction in __build…
skynetigor Jul 1, 2025
a0fd14d
refactor: finalize latest executions subquery construction in __build…
skynetigor Jul 1, 2025
0085196
refactor: enhance latest executions subquery construction in __build_…
skynetigor Jul 1, 2025
e75cd58
refactor: change latest executions subquery to use CTE in __build_bas…
skynetigor Jul 1, 2025
1e64df7
1
skynetigor Jul 1, 2025
6917fea
refactor: remove fetch_last_executions parameter and simplify base qu…
skynetigor Jul 1, 2025
4137a44
refactor: comment out unused code in __build_base_query function for …
skynetigor Jul 1, 2025
d914419
refactor: remove commented-out code in __build_base_query function fo…
skynetigor Jul 1, 2025
bdf75b0
refactor: specify MySQL image version as 8.0 in docker-compose for co…
skynetigor Jul 1, 2025
a3f481f
refactor: update MySQL image version to 9.3.0 in docker-compose for t…
skynetigor Jul 1, 2025
2b325aa
refactor: handle exceptions when retrieving provisioned workflows in …
skynetigor Jul 1, 2025
58948ea
refactor: streamline retrieval of provisioned workflows in WorkflowStore
skynetigor Jul 1, 2025
68c609f
refactor: check MySQL dialect before retrieving version in get_workfl…
skynetigor Jul 1, 2025
66d0a59
refactor: implement get_workflows method in workflow repositories for…
skynetigor Jul 1, 2025
67b217e
refactor: remove name_filter parameter from get_workflows method in w…
skynetigor Jul 1, 2025
e4380ca
refactor: add provisioning_check parameter to workflow methods for en…
skynetigor Jul 1, 2025
64c0962
refactor: enhance docstrings in WorkflowRepository for better clarity…
skynetigor Jul 1, 2025
fdc5756
refactor: remove delete_workflow_by_provisioned_file method and updat…
skynetigor Jul 1, 2025
e372226
refactor: improve variable naming for clarity in ElasticSearchWorkflo…
skynetigor Jul 1, 2025
3a56926
refactor: streamline document saving in ElasticSearchWorkflowReposito…
skynetigor Jul 1, 2025
7ce41ac
Merge branch 'main' into decouple-workflow-dal-from-bl
skynetigor Jul 1, 2025
e70b716
refactor: remove MySQL version check from get_workflows_with_last_exe…
skynetigor Jul 1, 2025
bd5b5ec
Merge branch 'decouple-workflow-dal-from-bl' of https://github.com/ke…
skynetigor Jul 1, 2025
34e69fb
Refactor code structure for improved readability and maintainability
skynetigor Jul 1, 2025
ab5b288
revert
skynetigor Jul 1, 2025
958ac6b
revert
skynetigor Jul 1, 2025
97e15d3
fix
skynetigor Jul 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
fix
  • Loading branch information
@skynetigor
skynetigor committed Jul 1, 2025
commit 97e15d3c63b94703e6bb8bb6688688216b3d351b
Loading
Loading
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy