python-ellar · eadwinCode · Jan 27, 2024 · Jan 21, 2024 · Jan 21, 2024 · Jan 21, 2024
diff --git a/Makefile b/Makefile
@@ -9,6 +9,7 @@ clean: ## Removing cached python compiled files
 	find . -name \*pyo | xargs  rm -fv
 	find . -name \*~  | xargs  rm -fv
 	find . -name __pycache__  | xargs  rm -rfv
+	find . -name .pytest_cache  | xargs  rm -rfv
 	find . -name .ruff_cache  | xargs  rm -rfv
 
 install: ## Install dependencies
@@ -23,14 +24,14 @@ lint:fmt ## Run code linters
 	mypy ellar_sql
 
 fmt format:clean ## Run code formatters
-	ruff format ellar_sql tests
-	ruff check --fix ellar_sql tests
+	ruff format ellar_sql tests examples
+	ruff check --fix ellar_sql tests examples
 
-test: ## Run tests
-	pytest tests
+test:clean ## Run tests
+	pytest
 
-test-cov: ## Run tests with coverage
-	pytest --cov=ellar_sql --cov-report term-missing tests
+test-cov:clean ## Run tests with coverage
+	pytest --cov=ellar_sql --cov-report term-missing
 
 pre-commit-lint: ## Runs Requires commands during pre-commit
 	make clean

diff --git a/README.md b/README.md
@@ -2,43 +2,30 @@
   <a href="#" target="blank"><img src="https://python-ellar.github.io/ellar/img/EllarLogoB.png" width="200" alt="Ellar Logo" /></a>
 </p>
 
-![Test](https://github.com/python-ellar/ellar-sqlachemy/actions/workflows/test_full.yml/badge.svg)
-![Coverage](https://img.shields.io/codecov/c/github/python-ellar/ellar-sqlalchemy)
-[![PyPI version](https://badge.fury.io/py/ellar-sqlachemy.svg)](https://badge.fury.io/py/ellar-sqlachemy)
-[![PyPI version](https://img.shields.io/pypi/v/ellar-sqlachemy.svg)](https://pypi.python.org/pypi/ellar-sqlachemy)
-[![PyPI version](https://img.shields.io/pypi/pyversions/ellar-sqlachemy.svg)](https://pypi.python.org/pypi/ellar-sqlachemy)
-
-## Project Status
-
-- [ ] Overall Completion - 85% done
-- [ ] Tests - 90% Complete
-- [x] Model class that transforms to SQLAlchemy Models or DeclarativeBase based on configuration
-- [x] Pydantic-like way to exporting model to dictionary object eg:`model.dict(exclude={'x', 'y', 'z'})`
-- [x] Support multiple database useful in models and queries
-- [x] Session management during request scope and outside request
-- [x] Service to manage SQLAlchemy Engine and Session creation, and Migration for async and sync Engines and Sessions
-- [x] Async first approach to database migration using Alembic
-- [x] Expose all Alembic commands to Ellar CLI 
-- [x] Module to config and setup SQLAlchemy dependencies and migration path
-- [x] SQLAlchemy Pagination for both Templating and API routes
-- [x] File and Image SQLAlchemy Columns integration with ellar storage
-- [ ] SQLAlchemy Django Like Query
-- [ ] Documentation
+![Test](https://github.com/python-ellar/ellar-sql/actions/workflows/test_full.yml/badge.svg)
+![Coverage](https://img.shields.io/codecov/c/github/python-ellar/ellar-sql)
+[![PyPI version](https://badge.fury.io/py/ellar-sql.svg)](https://badge.fury.io/py/ellar-sql)
+[![PyPI version](https://img.shields.io/pypi/v/ellar-sql.svg)](https://pypi.python.org/pypi/ellar-sql)
+[![PyPI version](https://img.shields.io/pypi/pyversions/ellar-sql.svg)](https://pypi.python.org/pypi/ellar-sql)
+
 
 ## Introduction
-Ellar SQLAlchemy Module simplifies the integration of SQLAlchemy and Alembic migration tooling into your ellar application.
+EllarSQL Module adds support for `SQLAlchemy` and `Alembic` package to your Ellar application
 
 ## Installation
 ```shell
-$(venv) pip install ellar-sqlalchemy
+$(venv) pip install ellar-sql
 ```
 
+This library was inspired by [Flask-SQLAlchemy](https://flask-sqlalchemy.palletsprojects.com/en/3.1.x/){target="_blank"}
+
 ## Features
-- Automatic table name
-- Session management during request and after request
-- Support both async/sync SQLAlchemy operations in Session, Engine, and Connection.
-- Multiple Database Support
-- Database migrations for both single and multiple databases either async/sync database engine
+
+- Migration
+- Single/Multiple Database
+- Pagination
+- Compatible with SQLAlchemy tools
+
 
 ## **Usage**
 In your ellar application, create a module called `db` or any name of your choice,
@@ -56,7 +43,7 @@ from ellar_sql.model import Model
 
 
 class Base(Model):
-  __base_config__ = {'make_declarative_base': True}
+  __base_config__ = {'as_base': True}
   __database__ = 'default'
 
   created_date: Mapped[datetime] = mapped_column(
@@ -104,7 +91,7 @@ from .controllers import DbController
             }, 
             echo=True, 
             migration_options={
-                'directory': '__main__/migrations'
+                'directory': 'my_migrations_folder'
             },
             models=['db.models.users']
         )

diff --git a/docs/advance/index.md b/docs/advance/index.md
diff --git a/docs/index.md b/docs/index.md
@@ -0,0 +1,112 @@
+<style>
+.md-content .md-typeset h1 { display: none; }
+</style>
+<p align="center">
+  <a href="#" target="blank"><img src="https://python-ellar.github.io/ellar/img/EllarLogoB.png" width="200" alt="Ellar Logo" /></a>
+</p>
+<p align="center">EllarSQL is an SQL database Ellar Module.</p>
+
+![Test](https://github.com/python-ellar/ellar-sql/actions/workflows/test_full.yml/badge.svg)
+![Coverage](https://img.shields.io/codecov/c/github/python-ellar/ellar-sql)
+[![PyPI version](https://badge.fury.io/py/ellar-sql.svg)](https://badge.fury.io/py/ellar-sql)
+[![PyPI version](https://img.shields.io/pypi/v/ellar-sql.svg)](https://pypi.python.org/pypi/ellar-sql)
+[![PyPI version](https://img.shields.io/pypi/pyversions/ellar-sql.svg)](https://pypi.python.org/pypi/ellar-sql)
+
+EllarSQL is an SQL database module, leveraging the robust capabilities of [SQLAlchemy](https://www.sqlalchemy.org/) to 
+seamlessly interact with SQL databases through Python code and objects.
+
+Engineered with precision, EllarSQL is meticulously designed to streamline the integration of **SQLAlchemy** within your 
+Ellar application. It introduces discerning usage patterns around pivotal objects 
+such as **model**, **session**, and **engine**, ensuring an efficient and coherent workflow.
+
+Notably, EllarSQL refrains from altering the fundamental workings or usage of SQLAlchemy. 
+This documentation is focused on the meticulous setup of EllarSQL. For an in-depth exploration of SQLAlchemy, 
+we recommend referring to the comprehensive [SQLAlchemy documentation](https://docs.sqlalchemy.org/).
+
+## **Feature Highlights**
+EllarSQL comes packed with a set of awesome features designed:
+
+- **Migration**: Enjoy an async-first migration solution that seamlessly handles both single and multiple database setups and for both async and sync database engines configuration.
+
+- **Single/Multiple Database**: EllarSQL provides an intuitive setup for models with different databases, allowing you to manage your data across various sources effortlessly.
+
+- **Pagination**: EllarSQL introduces SQLAlchemy Paginator for API/Templated routes, along with support for other fantastic SQLAlchemy pagination tools.
+
+- **Unlimited Compatibility**: EllarSQL plays nice with the entire SQLAlchemy ecosystem. Whether you're using third-party tools or exploring the vast SQLAlchemy landscape, EllarSQL seamlessly integrates with your preferred tooling.
+
+## **Requirements**
+EllarSQL core dependencies includes:
+
+- Python >= 3.8
+- Ellar >= 0.6.7
+- SQLAlchemy >= 2.0.16
+- Alembic >= 1.10.0
+
+## **Installation**
+
+```shell
+pip install ellar-sql
+```
+
+## **Quick Example**
+Let's create a simple `User` model.
+```python
+from ellar_sql import model
+
+
+class User(model.Model):
+    id: model.Mapped[int] = model.mapped_column(model.Integer, primary_key=True)
+    username: model.Mapped[str] = model.mapped_column(model.String, unique=True, nullable=False)
+    email: model.Mapped[str] = model.mapped_column(model.String)
+```
+Let's create `app.db` with `User` table in it. For that we need to set up `EllarSQLService` as shown below:
+
+```python
+from ellar_sql import EllarSQLService
+
+db_service = EllarSQLService(
+    databases='sqlite:///app.db', 
+    echo=True, 
+)
+
+db_service.create_all()
+```
+If you check your execution directory, you will see `sqlite` directory with `app.db`.
+
+Let's populate our `User` table. To do, we need a session, which is available at `db_service.session_factory`
+
+```python
+from ellar_sql import EllarSQLService, model
+
+
+class User(model.Model):
+    id: model.Mapped[int] = model.mapped_column(model.Integer, primary_key=True)
+    username: model.Mapped[str] = model.mapped_column(model.String, unique=True, nullable=False)
+    email: model.Mapped[str] = model.mapped_column(model.String)
+
+
+db_service = EllarSQLService(
+    databases='sqlite:///app.db',
+    echo=True,
+)
+
+db_service.create_all()
+
+session = db_service.session_factory()
+
+for i in range(50):
+    session.add(User(username=f'username-{i+1}', email=f'user{i+1}doe@example.com'))
+
+
+session.commit()
+rows = session.execute(model.select(User)).scalars()
+
+all_users = [row.dict() for row in rows]
+assert len(all_users) == 50
+
+session.close()
+```
+
+We have successfully seed `50` users to `User` table in `app.db`. 
+
+I know at this point you want to know more, so let's dive deep into the documents and [get started](https://githut.com/python-ellar/ellar-sql/models/).
diff --git a/docs/migrations/env.md b/docs/migrations/env.md
@@ -0,0 +1,175 @@
+# **Alembic Env**
+
+In the generated migration template, EllarSQL adopts an async-first approach for handling migration file generation. 
+This approach simplifies the execution of migrations for both `Session`, `Engine`, `AsyncSession`, and `AsyncEngine`, 
+but it also introduces a certain level of complexity.
+
+```python
+from logging.config import fileConfig
+
+from alembic import context
+from ellar.app import current_injector
+from ellar.threading import execute_coroutine_with_sync_worker
+
+from ellar_sql.migrations import SingleDatabaseAlembicEnvMigration
+from ellar_sql.services import EllarSQLService
+
+# this is the Alembic Config object, which provides
+# access to the values within the .ini file in use.
+config = context.config
+
+# Interpret the config file for Python logging.
+# This line sets up loggers basically.
+fileConfig(config.config_file_name)  # type:ignore[arg-type]
+
+# logger = logging.getLogger("alembic.env")
+# other values from the config, defined by the needs of env.py,
+# can be acquired:
+# my_important_option = config.get_main_option("my_important_option")
+# ... etc.
+
+
+async def main() -> None:
+    db_service: EllarSQLService = current_injector.get(EllarSQLService)
+
+    # initialize migration class
+    alembic_env_migration = SingleDatabaseAlembicEnvMigration(db_service)
+
+    if context.is_offline_mode():
+        alembic_env_migration.run_migrations_offline(context)  # type:ignore[arg-type]
+    else:
+        await alembic_env_migration.run_migrations_online(context)  # type:ignore[arg-type]
+
+
+execute_coroutine_with_sync_worker(main())
+```
+
+The EllarSQL migration package provides two main migration classes:
+
+- **SingleDatabaseAlembicEnvMigration**: Manages migrations for a **single** database configuration, catering to both `Engine` and `AsyncEngine`.
+- **MultipleDatabaseAlembicEnvMigration**: Manages migrations for **multiple** database configurations, covering both `Engine` and `AsyncEngine`.
+
+## **Customizing the Env file**
+
+To customize or edit the Env file, it is recommended to inherit from either `SingleDatabaseAlembicEnvMigration` or `MultipleDatabaseAlembicEnvMigration` based on your specific configuration. Make the necessary changes within the inherited class.
+
+If you prefer to write something from scratch, then the abstract class `AlembicEnvMigrationBase` is the starting point. This class includes three abstract methods and expects a `EllarSQLService` during initialization, as demonstrated below:
+
+```python
+class AlembicEnvMigrationBase:
+    def __init__(self, db_service: EllarSQLService) -> None:
+        self.db_service = db_service
+        self.use_two_phase = db_service.migration_options.use_two_phase
+
+    @abstractmethod
+    def default_process_revision_directives(
+        self,
+        context: "MigrationContext",
+        revision: RevisionArgs,
+        directives: t.List["MigrationScript"],
+    ) -> t.Any:
+        pass
+
+    @abstractmethod
+    def run_migrations_offline(self, context: "EnvironmentContext") -> None:
+        pass
+
+    @abstractmethod
+    async def run_migrations_online(self, context: "EnvironmentContext") -> None:
+        pass
+```
+
+The `run_migrations_online` and `run_migrations_offline` are all similar to the same function from Alembic env.py template.
+The `default_process_revision_directives` is a callback is used to prevent an auto-migration from being generated 
+when there are no changes to the schema described in details [here](http://alembic.zzzcomputing.com/en/latest/cookbook.html)
+
+### Example
+```python
+import logging
+from logging.config import fileConfig
+
+from alembic import context
+from ellar_sql.migrations import AlembicEnvMigrationBase
+from ellar_sql.model.database_binds import get_metadata
+from ellar.app import current_injector
+from ellar.threading import execute_coroutine_with_sync_worker
+from ellar_sql.services import EllarSQLService
+
+# This is the Alembic Config object, which provides
+# access to the values within the .ini file in use.
+config = context.config
+logger = logging.getLogger("alembic.env")
+# Interpret the config file for Python logging.
+# This line sets up loggers essentially.
+fileConfig(config.config_file_name)  # type:ignore[arg-type]
+
+class MyCustomMigrationEnv(AlembicEnvMigrationBase):
+    def default_process_revision_directives(
+        self,
+        context,
+        revision,
+        directives,
+    ) -> None:
+        if getattr(context.config.cmd_opts, "autogenerate", False):
+            script = directives[0]
+            if script.upgrade_ops.is_empty():
+                directives[:] = []
+                logger.info("No changes in schema detected.")
+
+    def run_migrations_offline(self, context: "EnvironmentContext") -> None:
+        """Run migrations in 'offline' mode.
+
+        This configures the context with just a URL
+        and not an Engine, though an Engine is acceptable
+        here as well.  By skipping the Engine creation
+        we don't even need a DBAPI to be available.
+
+        Calls to context.execute() here emit the given string to the
+        script output.
+
+        """
+        pass
+
+    async def run_migrations_online(self, context: "EnvironmentContext") -> None:
+        """Run migrations in 'online' mode.
+
+        In this scenario, we need to create an Engine
+        and associate a connection with the context.
+
+        """
+        key, engine = self.db_service.engines.popitem()
+        metadata = get_metadata(key, certain=True).metadata
+
+        conf_args = {}
+        conf_args.setdefault(
+            "process_revision_directives", self.default_process_revision_directives
+        )
+
+        with engine.connect() as connection:
+            context.configure(
+                connection=connection,
+                target_metadata=metadata,
+                **conf_args
+            )
+
+            with context.begin_transaction():
+                context.run_migrations()
+
+async def main() -> None:
+    db_service: EllarSQLService = current_injector.get(EllarSQLService)
+
+    # initialize migration class
+    alembic_env_migration = MyCustomMigrationEnv(db_service)
+
+    if context.is_offline_mode():
+        alembic_env_migration.run_migrations_offline(context)
+    else:
+        await alembic_env_migration.run_migrations_online(context)
+
+execute_coroutine_with_sync_worker(main())
+```
+
+This migration environment class, `MyCustomMigrationEnv`, inherits from `AlembicEnvMigrationBase` 
+and provides the necessary methods for offline and online migrations. 
+It utilizes the `EllarSQLService` to obtain the database engines and metadata for the migration process. 
+The `main` function initializes and executes the migration class, with specific handling for offline and online modes.