diff --git a/Makefile b/Makefile index f0229e2..cd3a058 100644 --- a/Makefile +++ b/Makefile @@ -9,6 +9,7 @@ clean: ## Removing cached python compiled files find . -name \*pyo | xargs rm -fv find . -name \*~ | xargs rm -fv find . -name __pycache__ | xargs rm -rfv + find . -name .pytest_cache | xargs rm -rfv find . -name .ruff_cache | xargs rm -rfv install: ## Install dependencies @@ -23,14 +24,14 @@ lint:fmt ## Run code linters mypy ellar_sql fmt format:clean ## Run code formatters - ruff format ellar_sql tests - ruff check --fix ellar_sql tests + ruff format ellar_sql tests examples + ruff check --fix ellar_sql tests examples -test: ## Run tests - pytest tests +test:clean ## Run tests + pytest -test-cov: ## Run tests with coverage - pytest --cov=ellar_sql --cov-report term-missing tests +test-cov:clean ## Run tests with coverage + pytest --cov=ellar_sql --cov-report term-missing pre-commit-lint: ## Runs Requires commands during pre-commit make clean diff --git a/README.md b/README.md index 7a25e44..d9558ba 100644 --- a/README.md +++ b/README.md @@ -2,43 +2,30 @@ Ellar Logo

-![Test](https://github.com/python-ellar/ellar-sqlachemy/actions/workflows/test_full.yml/badge.svg) -![Coverage](https://img.shields.io/codecov/c/github/python-ellar/ellar-sqlalchemy) -[![PyPI version](https://badge.fury.io/py/ellar-sqlachemy.svg)](https://badge.fury.io/py/ellar-sqlachemy) -[![PyPI version](https://img.shields.io/pypi/v/ellar-sqlachemy.svg)](https://pypi.python.org/pypi/ellar-sqlachemy) -[![PyPI version](https://img.shields.io/pypi/pyversions/ellar-sqlachemy.svg)](https://pypi.python.org/pypi/ellar-sqlachemy) - -## Project Status - -- [ ] Overall Completion - 85% done -- [ ] Tests - 90% Complete -- [x] Model class that transforms to SQLAlchemy Models or DeclarativeBase based on configuration -- [x] Pydantic-like way to exporting model to dictionary object eg:`model.dict(exclude={'x', 'y', 'z'})` -- [x] Support multiple database useful in models and queries -- [x] Session management during request scope and outside request -- [x] Service to manage SQLAlchemy Engine and Session creation, and Migration for async and sync Engines and Sessions -- [x] Async first approach to database migration using Alembic -- [x] Expose all Alembic commands to Ellar CLI -- [x] Module to config and setup SQLAlchemy dependencies and migration path -- [x] SQLAlchemy Pagination for both Templating and API routes -- [x] File and Image SQLAlchemy Columns integration with ellar storage -- [ ] SQLAlchemy Django Like Query -- [ ] Documentation +![Test](https://github.com/python-ellar/ellar-sql/actions/workflows/test_full.yml/badge.svg) +![Coverage](https://img.shields.io/codecov/c/github/python-ellar/ellar-sql) +[![PyPI version](https://badge.fury.io/py/ellar-sql.svg)](https://badge.fury.io/py/ellar-sql) +[![PyPI version](https://img.shields.io/pypi/v/ellar-sql.svg)](https://pypi.python.org/pypi/ellar-sql) +[![PyPI version](https://img.shields.io/pypi/pyversions/ellar-sql.svg)](https://pypi.python.org/pypi/ellar-sql) + ## Introduction -Ellar SQLAlchemy Module simplifies the integration of SQLAlchemy and Alembic migration tooling into your ellar application. +EllarSQL Module adds support for `SQLAlchemy` and `Alembic` package to your Ellar application ## Installation ```shell -$(venv) pip install ellar-sqlalchemy +$(venv) pip install ellar-sql ``` +This library was inspired by [Flask-SQLAlchemy](https://flask-sqlalchemy.palletsprojects.com/en/3.1.x/){target="_blank"} + ## Features -- Automatic table name -- Session management during request and after request -- Support both async/sync SQLAlchemy operations in Session, Engine, and Connection. -- Multiple Database Support -- Database migrations for both single and multiple databases either async/sync database engine + +- Migration +- Single/Multiple Database +- Pagination +- Compatible with SQLAlchemy tools + ## **Usage** In your ellar application, create a module called `db` or any name of your choice, @@ -56,7 +43,7 @@ from ellar_sql.model import Model class Base(Model): - __base_config__ = {'make_declarative_base': True} + __base_config__ = {'as_base': True} __database__ = 'default' created_date: Mapped[datetime] = mapped_column( @@ -104,7 +91,7 @@ from .controllers import DbController }, echo=True, migration_options={ - 'directory': '__main__/migrations' + 'directory': 'my_migrations_folder' }, models=['db.models.users'] ) diff --git a/docs/advance/index.md b/docs/advance/index.md new file mode 100644 index 0000000..e69de29 diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..6c90093 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,112 @@ + +

+ Ellar Logo +

+

EllarSQL is an SQL database Ellar Module.

+ +![Test](https://github.com/python-ellar/ellar-sql/actions/workflows/test_full.yml/badge.svg) +![Coverage](https://img.shields.io/codecov/c/github/python-ellar/ellar-sql) +[![PyPI version](https://badge.fury.io/py/ellar-sql.svg)](https://badge.fury.io/py/ellar-sql) +[![PyPI version](https://img.shields.io/pypi/v/ellar-sql.svg)](https://pypi.python.org/pypi/ellar-sql) +[![PyPI version](https://img.shields.io/pypi/pyversions/ellar-sql.svg)](https://pypi.python.org/pypi/ellar-sql) + +EllarSQL is an SQL database module, leveraging the robust capabilities of [SQLAlchemy](https://www.sqlalchemy.org/) to +seamlessly interact with SQL databases through Python code and objects. + +Engineered with precision, EllarSQL is meticulously designed to streamline the integration of **SQLAlchemy** within your +Ellar application. It introduces discerning usage patterns around pivotal objects +such as **model**, **session**, and **engine**, ensuring an efficient and coherent workflow. + +Notably, EllarSQL refrains from altering the fundamental workings or usage of SQLAlchemy. +This documentation is focused on the meticulous setup of EllarSQL. For an in-depth exploration of SQLAlchemy, +we recommend referring to the comprehensive [SQLAlchemy documentation](https://docs.sqlalchemy.org/). + +## **Feature Highlights** +EllarSQL comes packed with a set of awesome features designed: + +- **Migration**: Enjoy an async-first migration solution that seamlessly handles both single and multiple database setups and for both async and sync database engines configuration. + +- **Single/Multiple Database**: EllarSQL provides an intuitive setup for models with different databases, allowing you to manage your data across various sources effortlessly. + +- **Pagination**: EllarSQL introduces SQLAlchemy Paginator for API/Templated routes, along with support for other fantastic SQLAlchemy pagination tools. + +- **Unlimited Compatibility**: EllarSQL plays nice with the entire SQLAlchemy ecosystem. Whether you're using third-party tools or exploring the vast SQLAlchemy landscape, EllarSQL seamlessly integrates with your preferred tooling. + +## **Requirements** +EllarSQL core dependencies includes: + +- Python >= 3.8 +- Ellar >= 0.6.7 +- SQLAlchemy >= 2.0.16 +- Alembic >= 1.10.0 + +## **Installation** + +```shell +pip install ellar-sql +``` + +## **Quick Example** +Let's create a simple `User` model. +```python +from ellar_sql import model + + +class User(model.Model): + id: model.Mapped[int] = model.mapped_column(model.Integer, primary_key=True) + username: model.Mapped[str] = model.mapped_column(model.String, unique=True, nullable=False) + email: model.Mapped[str] = model.mapped_column(model.String) +``` +Let's create `app.db` with `User` table in it. For that we need to set up `EllarSQLService` as shown below: + +```python +from ellar_sql import EllarSQLService + +db_service = EllarSQLService( + databases='sqlite:///app.db', + echo=True, +) + +db_service.create_all() +``` +If you check your execution directory, you will see `sqlite` directory with `app.db`. + +Let's populate our `User` table. To do, we need a session, which is available at `db_service.session_factory` + +```python +from ellar_sql import EllarSQLService, model + + +class User(model.Model): + id: model.Mapped[int] = model.mapped_column(model.Integer, primary_key=True) + username: model.Mapped[str] = model.mapped_column(model.String, unique=True, nullable=False) + email: model.Mapped[str] = model.mapped_column(model.String) + + +db_service = EllarSQLService( + databases='sqlite:///app.db', + echo=True, +) + +db_service.create_all() + +session = db_service.session_factory() + +for i in range(50): + session.add(User(username=f'username-{i+1}', email=f'user{i+1}doe@example.com')) + + +session.commit() +rows = session.execute(model.select(User)).scalars() + +all_users = [row.dict() for row in rows] +assert len(all_users) == 50 + +session.close() +``` + +We have successfully seed `50` users to `User` table in `app.db`. + +I know at this point you want to know more, so let's dive deep into the documents and [get started](https://githut.com/python-ellar/ellar-sql/models/). diff --git a/docs/migrations/env.md b/docs/migrations/env.md new file mode 100644 index 0000000..1ec01b2 --- /dev/null +++ b/docs/migrations/env.md @@ -0,0 +1,175 @@ +# **Alembic Env** + +In the generated migration template, EllarSQL adopts an async-first approach for handling migration file generation. +This approach simplifies the execution of migrations for both `Session`, `Engine`, `AsyncSession`, and `AsyncEngine`, +but it also introduces a certain level of complexity. + +```python +from logging.config import fileConfig + +from alembic import context +from ellar.app import current_injector +from ellar.threading import execute_coroutine_with_sync_worker + +from ellar_sql.migrations import SingleDatabaseAlembicEnvMigration +from ellar_sql.services import EllarSQLService + +# this is the Alembic Config object, which provides +# access to the values within the .ini file in use. +config = context.config + +# Interpret the config file for Python logging. +# This line sets up loggers basically. +fileConfig(config.config_file_name) # type:ignore[arg-type] + +# logger = logging.getLogger("alembic.env") +# other values from the config, defined by the needs of env.py, +# can be acquired: +# my_important_option = config.get_main_option("my_important_option") +# ... etc. + + +async def main() -> None: + db_service: EllarSQLService = current_injector.get(EllarSQLService) + + # initialize migration class + alembic_env_migration = SingleDatabaseAlembicEnvMigration(db_service) + + if context.is_offline_mode(): + alembic_env_migration.run_migrations_offline(context) # type:ignore[arg-type] + else: + await alembic_env_migration.run_migrations_online(context) # type:ignore[arg-type] + + +execute_coroutine_with_sync_worker(main()) +``` + +The EllarSQL migration package provides two main migration classes: + +- **SingleDatabaseAlembicEnvMigration**: Manages migrations for a **single** database configuration, catering to both `Engine` and `AsyncEngine`. +- **MultipleDatabaseAlembicEnvMigration**: Manages migrations for **multiple** database configurations, covering both `Engine` and `AsyncEngine`. + +## **Customizing the Env file** + +To customize or edit the Env file, it is recommended to inherit from either `SingleDatabaseAlembicEnvMigration` or `MultipleDatabaseAlembicEnvMigration` based on your specific configuration. Make the necessary changes within the inherited class. + +If you prefer to write something from scratch, then the abstract class `AlembicEnvMigrationBase` is the starting point. This class includes three abstract methods and expects a `EllarSQLService` during initialization, as demonstrated below: + +```python +class AlembicEnvMigrationBase: + def __init__(self, db_service: EllarSQLService) -> None: + self.db_service = db_service + self.use_two_phase = db_service.migration_options.use_two_phase + + @abstractmethod + def default_process_revision_directives( + self, + context: "MigrationContext", + revision: RevisionArgs, + directives: t.List["MigrationScript"], + ) -> t.Any: + pass + + @abstractmethod + def run_migrations_offline(self, context: "EnvironmentContext") -> None: + pass + + @abstractmethod + async def run_migrations_online(self, context: "EnvironmentContext") -> None: + pass +``` + +The `run_migrations_online` and `run_migrations_offline` are all similar to the same function from Alembic env.py template. +The `default_process_revision_directives` is a callback is used to prevent an auto-migration from being generated +when there are no changes to the schema described in details [here](http://alembic.zzzcomputing.com/en/latest/cookbook.html) + +### Example +```python +import logging +from logging.config import fileConfig + +from alembic import context +from ellar_sql.migrations import AlembicEnvMigrationBase +from ellar_sql.model.database_binds import get_metadata +from ellar.app import current_injector +from ellar.threading import execute_coroutine_with_sync_worker +from ellar_sql.services import EllarSQLService + +# This is the Alembic Config object, which provides +# access to the values within the .ini file in use. +config = context.config +logger = logging.getLogger("alembic.env") +# Interpret the config file for Python logging. +# This line sets up loggers essentially. +fileConfig(config.config_file_name) # type:ignore[arg-type] + +class MyCustomMigrationEnv(AlembicEnvMigrationBase): + def default_process_revision_directives( + self, + context, + revision, + directives, + ) -> None: + if getattr(context.config.cmd_opts, "autogenerate", False): + script = directives[0] + if script.upgrade_ops.is_empty(): + directives[:] = [] + logger.info("No changes in schema detected.") + + def run_migrations_offline(self, context: "EnvironmentContext") -> None: + """Run migrations in 'offline' mode. + + This configures the context with just a URL + and not an Engine, though an Engine is acceptable + here as well. By skipping the Engine creation + we don't even need a DBAPI to be available. + + Calls to context.execute() here emit the given string to the + script output. + + """ + pass + + async def run_migrations_online(self, context: "EnvironmentContext") -> None: + """Run migrations in 'online' mode. + + In this scenario, we need to create an Engine + and associate a connection with the context. + + """ + key, engine = self.db_service.engines.popitem() + metadata = get_metadata(key, certain=True).metadata + + conf_args = {} + conf_args.setdefault( + "process_revision_directives", self.default_process_revision_directives + ) + + with engine.connect() as connection: + context.configure( + connection=connection, + target_metadata=metadata, + **conf_args + ) + + with context.begin_transaction(): + context.run_migrations() + +async def main() -> None: + db_service: EllarSQLService = current_injector.get(EllarSQLService) + + # initialize migration class + alembic_env_migration = MyCustomMigrationEnv(db_service) + + if context.is_offline_mode(): + alembic_env_migration.run_migrations_offline(context) + else: + await alembic_env_migration.run_migrations_online(context) + +execute_coroutine_with_sync_worker(main()) +``` + +This migration environment class, `MyCustomMigrationEnv`, inherits from `AlembicEnvMigrationBase` +and provides the necessary methods for offline and online migrations. +It utilizes the `EllarSQLService` to obtain the database engines and metadata for the migration process. +The `main` function initializes and executes the migration class, with specific handling for offline and online modes. diff --git a/docs/migrations/index.md b/docs/migrations/index.md new file mode 100644 index 0000000..397b4ba --- /dev/null +++ b/docs/migrations/index.md @@ -0,0 +1,154 @@ +# **Migrations** +EllarSQL also extends **Alembic** package +to add migration functionality and make database operations accessible through **EllarCLI** commandline interface. + +**EllarSQL** with Alembic does not override Alembic action rather provide Alembic all the configs/information +it needs to for a proper migration/database operations. +Its also still possible to use Alembic outside EllarSQL setup when necessary. + +This section is inspired by [`Flask Migrate`](https://flask-migrate.readthedocs.io/en/latest/#) + +## **Quick Example** +We assume you have set up `EllarSQLModule` in your application, and you have specified `migration_options`. + +Create a simple `User` model as shown below: + +```python +from ellar_sql import model + + +class User(model.Model): + id = model.Column(model.Integer, primary_key=True) + name = model.Column(model.String(128)) +``` + +### **Initialize migration template** +With the Model setup, run the command below + +```shell +# Initialize the database +ellar db init +``` + +Executing this command will incorporate a migrations folder into your application structure. +Ensure that the contents of this folder are included in version control alongside your other source files. + +Following the initialization, you can generate an initial migration using the command: + +```shell +# Generate the initial migration +ellar db migrate -m "Initial migration." +``` +Few things to do after generating a migration file: + +- Review and edit the migration script +- Alembic may not detect certain changes automatically, such as table and column name modifications or unnamed constraints. Refer to the [Alembic autogenerate documentation](https://alembic.sqlalchemy.org/en/latest/autogenerate.html#what-does-autogenerate-detect-and-what-does-it-not-detect) for a comprehensive list of limitations. +- Add the finalized migration script to version control +- Ensure that the edited script is committed along with your source code changes + +Apply the changes described in the migration script to your database + +```shell +ellar db upgrade +``` + +Whenever there are changes to the database models, it's necessary to repeat the `migrate` and `upgrade` commands. + +For synchronizing the database on another system, simply refresh the migrations folder from the source control repository +and execute the `upgrade` command. This ensures that the database structure aligns with the latest changes in the models. + +### **Multiple Database Migration** +If your application utilizes multiple databases, a distinct Alembic template for migration is required. +To enable this, include `-m` or `--multi` with the `db init` command, as demonstrated below: + +```shell +ellar db init --multi +``` + +## **Command Reference** +All Alembic commands are expose to Ellar CLI under `db` group after a successful `EllarSQLModule` setup. + +To see all the commands that are available run this command: +```shell +ellar db --help + +# output +Usage: ellar db [OPTIONS] COMMAND [ARGS]... + + - Perform Alembic Database Commands - + +Options: + --help Show this message and exit. + +Commands: + branches - Show current branch points + check Check if there are any new operations to migrate + current - Display the current revision for each database. + downgrade - Revert to a previous version + edit - Edit a revision file + heads - Show current available heads in the script directory + history - List changeset scripts in chronological order. + init Creates a new migration repository. + merge - Merge two revisions together, creating a new revision file + migrate - Autogenerate a new revision file (Alias for 'revision... + revision - Create a new revision file. + show - Show the revision denoted by the given symbol. + stamp - 'stamp' the revision table with the given revision; don't... + upgrade - Upgrade to a later version + +``` + +- `ellar db --help` + Shows a list of available commands. + + +- `ellar db revision [--message MESSAGE] [--autogenerate] [--sql] [--head HEAD] [--splice] [--branch-label BRANCH_LABEL] [--version-path VERSION_PATH] [--rev-id REV_ID]` + Creates an empty revision script. The script needs to be edited manually with the upgrade and downgrade changes. See Alembic’s documentation for instructions on how to write migration scripts. An optional migration message can be included. + + +- `ellar db migrate [--message MESSAGE] [--sql] [--head HEAD] [--splice] [--branch-label BRANCH_LABEL] [--version-path VERSION_PATH] [--rev-id REV_ID]` + Equivalent to revision --autogenerate. The migration script is populated with changes detected automatically. The generated script should be reviewed and edited as not all types of changes can be detected automatically. This command does not make any changes to the database, just creates the revision script. + + +- `ellar db check` + Checks that a migrate command would not generate any changes. If pending changes are detected, the command exits with a non-zero status code. + + +- `ellar db edit ` + Edit a revision script using $EDITOR. + + +- `ellar db upgrade [--sql] [--tag TAG] ` + Upgrades the database. If revision isn’t given, then "head" is assumed. + + +- `ellar db downgrade [--sql] [--tag TAG] ` + Downgrades the database. If revision isn’t given, then -1 is assumed. + + +- `ellar db stamp [--sql] [--tag TAG] ` + Sets the revision in the database to the one given as an argument, without performing any migrations. + + +- `ellar db current [--verbose]` + Shows the current revision of the database. + + +- `ellar db history [--rev-range REV_RANGE] [--verbose]` + Shows the list of migrations. If a range isn’t given, then the entire history is shown. + + +- `ellar db show ` + Show the revision denoted by the given symbol. + + +- `ellar db merge [--message MESSAGE] [--branch-label BRANCH_LABEL] [--rev-id REV_ID] ` + Merge two revisions together. Create a new revision file. + + +- `ellar db heads [--verbose] [--resolve-dependencies]` + Show current available heads in the revision script directory. + + +- `ellar db branches [--verbose]` + Show current branch points. diff --git a/docs/models/configuration.md b/docs/models/configuration.md new file mode 100644 index 0000000..7865821 --- /dev/null +++ b/docs/models/configuration.md @@ -0,0 +1,189 @@ +# **EllarSQLModule Config** +**`EllarSQLModule`** is an Ellar Dynamic Module that offers two ways of configuration: + +- `EllarSQLModule.register_setup()`: This method registers a `ModuleSetup` that depends on the application config. +- `EllarSQLModule.setup()`: This method immediately sets up the module with the provided options. + +While we've explored many examples using `EllarSQLModule.setup()`, this section will focus on the usage of `EllarSQLModule.register_setup()`. + +Before delving into that, let's first explore the setup options available for `EllarSQLModule`. +## **EllarSQLModule Configuration Parameters** + +- **databases**: _typing.Union[str, typing.Dict[str, typing.Any]]_: + + This field describes the options for your database engine, utilized by SQLAlchemy **Engine**, **Metadata**, and **Sessions**. There are three methods for setting these options, as illustrated below: + ```python + ## CASE 1 + databases = "sqlite//:memory:" + # This will result to + # databases = { + # 'default': { + # 'url': 'sqlite//:memory:' + # } + # } + + ## CASE 2 + databases = { + 'default': "sqlite//:memory:", + 'db2': "sqlite//:memory:", + } + # This will result to + # databases = { + # 'default': { + # 'url': 'sqlite//:memory:' + # }, + # 'db2': { + # 'url': 'sqlite//:memory:' + # }, + # } + + ## CASE 3 - With Extra Engine Options + databases = { + 'default': { + "url": "sqlite//:memory:", + "echo": True, + "connect_args": { + "check_same_thread": False + } + } + } + ``` + + +- **migration_options**: _typing.Union[typing.Dict[str, typing.Any], MigrationOption]_: + The migration options can be specified either in a dictionary object or as a `MigrationOption` schema instance. + These configurations are essential for defining the necessary settings for database migrations. The available options include: + - **directory**=`migrations`:directory to save alembic migration templates/env and migration versions + - **use_two_phase**=`True`: bool value that indicates use of two in migration SQLAlchemy session + - **context_configure**=`{compare_type: True, render_as_batch: True, include_object: callable}`: + key-value pair that will be passed to [`EnvironmentContext.configure`](https://alembic.sqlalchemy.org/en/latest/api/runtime.html#alembic.runtime.environment.EnvironmentContext.configure){target="_blank"}. + + Default **context_configure** set by EllarSQL: + + - **compare_type=True**: This option configures the automatic migration generation subsystem to detect column type changes. + - **render_as_batch=True**: This option generates migration scripts using [batch mode](https://alembic.sqlalchemy.org/en/latest/batch.html){target="_blank"}, an operational mode that works around limitations of many ALTER commands in the SQLite database by implementing a “move and copy” workflow. + - **include_object**: Skips model from auto gen when it's defined in table args eg: `__table_args__ = {"info": {"skip_autogen": True}}` + +- **session_options**: _t.Optional[t.Dict[str, t.Any]]_: + + A default key-value pair pass to [SQLAlchemy.Session()](https://docs.sqlalchemy.org/en/20/orm/session_api.html#sqlalchemy.orm.Session){target="_blank"} when creating a session. + +- **engine_options**: _t.Optional[t.Dict[str, t.Any]]_: + + A default key-value pair to pass to every database configuration engine configuration for [SQLAlchemy.create_engine()](https://docs.sqlalchemy.org/en/20/core/engines.html#sqlalchemy.create_engine){target="_blank"}. + + This overriden by configurations provided in `databases` parameters + +- **models**: _t.Optional[t.List[str]]_: list of python modules that defines `model.Model` models. By providing this, EllarSQL ensures models are discovered before Alembic CLI migration actions or any other database interactions with SQLAlchemy. + +- **echo**: _bool_: The default value for `echo` and `echo_pool` for every engine. This is useful to quickly debug the connections and queries issued from SQLAlchemy. + +- **root_path**: _t.Optional[str]_: The `root_path` for sqlite databases and migration base directory. Defaults to the execution path of `EllarSQLModule` + +## **Connection URL Format** +Refer to SQLAlchemy’s documentation on [Engine Configuration](https://docs.sqlalchemy.org/en/20/core/engines.html){target="_blank"} +for a comprehensive overview of syntax, dialects, and available options. + +The standard format for a basic database connection URL is as follows: Username, password, host, and port are optional +parameters based on the database type and specific configuration. +``` +dialect://username:password@host:port/database +``` +Here are some example connection strings: +```text +# SQLite, relative to Flask instance path +sqlite:///project.db + +# PostgreSQL +postgresql://scott:tiger@localhost/project + +# MySQL / MariaDB +mysql://scott:tiger@localhost/project +``` + +## **Default Driver Options** + +To enhance usability for web applications, default options have been configured for SQLite and MySQL engines. + +For SQLite, relative file paths are now relative to the **`root_path`** option rather than the current working directory. +Additionally, **in-memory** databases utilize a static **`pool`** and **`check_same_thread`** to ensure seamless operation across multiple requests. + +For `MySQL` (and `MariaDB`) servers, a default idle connection timeout of 8 hours has been set. This configuration helps avoid errors, +such as 2013: Lost connection to `MySQL` server during query. To preemptively recreate connections before hitting this timeout, +a default **`pool_recycle`** value of 2 hours (**7200** seconds) is applied. + +## **Timeout** + +Certain databases, including `MySQL` and `MariaDB`, might be set to close inactive connections after a certain duration, +which can lead to errors like 2013: Lost connection to MySQL server during query. +While this behavior is configured by default in MySQL and MariaDB, it could also be implemented by other database services. + +If you encounter such errors, consider adjusting the pool_recycle option in the engine settings to a value less than the database's timeout. + +Alternatively, you can explore setting pool_pre_ping if you anticipate frequent closure of connections, +especially in scenarios like running the database in a container that may undergo periodic restarts. + +For more in-depth information +on [dealing with disconnects](https://docs.sqlalchemy.org/core/pooling.html#dealing-with-disconnects){target="_blank"}, +refer to SQLAlchemy's documentation on handling connection issues. + +## **EllarSQLModule RegisterSetup** +As mentioned earlier, **EllarSQLModule** can be configured from the application through `EllarSQLModule.register_setup`. +This process registers a [ModuleSetup](https://python-ellar.github.io/ellar/basics/dynamic-modules/#modulesetup){target="_blank"} factory +that depends on the Application Config object. +The factory retrieves the `ELLAR_SQL` attribute from the config and validates the data before passing it to `EllarSQLModule` for setup. + +It's essential to note +that `ELLAR_SQL` will be a dictionary object with the [configuration parameters](#ellarsqlmodule-configuration-parameters) +mentioned above as keys. + +Here's a quick example: +```python title="db_learning/root_module.py" +from ellar.common import Module, exception_handler, IExecutionContext, JSONResponse, Response, IApplicationStartup +from ellar.app import App +from ellar.core import ModuleBase +from ellar_sql import EllarSQLModule, EllarSQLService +from .controller import UsersController + + +@Module( + modules=[EllarSQLModule.register_setup()], + controllers=[UsersController] +) +class ApplicationModule(ModuleBase, IApplicationStartup): + async def on_startup(self, app: App) -> None: + db_service = app.injector.get(EllarSQLService) + db_service.create_all() + + @exception_handler(404) + def exception_404_handler(cls, ctx: IExecutionContext, exc: Exception) -> Response: + return JSONResponse(dict(detail="Resource not found."), status_code=404) + +``` +Let's update `config.py`. + +```python +import typing as t +... + +class DevelopmentConfig(BaseConfig): + DEBUG: bool = True + + ELLAR_SQL: t.Dict[str, t.Any] = { + 'databases': { + 'default': 'sqlite:///app.db', + }, + 'echo': True, + 'migration_options': { + 'directory': 'migrations' # root directory will be determined based on where the module is instantiated. + }, + 'models': [] + } +``` +The registered ModuleSetup factory reads the `ELLAR_SQL` value and configures the `EllarSQLModule` appropriately. + +This approach is particularly useful when dealing with multiple environments. +It allows for seamless modification of the **ELLAR_SQL** values in various environments such as +Continuous Integration (CI), Development, Staging, or Production. +You can easily change the settings for each environment +and export the configurations as a string to be imported into `ELLAR_CONFIG_MODULE`. diff --git a/docs/models/extra-fields.md b/docs/models/extra-fields.md new file mode 100644 index 0000000..752f617 --- /dev/null +++ b/docs/models/extra-fields.md @@ -0,0 +1,39 @@ +# **Extra Column Types** +EllarSQL comes with extra column type descriptor that will come in handy in your project. They include + +- [GUID](#guid-column) +- [IPAddress](#ipaddress-column) + +## **GUID Column** +GUID, Global Unique Identifier of 128-bit text string can be used as a unique identifier in a table. +For applications that require a GUID type of primary, this can be a use resource. +It uses `UUID` type in Postgres and `CHAR(32)` in other SQL databases. + +```python +import uuid +from ellar_sql import model + +class Guid(model.Model): + id: model.Mapped[uuid.uuid4] = model.mapped_column( + "id", + model.GUID(), + nullable=False, + unique=True, + primary_key=True, + default=uuid.uuid4, + ) +``` + +## **IPAddress Column** +`GenericIP` column type validates and converts column value to `ipaddress.IPv4Address` or `ipaddress.IPv6Address`. +It uses `INET` type in Postgres and `CHAR(45)` in other SQL databases. + +```python +import typing as t +import ipaddress +from ellar_sql import model + +class IPAddress(model.Model): + id = model.Column(model.Integer, primary_key=True) + ip: model.Mapped[t.Union[ipaddress.IPv4Address, ipaddress.IPv6Address]] = model.Column(model.GenericIP) +``` diff --git a/docs/models/index.md b/docs/models/index.md new file mode 100644 index 0000000..960b64f --- /dev/null +++ b/docs/models/index.md @@ -0,0 +1,150 @@ +# **Quick Start** +In this segment, we will walk through the process of configuring **EllarSQL** within your Ellar application, +ensuring that all essential services are registered, configurations are set, and everything is prepared for immediate use. + +Before we delve into the setup instructions, it is assumed that you possess a comprehensive +understanding of how [Ellar Modules](https://python-ellar.github.io/ellar/basics/dynamic-modules/#module-dynamic-setup){target="_blank"} +operate. + +## **Installation** +Let us install all the required packages, assuming that your Python environment has been properly configured: + +#### **For Existing Project:** +```shell +pip install ellar-sql +``` + +#### **For New Project**: +```shell +pip install ellar ellar-cli ellar-sql +``` + +After a successful package installation, we need to scaffold a new project using `ellar` cli tool +```shell +ellar new db-learning +``` +This will scaffold `db-learning` project with necessary file structure shown below. +``` +path/to/db-learning/ +├─ db_learning/ +│ ├─ apps/ +│ │ ├─ __init__.py +│ ├─ core/ +│ ├─ config.py +│ ├─ domain +│ ├─ root_module.py +│ ├─ server.py +│ ├─ __init__.py +├─ tests/ +│ ├─ __init__.py +├─ pyproject.toml +├─ README.md +``` +Next, in `db_learning/` directory, we need to create a `models.py`. It will hold all our SQLAlchemy ORM Models for now. + +## **Creating a Model** +In `models.py`, we use `ellar_sql.model.Model` to create our SQLAlchemy ORM Models. + +```python title="db_learning/model.py" +from ellar_sql import model + + +class User(model.Model): + id: model.Mapped[int] = model.mapped_column(model.Integer, primary_key=True) + username: model.Mapped[str] = model.mapped_column(model.String, unique=True, nullable=False) + email: model.Mapped[str] = model.mapped_column(model.String) +``` + +!!!info + `ellar_sql.model` also exposes `sqlalchemy`, `sqlalchemy.orm` and `sqlalchemy.event` imports just for ease of import reference + +## **Create A UserController** +Let's create a controller that exposes our user data. + +```python title="db_learning/controller.py" +import ellar.common as ecm +from ellar.pydantic import EmailStr +from ellar_sql import model, get_or_404 +from .models import User + + +@ecm.Controller +class UsersController(ecm.ControllerBase): + @ecm.post("/users") + def create_user(self, username: ecm.Body[str], email: ecm.Body[EmailStr], session: ecm.Inject[model.Session]): + user = User(username=username, email=email) + + session.add(user) + session.commit() + session.refresh(user) + + return user.dict() + + @ecm.get("/users/{user_id:int}") + def user_by_id(self, user_id: int): + user = get_or_404(User, user_id) + return user.dict() + + @ecm.get("/") + async def user_list(self, session: ecm.Inject[model.Session]): + stmt = model.select(User) + rows = session.execute(stmt.offset(0).limit(100)).scalars() + return [row.dict() for row in rows] + + @ecm.get("/{user_id:int}") + async def user_delete(self, user_id: int, session: ecm.Inject[model.Session]): + user = get_or_404(User, user_id) + session.delete(user) + return {'detail': f'User id={user_id} Deleted successfully'} +``` + +## **EllarSQLModule Setup** +In the `root_module.py` file, two main tasks need to be performed: + +1. Register the `UsersController` to make the `/users` endpoint available when starting the application. +2. Configure the `EllarSQLModule`, which will set up and register essential services such as `EllarSQLService`, `Session`, and `Engine`. + +```python title="db_learning/root_module.py" +from ellar.common import Module, exception_handler, IExecutionContext, JSONResponse, Response, IApplicationStartup +from ellar.app import App +from ellar.core import ModuleBase +from ellar_sql import EllarSQLModule, EllarSQLService +from .controller import UsersController + +@Module( + modules=[EllarSQLModule.setup( + databases={ + 'default': { + 'url': 'sqlite:///app.db', + 'echo': True + } + }, + migration_options={'directory': 'migrations'} + )], + controllers=[UsersController] +) +class ApplicationModule(ModuleBase, IApplicationStartup): + async def on_startup(self, app: App) -> None: + db_service = app.injector.get(EllarSQLService) + db_service.create_all() + + @exception_handler(404) + def exception_404_handler(cls, ctx: IExecutionContext, exc: Exception) -> Response: + return JSONResponse(dict(detail="Resource not found."), status_code=404) +``` + +In the provided code snippet: + +- We registered `UserController` and `EllarSQLModule` with specific configurations for the database and migration options. For more details on [`EllarSQLModule` configurations](./configuration.md#ellarsqlmodule-config). + +- In the `on_startup` method, we obtained the `EllarSQLService` from the Ellar Dependency Injection container using `EllarSQLModule`. Subsequently, we invoked the `create_all()` method to generate the necessary SQLAlchemy tables. + +With these configurations, the application is now ready for testing. +```shell +ellar runserver --reload +``` +Additionally, please remember to uncomment the configurations for the `OpenAPIModule` in the `server.py` +file to enable visualization and interaction with the `/users` endpoint. + +Once done, +you can access the OpenAPI documentation at [http://127.0.0.1:8000/docs](http://127.0.0.1:8000/docs){target="_blank"}. diff --git a/docs/models/models.md b/docs/models/models.md new file mode 100644 index 0000000..f5edea2 --- /dev/null +++ b/docs/models/models.md @@ -0,0 +1,323 @@ +# **Models and Tables** +The `ellar_sql.model.Model` class acts as a factory for creating `SQLAlchemy` models, and +associating the generated models with the corresponding **Metadata** through their designated **`__database__`** key. + + +This class can be configured through the `__base_config__` attribute, allowing you to specify how your `SQLAlchemy` model should be created. +The `__base_config__` attribute can be of type `ModelBaseConfig`, which is a dataclass, or a dictionary with keys that +match the attributes of `ModelBaseConfig`. + +Attributes of `ModelBaseConfig`: + +- **as_base**: Indicates whether the class should be treated as a `Base` class for other model definitions, similar to creating a Base from a [DeclarativeBase](https://docs.sqlalchemy.org/en/20/orm/mapping_api.html#sqlalchemy.orm.DeclarativeBase){target="_blank"} or [DeclarativeBaseNoMeta](https://docs.sqlalchemy.org/en/20/orm/mapping_api.html#sqlalchemy.orm.DeclarativeBaseNoMeta){target="_blank"} class. *(Default: False)* +- **use_base**: Specifies the base classes that will be used to create the `SQLAlchemy` model. *(Default: [])* + +## **Creating a Base Class** +`Model` treats each model as a standalone entity. Each instance of `model.Model` creates a distinct **declarative** base for itself, using the `__database__` key as a reference to determine its associated **Metadata**. Consequently, models sharing the same `__database__` key will utilize the same **Metadata** object. + +Let's explore how we can create a `Base` model using `Model`, similar to the approach in traditional `SQLAlchemy`. + +```python +from ellar_sql import model, ModelBaseConfig + + +class Base(model.Model): + __base_config__ = ModelBaseConfig(as_base=True, use_bases=[model.DeclarativeBase]) + + +assert issubclass(Base, model.DeclarativeBase) +``` + +If you are interested in [SQLAlchemy’s native support for data classes](https://docs.sqlalchemy.org/en/20/changelog/whatsnew_20.html#native-support-for-dataclasses-mapped-as-orm-models){target="_blank"}, +then you can add `MappedAsDataclass` to `use_bases` as shown below: + +```python +from ellar_sql import model, ModelBaseConfig + + +class Base(model.Model): + __base_config__ = ModelBaseConfig(as_base=True, use_bases=[model.DeclarativeBase, model.MappedAsDataclass]) + +assert issubclass(Base, model.MappedAsDataclass) +``` + +In the examples above, `Base` classes are created, all subclassed from the `use_bases` provided, and with the `as_base` +option, the factory creates the `Base` class as a `Base`. + +## Create base with MetaData +You can also configure the SQLAlchemy object with a custom [`MetaData`](https://docs.sqlalchemy.org/en/20/core/metadata.html#sqlalchemy.schema.MetaData){target="_blank"} object. +For instance, you can define a specific **naming convention** for constraints, ensuring consistency and predictability in constraint names. +This can be particularly beneficial during migrations, as detailed by [Alembic](https://alembic.sqlalchemy.org/en/latest/naming.html){target="_blank"}. + +For example: + +```python +from ellar_sql import model, ModelBaseConfig + +class Base(model.Model): + __base_config__ = ModelBaseConfig(as_base=True, use_bases=[model.DeclarativeBase]) + + metadata = model.MetaData(naming_convention={ + "ix": 'ix_%(column_0_label)s', + "uq": "uq_%(table_name)s_%(column_0_name)s", + "ck": "ck_%(table_name)s_%(constraint_name)s", + "fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s", + "pk": "pk_%(table_name)s" + }) +``` + +## **Abstract Models and Mixins** +If the desired behavior is only applicable to specific models rather than all models, +you can use an abstract model base class to customize only those models. +For example, if certain models need to track their creation or update **timestamps**, t +his approach allows for targeted customization. + +```python +from datetime import datetime, timezone +from ellar_sql import model +from sqlalchemy.orm import Mapped, mapped_column + +class TimestampModel(model.Model): + __abstract__ = True + created: Mapped[datetime] = mapped_column(default=lambda: datetime.now(timezone.utc)) + updated: Mapped[datetime] = mapped_column(default=lambda: datetime.now(timezone.utc), onupdate=lambda: datetime.now(timezone.utc)) + + +class BookAuthor(model.Model): + id: Mapped[int] = mapped_column(primary_key=True) + name: Mapped[str] = mapped_column(unique=True) + + +class Book(TimestampModel): + id: Mapped[int] = mapped_column(primary_key=True) + title: Mapped[str] +``` + +This can also be done with a mixin class, inherited separately. +```python +from datetime import datetime, timezone +from ellar_sql import model +from sqlalchemy.orm import Mapped, mapped_column + +class TimestampModel: + created: Mapped[datetime] = mapped_column(default=lambda: datetime.now(timezone.utc)) + updated: Mapped[datetime] = mapped_column(default=lambda: datetime.now(timezone.utc), onupdate=lambda: datetime.now(timezone.utc)) + + +class Book(model.Model, TimestampModel): + id: Mapped[int] = mapped_column(primary_key=True) + title: Mapped[str] +``` + +## **Defining Models** +Unlike plain SQLAlchemy, **EllarSQL** models will automatically generate a table name if the `__tablename__` attribute is not set, +provided a primary key column is defined. + +```python +from ellar_sql import model + + +class User(model.Model): + id: model.Mapped[int] = model.mapped_column(primary_key=True) + username: model.Mapped[str] = model.mapped_column(unique=True) + email: model.Mapped[str] + + +class UserAddress(model.Model): + __tablename__ = 'user-address' + id: model.Mapped[int] = model.mapped_column(primary_key=True) + address: model.Mapped[str] = model.mapped_column(unique=True) + +assert User.__tablename__ == 'user' +assert UserAddress.__tablename__ == 'user-address' +``` +For a comprehensive guide on defining model classes declaratively, refer to +[SQLAlchemy’s declarative documentation](https://docs.sqlalchemy.org/en/20/orm/declarative_tables.html){target="_blank"}. +This resource provides detailed information and insights into the declarative approach for defining model classes. + +## **Defining Tables** +The table class is designed to receive a table name, followed by **columns** and other table **components** such as constraints. + +EllarSQL enhances the functionality of the SQLAlchemy Table +by facilitating the selection of **Metadata** based on the `__database__` argument. + +Directly creating a table proves particularly valuable when establishing **many-to-many** relationships. +In such cases, the association table doesn't need its dedicated model class; rather, it can be conveniently accessed +through the relevant relationship attributes on the associated models. + +```python +from ellar_sql import model + +author_book_m2m = model.Table( + "author_book", + model.Column("book_author_id", model.ForeignKey(BookAuthor.id), primary_key=True), + model.Column("book_id", model.ForeignKey(Book.id), primary_key=True), +) +``` + +## **Quick Tutorial** +In this section, we'll delve into straightforward **CRUD** operations using the ORM objects. +However, if you're not well-acquainted with SQLAlchemy, +feel free to explore their tutorial +on [ORM](https://docs.sqlalchemy.org/tutorial/orm_data_manipulation.html){target="_blank"} +for a more comprehensive understanding. + +Having understood, `Model` usage. Let's create a `User` model + +```python +from ellar_sql import model + + +class User(model.Model): + id: model.Mapped[int] = model.mapped_column(primary_key=True) + username: model.Mapped[str] = model.mapped_column(unique=True) + full_name: model.Mapped[str] = model.mapped_column(model.String) +``` +We have created a `User` model but the data does not exist. Let's fix that + +```python +from ellar.app import current_injector +from ellar_sql import EllarSQLService + +db_service = current_injector.get(EllarSQLService) +db_service.create_all() +``` + +### **Insert** +To insert a data, you need a session +```python +import ellar.common as ecm +from .model import User + +@ecm.post('/create') +def create_user(): + session = User.get_db_session() + squidward = User(name="squidward", fullname="Squidward Tentacles") + session.add(squidward) + + session.commit() + + return squidward.dict(exclude={'id'}) +``` +In the above illustration, `squidward` +data was converted to `dictionary` object by calling `.dict()` and excluding the `id` as shown below. + +_It's important to note this functionality has not been extended to a relationship objects in an SQLAlchemy ORM object_. + +### **Update** +To update, make changes to the ORM object and commit. +```python +import ellar.common as ecm +from .model import User + + +@ecm.put('/update') +def update_user(): + session = User.get_db_session() + squidward = session.get(User, 1) + + squidward.fullname = 'EllarSQL' + session.commit() + + return squidward.dict() + +``` +### **Delete** +To delete, pass the ORM object to `session.delete()`. +```python +import ellar.common as ecm +from .model import User + + +@ecm.delete('/delete') +def delete_user(): + session = User.get_db_session() + squidward = session.get(User, 1) + + session.delete(squidward) + session.commit() + + return '' + +``` +After modifying data, you must call `session.commit()` to commit the changes to the database. +Otherwise, changes may not be persisted to the database. + +## **View Utilities** +EllarSQL provides some utility query functions to check missing entities and raise 404 Not found if not found. + +- **get_or_404**: It will raise a 404 error if the row with the given id does not exist; otherwise, it will return the corresponding instance. +- **first_or_404**: It will raise a 404 error if the query does not return any results; otherwise, it will return the first result. +- **one_or_404**(): It will raise a 404 error if the query does not return exactly one result; otherwise, it will return the result. + +```python +import ellar.common as ecm +from ellar_sql import get_or_404, one_or_404, model + +@ecm.get("/user-by-id/{user_id:int}") +def user_by_id(user_id: int): + user = get_or_404(User, user_id) + return user.dict() + +@ecm.get("/user-by-name/{name:str}") +def user_by_username(name: str): + user = one_or_404(model.select(User).filter_by(name=name), error_message=f"No user named '{name}'.") + return user.dict() +``` + +## **Accessing Metadata and Engines** + +In the process of `EllarSQLModule` setup, three services are registered to the Ellar IoC container. + +- `EllarSQLService`: Which manages models, metadata, engines and sessions +- `Engine`: SQLAlchemy Engine of the default database configuration +- `Session`SQLAlchemy Session of the default database configuration + +Although with `EllarSQLService` you can get the `engine` and `session`. It's there for easy of access. + +```python +import sqlalchemy as sa +import sqlalchemy.orm as sa_orm +from ellar.app import current_injector +from ellar_sql import EllarSQLService + +db_service = current_injector.get(EllarSQLService) + +assert isinstance(db_service.engine, sa.Engine) +assert isinstance(db_service.session_factory(), sa_orm.Session) +``` +#### **Important Constraints** +- EllarSQLModule `databases` options for `SQLAlchemy.ext.asyncio.AsyncEngine` will register `SQLAlchemy.ext.asyncio.AsyncEngine` + and `SQLAlchemy.ext.asyncio.AsyncSession` +- EllarSQLModule `databases` options for `SQLAlchemy.Engine` will register `SQLAlchemy.Engine` and `SQLAlchemy.orm.Session`. +- `EllarSQL.get_all_metadata()` retrieves all configured metadatas +- `EllarSQL.get_metadata()` retrieves metadata by `__database__` key or `default` is no parameter is passed. + +```python +import sqlalchemy as sa +import sqlalchemy.orm as sa_orm +from ellar.app import current_injector + +# get engine from DI +default_engine = current_injector.get(sa.Engine) +# get session from DI +session = current_injector.get(sa_orm.Session) + + +assert isinstance(default_engine, sa.Engine) +assert isinstance(session, sa_orm.Session) +``` +For Async Database options +```python +from sqlalchemy.ext.asyncio import AsyncSession, AsyncEngine +from ellar.app import current_injector + +# get engine from DI +default_engine = current_injector.get(AsyncEngine) +# get session from DI +session = current_injector.get(AsyncSession) + + +assert isinstance(default_engine, AsyncEngine) +assert isinstance(session, AsyncSession) +``` diff --git a/docs/multiple/index.md b/docs/multiple/index.md new file mode 100644 index 0000000..3235689 --- /dev/null +++ b/docs/multiple/index.md @@ -0,0 +1,94 @@ +# **Multiple Databases** + +SQLAlchemy has the capability to establish connections with multiple databases simultaneously, referring to these connections as "binds." + +EllarSQL simplifies the management of binds by associating each engine with a short string identifier, `__database__`. +Subsequently, each model and table is linked to a `__database__`, and during a query, +the session selects the appropriate engine based on the `__database__` of the entity being queried. +In the absence of a specified `__database__`, the default engine is employed. + +## **Configuring Multiple Databases** + +In EllarSQL, database configuration begins with the setup of the default database, +followed by additional databases, as exemplified in the `EllarSQLModule` configurations: + +```python +from ellar_sql import EllarSQLModule + +EllarSQLModule.setup( + databases={ + "default": "postgresql:///main", + "meta": "sqlite:////path/to/meta.db", + "auth": { + "url": "mysql://localhost/users", + "pool_recycle": 3600, + }, + }, + migration_options={'directory': 'migrations'} +) +``` + +## **Defining Models and Tables with Different Databases** + +**EllarSQL** creates **Metadata** and an **Engine** for each configured database. +Models and tables associated with a specific `__database__` key are registered with the corresponding **Metadata**. +During a session query, the session employs the related `Engine`. + +To designate the database for a model, set the `__database__` class attribute. +Not specifying a `__database__` key is equivalent to setting it to `default`: + +### **In Models** + +```python +from ellar_sql import model + +class User(model.Model): + __database__ = "auth" + id = model.Column(model.Integer, primary_key=True) +``` +Models inheriting from an already existing model will share the same `database` key unless they are overriden. + +!!!info + Its importance to not that `model.Model` has `__database__` value equals `default` + +### **In Tables** +To specify the database for a table, utilize the `__database__` keyword argument: + +```python +from ellar_sql import model + +user_table = model.Table( + "user", + model.Column("id", model.Integer, primary_key=True), + __database__="auth", +) +``` + +!!!info + Ultimately, the session references the database key associated with the metadata or table, an association established during creation. + Consequently, changing the **database** key after creating a model or table has **no effect**. + +## **Creating and Dropping Tables** + +The `create_all()` and `drop_all()` methods operating are all part of the `EllarSQLService`. +It also requires the `database` argument to target a specific database. + +```python +# Create tables for all binds +from ellar.app import current_injector +from ellar_sql import EllarSQLService + +db_service = current_injector.get(EllarSQLService) + +# Create tables for all configured databases +db_service.create_all() + +# Create tables for the 'default' and "auth" databases +db_service.create_all('default', "auth") + +# Create tables for the "meta" database +db_service.create_all("meta") + +# Drop tables for the 'default' database +db_service.drop_all('default') +``` diff --git a/docs/pagination/index.md b/docs/pagination/index.md new file mode 100644 index 0000000..2479666 --- /dev/null +++ b/docs/pagination/index.md @@ -0,0 +1,171 @@ +# **Pagination** + +Pagination is a common practice for large datasets, +enhancing user experience by breaking content into manageable pages. +It optimizes load times and navigation and allows users to explore extensive datasets with ease +while maintaining system performance and responsiveness. + +EllarSQL offers two styles of pagination: + +- **PageNumberPagination**: This pagination internally configures items `per_page` and max item size (`max_size`) and, allows users to set the `page` property. +- **LimitOffsetPagination**: This pagination internally configures max item size (`max_limit`) and, allows users to set the `limit` and `offset` properties. + +EllarSQL pagination is activated when a route function is decorated with `paginate` function. +The result of the route function is expected to be a `SQLAlchemy.sql.Select` instance or a `Model` type. + +For example: + +```python +import ellar.common as ec +from ellar_sql import model, paginate +from .models import User +from .schemas import UserSchema + + +@ec.get('/users') +@paginate(item_schema=UserSchema) +def list_users(): + return model.select(User) +``` + +### **paginate properties** + +- **pagination_class**: _t.Optional[t.Type[PaginationBase]]=None_: specifies pagination style to use. if not set, it will be set to `PageNumberPagination` +- **model**: _t.Optional[t.Type[ModelBase]]=None_: specifies a `Model` type to get list of data. If set, route function can return `None` or override by returning a **select/filtered** statement +- **as_template_context**: _bool=False_: indicates that the paginator object be added to template context. See [Template Pagination](#template-pagination) +- **item_schema**: _t.Optional[t.Type[BaseModel]]=None_: This is required if `template_context` is False. It is used to **serialize** the SQLAlchemy model and create a **response-schema/docs**. +- **paginator_options**:_t.Any_: keyword argument for configuring `pagination_class` set to use for pagination. + +## **API Pagination** +API pagination simply means pagination in an API route function. +This requires `item_schema` for the paginate decorator +to create a `200` response documentation for the decorated route and for the paginated result to be serialized to json. + +```python +import ellar.common as ec +from ellar_sql import paginate +from .models import User + + +class UserSchema(ec.Serializer): + id: int + username: str + email: str + + +@ec.get('/users') +@paginate(item_schema=UserSchema, per_page=100) +def list_users(): + return User +``` +We can also rewrite the illustration above since we are not making any modification to the User query. + +```python +... + +@ec.get('/users') +@paginate(model=User, item_schema=UserSchema) +def list_users(): + pass +``` + +## **Template Pagination** +This is for route functions +decorated with [`render`](https://python-ellar.github.io/ellar/overview/custom_decorators/#render) function +that need to be paginated. +For this to happen, `paginate` +function need to return a context and this is achieved by setting `as_template_context=True` + +```python +import ellar.common as ec +from ellar_sql import model, paginate +from .models import User + + +@ec.get('/users') +@ec.render('list.html') +@paginate(as_template_context=True) +def list_users(): + return model.select(User), {'name': 'Template Pagination'} # pagination model, template context +``` +In the illustration above, a tuple of select statement and a template context was returned. +The template context will be updated with a [`paginator`](https://github.com/python-ellar/ellar-sql/blob/master/ellar_sql/pagination/base.py) as an extra key by the `paginate` function +before been processed by `render` function. + +We can re-write the example above to return just the template context since there is no form of +filter directly affecting the `User` model query. +```python +... + +@ec.get('/users') +@ec.render('list.html') +@paginate(model=model.select(User), as_template_context=True) +def list_users(): + return {'name': 'Template Pagination'} +``` +Also, in the `list.html` we have the following codes: +```html + + +

{{ name }}

+{% macro render_pagination(paginator, endpoint) %} +
+ {{ paginator.first }} - {{ paginator.last }} of {{ paginator.total }} +
+
+ {% for page in paginator.iter_pages() %} + {% if page %} + {% if page != paginator.page %} + {{ page }} + {% else %} + {{ page }} + {% endif %} + {% else %} + + {% endif %} + {% endfor %} +
+{% endmacro %} + +
    + {% for user in paginator %} +
  • {{ user.id }} @ {{ user.name }} + {% endfor %} +
+{{render_pagination(paginator=paginator, endpoint="list_users") }} + +``` + +The `paginator` object in the template context has a `iter_pages()` method which produces up to three group of numbers, +seperated by `None`. + +It defaults to showing 2 page numbers at either edge, +2 numbers before the current, the current, and 4 numbers after the current. +For example, if there are 20 pages and the current page is 7, the following values are yielded. +``` +paginator.iter_pages() +[1, 2, None, 5, 6, 7, 8, 9, 10, 11, None, 19, 20] +``` +The `total` attribute showcases the total number of results, while `first` and `last` display the range of items on the current page. + +The accompanying Jinja macro renders a simple pagination widget. +```html +{% macro render_pagination(paginator, endpoint) %} +
+ {{ paginator.first }} - {{ paginator.last }} of {{ paginator.total }} +
+
+ {% for page in paginator.iter_pages() %} + {% if page %} + {% if page != paginator.page %} + {{ page }} + {% else %} + {{ page }} + {% endif %} + {% else %} + + {% endif %} + {% endfor %} +
+{% endmacro %} +``` diff --git a/docs/stylesheets/extra.css b/docs/stylesheets/extra.css new file mode 100644 index 0000000..5699dcb --- /dev/null +++ b/docs/stylesheets/extra.css @@ -0,0 +1,28 @@ +.md-header { + background-color: #005662 !important; +} +nav.md-nav--primary ul.md-nav__list { + text-transform: uppercase; + font-weight: bold; +} + +nav.md-nav--primary ul.md-nav__list nav.md-nav ul.md-nav__list{ + text-transform: none; + font-weight: normal; +} + +nav.md-nav--primary ul.md-nav__list li.md-nav__item--active.md-nav__item--nested > label.md-nav__link{ + color: var(--md-typeset-a-color); +} + +.md-grid { + margin-left: auto; + margin-right: auto; + max-width: 80%; +} + +:root { + --md-primary-fg-color: #00b4cc; + --md-primary-fg-color--light: #2568a7; + --md-primary-fg-color--dark: #00b4cc; +} diff --git a/docs/testing/index.md b/docs/testing/index.md new file mode 100644 index 0000000..ae470db --- /dev/null +++ b/docs/testing/index.md @@ -0,0 +1,311 @@ +# **Testing EllarSQL Models** +There are various approaches to testing SQLAlchemy models, but in this section, we will focus on setting +up a good testing environment for EllarSQL models using the +Ellar [Test](https://python-ellar.github.io/ellar/basics/testing/){target="_blank"} factory and pytest. + +For an effective testing environment, it is recommended to utilize the `EllarSQLModule.register_setup()` +approach to set up the **EllarSQLModule**. This allows you to add a new configuration for `ELLAR_SQL` +specific to your testing database, preventing interference with production or any other databases in use. + +### **Defining TestConfig** +There are various methods for configuring test settings in Ellar, +as outlined +[here](https://python-ellar.github.io/ellar/basics/testing/#overriding-application-conf-during-testing){target="_blank"}. +However, in this section, we will adopt the 'in a file' approach. + +Within the `db_learning/config.py` file, include the following code: + +```python title="db_learning/config.py" +import typing as t +... + +class DevelopmentConfig(BaseConfig): + DEBUG: bool = True + # Configuration through Config + ELLAR_SQL: t.Dict[str, t.Any] = { + 'databases': { + 'default': 'sqlite:///project.db', + }, + 'echo': True, + 'migration_options': { + 'directory': 'migrations' + }, + 'models': ['models'] + } + +class TestConfig(BaseConfig): + DEBUG = False + + ELLAR_SQL: t.Dict[str, t.Any] = { + **DevelopmentConfig.ELLAR_SQL, + 'databases': { + 'default': 'sqlite:///test.db', + }, + 'echo': False, + } +``` + +This snippet demonstrates the 'in a file' approach to setting up the `TestConfig` class within the same `db_learning/config.py` file. + +#### **Changes made:** +1. Updated the `databases` section to use `sqlite+aiosqlite:///test.db` for the testing database. +2. Set `echo` to `True` to enable SQLAlchemy output during testing for cleaner logs. +3. Preserved the `migration_options` and `models` configurations from `DevelopmentConfig`. + +Also, feel free to further adjust it based on your specific testing requirements! + +## **Test Fixtures** +After defining `TestConfig`, we need to add some pytest fixtures to set up **EllarSQLModule** and another one +that returns a `session` for testing purposes. Additionally, we need to export `ELLAR_CONFIG_MODULE` +to point to the newly defined **TestConfig**. + +```python title="tests/conftest.py" +import os +import pytest +from ellar.common.constants import ELLAR_CONFIG_MODULE +from ellar.testing import Test +from ellar_sql import EllarSQLService +from db_learning.root_module import ApplicationModule + +# Setting the ELLAR_CONFIG_MODULE environment variable to TestConfig +os.environ.setdefault(ELLAR_CONFIG_MODULE, "db_learning.config:TestConfig") + +# Fixture for creating a test module +@pytest.fixture(scope='session') +def tm(): + test_module = Test.create_test_module(modules=[ApplicationModule]) + yield test_module + +# Fixture for creating a database session for testing +@pytest.fixture(scope='session') +def db(tm): + db_service = tm.get(EllarSQLService) + + # Creating all tables + db_service.create_all() + + yield + + # Dropping all tables after the tests + db_service.drop_all() + +# Fixture for creating a database session for testing +@pytest.fixture(scope='session') +def db_session(db, tm): + db_service = tm.get(EllarSQLService) + + yield db_service.session_factory() + + # Removing the session factory + db_service.session_factory.remove() +``` + +The provided fixtures help in setting up a testing environment for EllarSQL models. +The `Test.create_test_module` method creates a **TestModule** for initializing your Ellar application, +and the `db_session` fixture initializes a database session for testing, creating and dropping tables as needed. + +If you are working with asynchronous database drivers, you can convert `db_session` +into an async function to handle coroutines seamlessly. + +## **Alembic Migration with Test Fixture** +In cases where there are already generated database migration files, and there is a need to apply migrations during testing, this can be achieved as shown in the example below: + +```python title="tests/conftest.py" +import os +import pytest +from ellar.common.constants import ELLAR_CONFIG_MODULE +from ellar.testing import Test +from ellar_sql import EllarSQLService +from ellar_sql.cli.handlers import CLICommandHandlers +from db_learning.root_module import ApplicationModule + +# Setting the ELLAR_CONFIG_MODULE environment variable to TestConfig +os.environ.setdefault(ELLAR_CONFIG_MODULE, "db_learning.config:TestConfig") + +# Fixture for creating a test module +@pytest.fixture(scope='session') +def tm(): + test_module = Test.create_test_module(modules=[ApplicationModule]) + yield test_module + + +# Fixture for creating a database session for testing +@pytest.fixture(scope='session') +async def db(tm): + db_service = tm.get(EllarSQLService) + + # Applying migrations using Alembic + async with tm.create_application().application_context(): + cli = CLICommandHandlers(db_service) + cli.migrate() + + yield + + # Downgrading migrations after testing + async with tm.create_application().application_context(): + cli = CLICommandHandlers(db_service) + cli.downgrade() + +# Fixture for creating an asynchronous database session for testing +@pytest.fixture(scope='session') +async def db_session(db, tm): + db_service = tm.get(EllarSQLService) + + yield db_service.session_factory() + + # Removing the session factory + db_service.session_factory.remove() +``` + +The `CLICommandHandlers` class wraps all `Alembic` functions executed through the Ellar command-line interface. +It can be used in conjunction with the application context to initialize all model tables during testing as shown in the illustration above. +`db_session` pytest fixture also ensures that migrations are applied and then downgraded after testing, +maintaining a clean and consistent test database state. + +## **Testing a Model** +After setting up the testing database and creating a session, let's test the insertion of a user model into the database. + +In `db_learning/models.py`, we have a user model: + +```python title="db_learning/model.py" +from ellar_sql import model + +class User(model.Model): + id: model.Mapped[int] = model.mapped_column(model.Integer, primary_key=True) + username: model.Mapped[str] = model.mapped_column(model.String, unique=True, nullable=False) + email: model.Mapped[str] = model.mapped_column(model.String) +``` + +Now, create a file named `test_user_model.py`: + +```python title="tests/test_user_model.py" +import pytest +import sqlalchemy.exc as sa_exc +from db_learning.models import User + +def test_username_must_be_unique(db_session): + # Creating and adding the first user + user1 = User(username='ellarSQL', email='ellarsql@gmail.com') + db_session.add(user1) + db_session.commit() + + # Attempting to add a second user with the same username + user2 = User(username='ellarSQL', email='ellarsql2@gmail.com') + db_session.add(user2) + + # Expecting an IntegrityError due to unique constraint violation + with pytest.raises(sa_exc.IntegrityError): + db_session.commit() +``` + +In this test, we are checking whether the unique constraint on the `username` +field is enforced by attempting to insert two users with the same username. +The test expects an `IntegrityError` to be raised, indicating a violation of the unique constraint. +This ensures that the model behaves correctly and enforces the specified uniqueness requirement. + +## **Testing Factory Boy** +[factory-boy](https://pypi.org/project/factory-boy/){target="_blank"} provides a convenient and flexible way to create mock objects, supporting various ORMs like Django, MongoDB, and SQLAlchemy. EllarSQL extends `factory.alchemy.SQLAlchemy` to offer a Model factory solution compatible with both synchronous and asynchronous database drivers. + +To get started, you need to install `factory-boy`: + +```shell +pip install factory-boy +``` + +Now, let's create a factory for our user model in `tests/factories.py`: + +```python title="tests/factories.py" +import factory +from ellar_sql.factory import EllarSQLFactory, SESSION_PERSISTENCE_FLUSH +from db_learning.models import User +from . import common + +class UserFactory(EllarSQLFactory): + class Meta: + model = User + sqlalchemy_session_persistence = SESSION_PERSISTENCE_FLUSH + sqlalchemy_session_factory = lambda: common.Session() + + username = factory.Faker('username') + email = factory.Faker('email') +``` + +The `UserFactory` depends on a database session. Since the pytest fixture we created applies to it, +we also need a session factory in `tests/common.py`: + +```python title="tests/common.py" +from sqlalchemy import orm + +Session = orm.scoped_session(orm.sessionmaker()) +``` + +Additionally, we require a fixture responsible for configuring the Factory session in `tests/conftest.py`: + +```python title="tests/conftest.py" +import os +import pytest +import sqlalchemy as sa +from ellar.common.constants import ELLAR_CONFIG_MODULE +from ellar.testing import Test +from ellar_sql import EllarSQLService +from db_learning.root_module import ApplicationModule +from . import common + +os.environ.setdefault(ELLAR_CONFIG_MODULE, "db_learning.config:TestConfig") + +@pytest.fixture(scope='session') +def tm(): + test_module = Test.create_test_module(modules=[ApplicationModule]) + yield test_module + +# Fixture for creating a database session for testing +@pytest.fixture(scope='session') +def db(tm): + db_service = tm.get(EllarSQLService) + + # Creating all tables + db_service.create_all() + + yield + + # Dropping all tables after the tests + db_service.drop_all() + +# Fixture for creating a database session for testing +@pytest.fixture(scope='session') +def db_session(db, tm): + db_service = tm.get(EllarSQLService) + + yield db_service.session_factory() + + # Removing the session factory + db_service.session_factory.remove() + +@pytest.fixture +def factory_session(db, tm): + engine = tm.get(sa.Engine) + common.Session.configure(bind=engine) + yield + common.Session.remove() +``` + +In the `factory_session` fixture, we retrieve the `Engine` registered in the DI container by **EllarSQLModule**. +Using this engine, we configure the common `Session`. It's important to note that if you are using an +async database driver, **EllarSQLModule** will register `AsyncEngine`. + +With this setup, we can rewrite our `test_username_must_be_unique` test using `UserFactory` and `factory_session`: + +```python title="tests/test_user_model.py" +import pytest +import sqlalchemy.exc as sa_exc +from .factories import UserFactory + +def test_username_must_be_unique(factory_session): + user1 = UserFactory() + with pytest.raises(sa_exc.IntegrityError): + UserFactory(username=user1.username) +``` + +This test yields the same result as before. +Refer to the [factory-boy documentation](https://factoryboy.readthedocs.io/en/stable/orms.html#sqlalchemy) +for more features and tutorials. diff --git a/ellar_sql/__init__.py b/ellar_sql/__init__.py index cbbc373..517946c 100644 --- a/ellar_sql/__init__.py +++ b/ellar_sql/__init__.py @@ -1,18 +1,16 @@ -"""Ellar SQLAlchemy Module - Adds support for SQLAlchemy and Alembic package to your Ellar web Framework""" +"""EllarSQL Module adds support for SQLAlchemy and Alembic package to your Ellar application""" __version__ = "0.0.1" +from .model.database_binds import get_all_metadata, get_metadata from .module import EllarSQLModule from .pagination import LimitOffsetPagination, PageNumberPagination, paginate from .query import ( first_or_404, - first_or_404_async, get_or_404, - get_or_404_async, one_or_404, - one_or_404_async, ) -from .schemas import MigrationOption, SQLAlchemyConfig +from .schemas import MigrationOption, ModelBaseConfig, SQLAlchemyConfig from .services import EllarSQLService __all__ = [ @@ -20,13 +18,13 @@ "EllarSQLService", "SQLAlchemyConfig", "MigrationOption", - "get_or_404_async", "get_or_404", - "first_or_404_async", "first_or_404", - "one_or_404_async", "one_or_404", "paginate", "PageNumberPagination", "LimitOffsetPagination", + "ModelBaseConfig", + "get_metadata", + "get_all_metadata", ] diff --git a/ellar_sql/cli/commands.py b/ellar_sql/cli/commands.py index dee6a53..d374806 100644 --- a/ellar_sql/cli/commands.py +++ b/ellar_sql/cli/commands.py @@ -406,13 +406,20 @@ def check(ctx: click.Context, directory): default=None, help='Migration script directory (default is "migrations")', ) +@click.option( + "-m", + "--multiple", + default=False, + is_flag=True, + help='Use multiple migration template (default is "False")', +) @click.option( "--package", is_flag=True, help="Write empty __init__.py files to the environment and " "version locations", ) @click.pass_context -def init(ctx: click.Context, directory, package): +def init(ctx: click.Context, directory, multiple, package): """Creates a new migration repository.""" handler = _get_handler_context(ctx) - handler.alembic_init(directory, package) + handler.alembic_init(directory, multiple, package) diff --git a/ellar_sql/cli/handlers.py b/ellar_sql/cli/handlers.py index 53a9676..88a3536 100644 --- a/ellar_sql/cli/handlers.py +++ b/ellar_sql/cli/handlers.py @@ -75,7 +75,12 @@ def get_config( return config @_catch_errors - def alembic_init(self, directory: str | None = None, package: bool = False) -> None: + def alembic_init( + self, + directory: str | None = None, + multiple: bool = False, + package: bool = False, + ) -> None: """Creates a new migration repository""" if directory is None: directory = self.db_service.migration_options.directory @@ -84,7 +89,12 @@ def alembic_init(self, directory: str | None = None, package: bool = False) -> N config.set_main_option("script_location", directory) config.config_file_name = os.path.join(directory, "alembic.ini") - command.init(config, directory, template="basic", package=package) + template_name = "single" + + if multiple: + template_name = "multiple" + + command.init(config, directory, template=template_name, package=package) @_catch_errors def revision( diff --git a/ellar_sql/exceptions.py b/ellar_sql/exceptions.py deleted file mode 100644 index 953e705..0000000 --- a/ellar_sql/exceptions.py +++ /dev/null @@ -1,6 +0,0 @@ -# class NoMatch(Exception): -# pass -# -# -# class MultipleMatches(Exception): -# pass diff --git a/ellar_sql/factory/__init__.py b/ellar_sql/factory/__init__.py new file mode 100644 index 0000000..fed30b6 --- /dev/null +++ b/ellar_sql/factory/__init__.py @@ -0,0 +1,16 @@ +try: + import factory +except ImportError as im_ex: # pragma: no cover + raise RuntimeError( + "factory-boy not found. Please run `pip install factory-boy`" + ) from im_ex + +from factory.alchemy import SESSION_PERSISTENCE_COMMIT, SESSION_PERSISTENCE_FLUSH + +from .base import EllarSQLFactory + +__all__ = [ + "EllarSQLFactory", + "SESSION_PERSISTENCE_COMMIT", + "SESSION_PERSISTENCE_FLUSH", +] diff --git a/ellar_sql/factory/base.py b/ellar_sql/factory/base.py new file mode 100644 index 0000000..e34adde --- /dev/null +++ b/ellar_sql/factory/base.py @@ -0,0 +1,114 @@ +import typing as t + +import sqlalchemy as sa +import sqlalchemy.orm as sa_orm +from ellar.threading import execute_coroutine_with_sync_worker +from factory.alchemy import ( + SESSION_PERSISTENCE_COMMIT, + SESSION_PERSISTENCE_FLUSH, + SQLAlchemyModelFactory, + SQLAlchemyOptions, +) +from factory.errors import FactoryError +from sqlalchemy.exc import IntegrityError, NoResultFound +from sqlalchemy.ext.asyncio import AsyncSession + +from ellar_sql.model.base import ModelBase + +T = t.TypeVar("T", bound=ModelBase) + + +class EllarSQLOptions(SQLAlchemyOptions): + @staticmethod + def _check_has_sqlalchemy_session_set(meta, value): + if value and hasattr(meta, "sqlalchemy_session"): + raise RuntimeError( + "Provide either a sqlalchemy_session or a sqlalchemy_session_factory, not both" + ) + + +class EllarSQLFactory(SQLAlchemyModelFactory): + """Factory for EllarSQL models.""" + + _options_class = EllarSQLOptions + + class Meta: + abstract = True + + @classmethod + def _session_execute( + cls, session_func: t.Callable, *args: t.Any, **kwargs: t.Any + ) -> t.Union[sa.Result, sa.CursorResult, t.Any]: + res = session_func(*args, **kwargs) + if isinstance(res, t.Coroutine): + res = execute_coroutine_with_sync_worker(res) + return res + + @classmethod + def _get_or_create( + cls, + model_class: t.Type[T], + session: t.Union[sa_orm.Session, AsyncSession], + args: t.Tuple[t.Any], + kwargs: t.Dict[str, t.Any], + ): + key_fields = {} + for field in cls._meta.sqlalchemy_get_or_create: + if field not in kwargs: + raise FactoryError( + "sqlalchemy_get_or_create - " + "Unable to find initialization value for '%s' in factory %s" + % (field, cls.__name__) + ) + key_fields[field] = kwargs.pop(field) + stmt = sa.select(model_class).filter_by(*args, **key_fields) # type:ignore[call-arg] + + res = cls._session_execute(session.execute, stmt) + obj = res.scalar() + + if not obj: + try: + obj = cls._save(model_class, session, args, {**key_fields, **kwargs}) + except IntegrityError as e: + cls._session_execute(session.rollback) + + if cls._original_params is None: + raise e + + get_or_create_params = { + lookup: value + for lookup, value in cls._original_params.items() + if lookup in cls._meta.sqlalchemy_get_or_create + } + if get_or_create_params: + try: + stmt = sa.select(model_class).filter_by(**get_or_create_params) + res = cls._session_execute(session.execute, stmt) + obj = res.scalar_one() + except NoResultFound: + # Original params are not a valid lookup and triggered a create(), + # that resulted in an IntegrityError. + raise e from None + else: + raise e + + return obj + + @classmethod + def _save( + cls, + model_class: t.Type[T], + session: t.Union[sa_orm.Session, AsyncSession], + args: t.Tuple[t.Any], + kwargs: t.Dict[str, t.Any], + ) -> T: + session_persistence = cls._meta.sqlalchemy_session_persistence + + obj = model_class(*args, **kwargs) # type:ignore[call-arg] + session.add(obj) + if session_persistence == SESSION_PERSISTENCE_FLUSH: + cls._session_execute(session.flush) + elif session_persistence == SESSION_PERSISTENCE_COMMIT: + cls._session_execute(session.commit) + cls._session_execute(session.refresh, obj) + return obj diff --git a/ellar_sql/migrations/multiple.py b/ellar_sql/migrations/multiple.py index 13e5e28..2f396cf 100644 --- a/ellar_sql/migrations/multiple.py +++ b/ellar_sql/migrations/multiple.py @@ -5,7 +5,7 @@ import sqlalchemy as sa from sqlalchemy.ext.asyncio import AsyncConnection, AsyncEngine -from ellar_sql.model.database_binds import get_database_bind +from ellar_sql.model.database_binds import get_metadata from ellar_sql.types import RevisionArgs from .base import AlembicEnvMigrationBase @@ -81,7 +81,9 @@ def default_process_revision_directives( directives[:] = [] logger.info("No changes in schema detected.") - def run_migrations_offline(self, context: "EnvironmentContext") -> None: + def run_migrations_offline( + self, context: "EnvironmentContext" + ) -> None: # pragma:no cover """Run migrations in 'offline' mode. This configures the context with just a URL @@ -102,7 +104,7 @@ def run_migrations_offline(self, context: "EnvironmentContext") -> None: logger.info("Migrating database %s" % key) url = str(engine.url).replace("%", "%%") - metadata = get_database_bind(key, certain=True) + metadata = get_metadata(key, certain=True).metadata file_ = "%s.sql" % key logger.info("Writing output to %s" % file_) @@ -168,7 +170,7 @@ async def _compute_engine_info(self) -> t.List[DatabaseInfo]: res = [] for key, engine in self.db_service.engines.items(): - metadata = get_database_bind(key, certain=True) + metadata = get_metadata(key, certain=True).metadata if engine.dialect.is_async: async_engine = AsyncEngine(engine) diff --git a/ellar_sql/migrations/single.py b/ellar_sql/migrations/single.py index 4d2eb57..4b87148 100644 --- a/ellar_sql/migrations/single.py +++ b/ellar_sql/migrations/single.py @@ -5,7 +5,7 @@ import sqlalchemy as sa from sqlalchemy.ext.asyncio import AsyncEngine -from ellar_sql.model.database_binds import get_database_bind +from ellar_sql.model.database_binds import get_metadata from ellar_sql.types import RevisionArgs from .base import AlembicEnvMigrationBase @@ -37,7 +37,9 @@ def default_process_revision_directives( directives[:] = [] logger.info("No changes in schema detected.") - def run_migrations_offline(self, context: "EnvironmentContext") -> None: + def run_migrations_offline( + self, context: "EnvironmentContext" + ) -> None: # pragma:no cover """Run migrations in 'offline' mode. This configures the context with just a URL @@ -51,7 +53,7 @@ def run_migrations_offline(self, context: "EnvironmentContext") -> None: """ key, engine = self.db_service.engines.popitem() - metadata = get_database_bind(key, certain=True) + metadata = get_metadata(key, certain=True).metadata conf_args = self.get_user_context_configurations() @@ -93,7 +95,7 @@ async def run_migrations_online(self, context: "EnvironmentContext") -> None: """ key, engine = self.db_service.engines.popitem() - metadata = get_database_bind(key, certain=True) + metadata = get_metadata(key, certain=True).metadata migration_action_partial = functools.partial( self._migration_action, metadata=metadata, context=context diff --git a/ellar_sql/model/__init__.py b/ellar_sql/model/__init__.py index 8ecf7ec..b236b6b 100644 --- a/ellar_sql/model/__init__.py +++ b/ellar_sql/model/__init__.py @@ -6,15 +6,7 @@ from .base import Model from .table import Table -from .typeDecorator import ( - GUID, - CroppingDetails, - FileField, - FileObject, - GenericIP, - ImageFileField, - ImageFileObject, -) +from .typeDecorator import GUID, GenericIP from .utils import make_metadata if t.TYPE_CHECKING: @@ -24,18 +16,7 @@ from .table import Table -__all__ = [ - "Model", - "Table", - "make_metadata", - "GUID", - "GenericIP", - "FileObject", - "FileField", - "ImageFileField", - "ImageFileObject", - "CroppingDetails", -] +__all__ = ["Model", "Table", "make_metadata", "GUID", "GenericIP"] def __getattr__(name: str) -> t.Any: diff --git a/ellar_sql/model/base.py b/ellar_sql/model/base.py index 4096526..4bb5d9f 100644 --- a/ellar_sql/model/base.py +++ b/ellar_sql/model/base.py @@ -9,7 +9,7 @@ from ellar_sql.constant import DATABASE_BIND_KEY, DATABASE_KEY, DEFAULT_KEY from ellar_sql.schemas import ModelBaseConfig -from .database_binds import get_database_bind, has_database_bind, update_database_binds +from .database_binds import get_metadata, has_metadata, update_database_metadata from .mixins import ( DatabaseBindKeyMixin, ModelDataExportMixin, @@ -23,10 +23,12 @@ def _update_metadata(namespace: t.Dict[str, t.Any]) -> None: metadata = namespace.get("metadata", None) if metadata and database_key: - if not has_database_bind(database_key): + if not has_metadata(database_key): # verify the key exist and store the metadata metadata.info[DATABASE_BIND_KEY] = database_key - update_database_binds(database_key, metadata) + update_database_metadata( + database_key, metadata, sa_orm.registry(metadata=metadata) + ) # if we have saved metadata then, its save to remove it and allow # DatabaseBindKeyMixin to set it back when the class if fully created. namespace.pop("metadata") @@ -56,9 +58,7 @@ def __new__( options = namespace.pop( "__base_config__", - ModelBaseConfig( - make_declarative_base=False, use_bases=[sa_orm.DeclarativeBase] - ), + ModelBaseConfig(as_base=False, use_bases=[sa_orm.DeclarativeBase]), ) if isinstance(options, dict): options = ModelBaseConfig(**options) @@ -66,7 +66,7 @@ def __new__( options, ModelBaseConfig ), f"{options.__class__} is not a support ModelMetaOptions" - if options.make_declarative_base: + if options.as_base: declarative_bases = _get_declarative_bases(options.use_bases) working_base = list(bases) @@ -109,29 +109,30 @@ def __new__( lambda ns: ns.update(namespace), ) - if not has_database_bind(DEFAULT_KEY): + # model = t.cast(t.Type[sa_orm.DeclarativeBase], model) + + if not has_metadata(DEFAULT_KEY): # Use the model's metadata as the default metadata. model.metadata.info[DATABASE_BIND_KEY] = DEFAULT_KEY - update_database_binds(DEFAULT_KEY, model.metadata) - elif not has_database_bind(model.metadata.info.get(DATABASE_BIND_KEY)): + update_database_metadata(DEFAULT_KEY, model.metadata, model.registry) + elif not has_metadata(model.metadata.info.get(DATABASE_BIND_KEY)): # Use the passed in default metadata as the model's metadata. - model.metadata = get_database_bind(DEFAULT_KEY, certain=True) + model.metadata = get_metadata(DEFAULT_KEY, certain=True) return model # _update_metadata(namespace) + __base_config__ = ModelBaseConfig(use_bases=options.use_bases, as_base=True) base = ModelMeta( "ModelBase", bases, - { - "__base_config__": ModelBaseConfig( - use_bases=options.use_bases, make_declarative_base=True - ) - }, + {"__base_config__": __base_config__}, ) - return types.new_class(name, (base,), {}, lambda ns: ns.update(namespace)) + return types.new_class( + name, (base,), {"options": options}, lambda ns: ns.update(namespace) + ) class ModelBase(ModelDataExportMixin): @@ -159,9 +160,6 @@ class ModelBase(ModelDataExportMixin): def __init__(self, **kwargs: t.Any) -> None: ... - def dict(self, exclude: t.Optional[t.Set[str]] = None) -> t.Dict[str, t.Any]: - ... - def _sa_inspect_type(self) -> sa.Mapper["ModelBase"]: ... diff --git a/ellar_sql/model/database_binds.py b/ellar_sql/model/database_binds.py index a55c32a..02d6b63 100644 --- a/ellar_sql/model/database_binds.py +++ b/ellar_sql/model/database_binds.py @@ -1,23 +1,50 @@ import typing as t import sqlalchemy as sa +import sqlalchemy.orm as sa_orm -__model_database_metadata__: t.Dict[str, sa.MetaData] = {} +from ellar_sql.constant import DEFAULT_KEY -def update_database_binds(key: str, value: sa.MetaData) -> None: - __model_database_metadata__[key] = value +class DatabaseMetadata(t.NamedTuple): + metadata: sa.MetaData + registry: sa_orm.registry -def get_database_binds() -> t.Dict[str, sa.MetaData]: +__model_database_metadata__: t.Dict[str, DatabaseMetadata] = {} + + +def update_database_metadata( + database_key: str, value: sa.MetaData, registry: sa_orm.registry +) -> None: + """ + Update a metadata based on a database key + """ + __model_database_metadata__[database_key] = DatabaseMetadata( + metadata=value, registry=registry + ) + + +def get_all_metadata() -> t.Dict[str, DatabaseMetadata]: + """ + Get all metadata available in your application + """ return __model_database_metadata__.copy() -def get_database_bind(key: str, certain: bool = False) -> sa.MetaData: +def get_metadata( + database_key: str = DEFAULT_KEY, certain: bool = False +) -> DatabaseMetadata: + """ + Gets Metadata associated with a database key + """ if certain: - return __model_database_metadata__[key] - return __model_database_metadata__.get(key) # type:ignore[return-value] + return __model_database_metadata__[database_key] + return __model_database_metadata__.get(database_key) # type:ignore[return-value] -def has_database_bind(key: str) -> bool: - return key in __model_database_metadata__ +def has_metadata(database_key: str) -> bool: + """ + Checks if a metadata exist for a database key + """ + return database_key in __model_database_metadata__ diff --git a/ellar_sql/model/mixins.py b/ellar_sql/model/mixins.py index 9dd508d..158c010 100644 --- a/ellar_sql/model/mixins.py +++ b/ellar_sql/model/mixins.py @@ -1,10 +1,23 @@ import typing as t import sqlalchemy as sa +import sqlalchemy.orm as sa_orm +from pydantic.v1 import BaseModel from ellar_sql.constant import ABSTRACT_KEY, DATABASE_KEY, DEFAULT_KEY, TABLE_KEY +from ellar_sql.model.utils import ( + camel_to_snake_case, + make_metadata, + should_set_table_name, +) +from ellar_sql.schemas import ModelBaseConfig, ModelMetaStore -from .utils import camel_to_snake_case, make_metadata, should_set_table_name + +class asss(BaseModel): + sd: str + + +IncEx = t.Union[t.Set[int], t.Set[str], t.Dict[int, t.Any], t.Dict[str, t.Any]] if t.TYPE_CHECKING: from .base import ModelBase @@ -16,78 +29,6 @@ def get_registered_models() -> t.Dict[str, t.Type["ModelBase"]]: return __ellar_sqlalchemy_models__.copy() -# -# def __name_mixin__(cls: t.Type, *args:t.Any, is_subclass:bool=False, **kwargs:t.Any) -> None: -# if should_set_table_name(cls) and not kwargs.get('__skip_mixins__'): -# cls.__tablename__ = camel_to_snake_case(cls.__name__) -# -# if is_subclass: -# super(NameMixin, cls).__init_subclass__(**kwargs) -# else: -# super(NameMetaMixin, cls).__init__(*args, **kwargs) -# -# -# def __database_key_bind__(cls: t.Type, *args: t.Any, is_subclass: bool = False, **kwargs: t.Any) -> None: -# if not ("metadata" in cls.__dict__ or TABLE_KEY in cls.__dict__) and hasattr( -# cls, DATABASE_KEY -# ) and not kwargs.get('__skip_mixins__'): -# database_bind_key = getattr(cls, DATABASE_KEY, DEFAULT_KEY) -# parent_metadata = getattr(cls, "metadata", None) -# metadata = make_metadata(database_bind_key) -# -# if metadata is not parent_metadata: -# cls.metadata = metadata -# if is_subclass: -# super(DatabaseBindKeyMixin, cls).__init_subclass__(**kwargs) -# else: -# super(DatabaseBindKeyMetaMixin, cls).__init__(*args, **kwargs) -# -# -# def __model_track__(cls: t.Type, *args: t.Any, is_subclass: bool = False, **kwargs: t.Any) -> None: -# if is_subclass: -# super(ModelTrackMixin, cls).__init_subclass__(**kwargs) -# else: -# super(ModelTrackMetaMixin, cls).__init__(*args, **kwargs) -# -# if TABLE_KEY in cls.__dict__ and ABSTRACT_KEY not in cls.__dict__ and not kwargs.get('__skip_mixins__'): -# __ellar_sqlalchemy_models__[str(cls)] = cls # type:ignore[assignment] -# -# -# NameMixin = types.new_class( -# "NameMixin", (), {}, -# lambda ns: ns.update({"__init_subclass__": lambda cls, **kw: __name_mixin__(cls, **kw, is_subclass=True)}) -# ) -# -# NameMetaMixin = types.new_class( -# "NameMixin", (type, ), {}, -# lambda ns: ns.update({"__init__": __name_mixin__}) -# ) -# -# DatabaseBindKeyMixin = types.new_class( -# "DatabaseBindKeyMixin", (), {}, -# lambda ns: ns.update({"__init_subclass__": lambda cls, **kw: __database_key_bind__(cls, **kw, is_subclass=True)}) -# ) -# -# DatabaseBindKeyMetaMixin = types.new_class( -# "DatabaseBindKeyMetaMixin", (type,), {}, -# lambda ns: ns.update({"__init__": __database_key_bind__}) -# ) -# -# ModelTrackMixin = types.new_class( -# "ModelTrackMixin", (), {}, -# lambda ns: ns.update({"__init_subclass__": lambda cls, **kw: __model_track__(cls, **kw, is_subclass=True)}) -# ) -# -# ModelTrackMetaMixin = types.new_class( -# "ModelTrackMetaMixin", (type,), {}, -# lambda ns: ns.update({"__init__": __model_track__}) -# ) -# -# -# class DefaultBaseMeta(DatabaseBindKeyMetaMixin, NameMetaMixin, ModelTrackMetaMixin, type(DeclarativeBase)): -# pass - - class NameMixin: metadata: sa.MetaData __tablename__: str @@ -102,7 +43,6 @@ def __init_subclass__(cls, **kwargs: t.Dict[str, t.Any]) -> None: class DatabaseBindKeyMixin: metadata: sa.MetaData - __dnd__ = "Ellar" def __init_subclass__(cls, **kwargs: t.Dict[str, t.Any]) -> None: if not ("metadata" in cls.__dict__ or TABLE_KEY in cls.__dict__) and hasattr( @@ -110,25 +50,41 @@ def __init_subclass__(cls, **kwargs: t.Dict[str, t.Any]) -> None: ): database_bind_key = getattr(cls, DATABASE_KEY, DEFAULT_KEY) parent_metadata = getattr(cls, "metadata", None) - metadata = make_metadata(database_bind_key) + db_metadata = make_metadata(database_bind_key) - if metadata is not parent_metadata: - cls.metadata = metadata + if db_metadata.metadata is not parent_metadata: + cls.metadata = db_metadata.metadata + cls.registry = db_metadata.registry # type:ignore[attr-defined] super().__init_subclass__(**kwargs) class ModelTrackMixin: metadata: sa.MetaData + __mms__: ModelMetaStore + __table__: sa.Table def __init_subclass__(cls, **kwargs: t.Dict[str, t.Any]) -> None: + options: ModelBaseConfig = kwargs.pop( # type:ignore[assignment] + "options", + ModelBaseConfig(as_base=False, use_bases=[sa_orm.DeclarativeBase]), + ) + super().__init_subclass__(**kwargs) if TABLE_KEY in cls.__dict__ and ABSTRACT_KEY not in cls.__dict__: __ellar_sqlalchemy_models__[str(cls)] = cls # type:ignore[assignment] + cls.__mms__ = ModelMetaStore( + base_config=options, + pk_column=None, + columns=list(cls.__table__.columns), # type:ignore[arg-type] + ) + class ModelDataExportMixin: + __mms__: t.Optional[ModelMetaStore] = None + def __repr__(self) -> str: state = sa.inspect(self) assert state is not None @@ -142,13 +98,52 @@ def __repr__(self) -> str: return f"<{type(self).__name__} {pk}>" - def dict(self, exclude: t.Optional[t.Set[str]] = None) -> t.Dict[str, t.Any]: + def _calculate_keys( + self, + data: t.Dict[str, t.Any], + include: t.Optional[t.Set[str]], + exclude: t.Optional[t.Set[str]], + ) -> t.Set[str]: + keys: t.Set[str] = set(data.keys()) + + if include is None and exclude is None: + return keys + + if include is not None: + keys &= include + + if exclude: + keys -= exclude + + return keys + + def _iter( + self, + include: t.Optional[t.Set[str]], + exclude: t.Optional[t.Set[str]], + exclude_none: bool = False, + ) -> t.Generator[t.Tuple[str, t.Any], None, None]: + data = dict(self.__dict__) + + if len(data.keys()) != len(self.__mms__.columns): + data = {c.key: getattr(self, c.key, None) for c in self.__mms__.columns} + + allowed_keys = self._calculate_keys(include=include, exclude=exclude, data=data) + + for field_key, v in data.items(): + if (allowed_keys is not None and field_key not in allowed_keys) or ( + exclude_none and v is None + ): + continue + yield field_key, v + + def dict( + self, + include: t.Optional[t.Set[str]] = None, + exclude: t.Optional[t.Set[str]] = None, + exclude_none: bool = False, + ) -> t.Dict[str, t.Any]: # TODO: implement advance exclude and include that goes deep into relationships too - _exclude: t.Set[str] = set() if not exclude else exclude - - tuple_generator = ( - (k, v) - for k, v in self.__dict__.items() - if k not in _exclude and not k.startswith("_sa") + return dict( + self._iter(include=include, exclude_none=exclude_none, exclude=exclude) ) - return dict(tuple_generator) diff --git a/ellar_sql/model/table.py b/ellar_sql/model/table.py index 5e0dd50..3f816ca 100644 --- a/ellar_sql/model/table.py +++ b/ellar_sql/model/table.py @@ -59,5 +59,7 @@ def __new__( if not args or (len(args) >= 2 and isinstance(args[1], sa.MetaData)): return super().__new__(cls, *args, **kwargs) - metadata = make_metadata(__database__ or DEFAULT_KEY) - return super().__new__(cls, *[args[0], metadata, *args[1:]], **kwargs) + db_metadata = make_metadata(__database__ or DEFAULT_KEY) + return super().__new__( + cls, *[args[0], db_metadata.metadata, *args[1:]], **kwargs + ) diff --git a/ellar_sql/model/typeDecorator/__init__.py b/ellar_sql/model/typeDecorator/__init__.py index 7a31880..c965d1c 100644 --- a/ellar_sql/model/typeDecorator/__init__.py +++ b/ellar_sql/model/typeDecorator/__init__.py @@ -1,15 +1,7 @@ -from .file import FileField, FileFieldBase, FileObject from .guid import GUID -from .image import CroppingDetails, ImageFileField, ImageFileObject from .ipaddress import GenericIP __all__ = [ "GUID", "GenericIP", - "CroppingDetails", - "FileField", - "ImageFileField", - "FileObject", - "FileFieldBase", - "ImageFileObject", ] diff --git a/ellar_sql/model/typeDecorator/file/__init__.py b/ellar_sql/model/typeDecorator/file/__init__.py deleted file mode 100644 index b2aeb00..0000000 --- a/ellar_sql/model/typeDecorator/file/__init__.py +++ /dev/null @@ -1,9 +0,0 @@ -from .base import FileFieldBase -from .field import FileField -from .file_info import FileObject - -__all__ = [ - "FileObject", - "FileField", - "FileFieldBase", -] diff --git a/ellar_sql/model/typeDecorator/file/base.py b/ellar_sql/model/typeDecorator/file/base.py deleted file mode 100644 index 9f36d1d..0000000 --- a/ellar_sql/model/typeDecorator/file/base.py +++ /dev/null @@ -1,158 +0,0 @@ -import json -import time -import typing as t -import uuid -from abc import abstractmethod - -import sqlalchemy as sa -from ellar.common import UploadFile -from ellar.core.files.storages import BaseStorage -from ellar.core.files.storages.utils import get_valid_filename - -from ellar_sql.model.typeDecorator.exceptions import ( - ContentTypeValidationError, - InvalidFileError, - MaximumAllowedFileLengthError, -) -from ellar_sql.model.typeDecorator.mimetypes import ( - guess_extension, - magic_mime_from_buffer, -) -from ellar_sql.model.utils import get_length - -from .file_info import FileObject - -T = t.TypeVar("T", bound="FileObject") - - -class FileFieldBase(t.Generic[T]): - @property - @abstractmethod - def file_object_type(self) -> t.Type[T]: - ... - - def load_dialect_impl(self, dialect: sa.Dialect) -> t.Any: - if dialect.name == "sqlite": - return dialect.type_descriptor(sa.String()) - else: - return dialect.type_descriptor(sa.JSON()) - - def __init__( - self, - *args: t.Any, - storage: t.Optional[BaseStorage] = None, - allowed_content_types: t.Optional[t.List[str]] = None, - max_size: t.Optional[int] = None, - **kwargs: t.Any, - ): - if allowed_content_types is None: - allowed_content_types = [] - super().__init__(*args, **kwargs) - - self._storage = storage - self._allowed_content_types = allowed_content_types - self._max_size = max_size - - @property - def storage(self) -> BaseStorage: - assert self._storage - return self._storage - - def validate(self, file: T) -> None: - if ( - self._allowed_content_types - and file.content_type not in self._allowed_content_types - ): - raise ContentTypeValidationError( - file.content_type, self._allowed_content_types - ) - if self._max_size and file.file_size > self._max_size: - raise MaximumAllowedFileLengthError(self._max_size) - - self.storage.validate_file_name(file.filename) - - def load_from_str(self, data: str) -> T: - data_dict = t.cast(t.Dict[str, t.Any], json.loads(data)) - return self.load(data_dict) - - def load(self, data: t.Dict[str, t.Any]) -> T: - if "service_name" in data: - data.pop("service_name") - return self.file_object_type(storage=self.storage, **data) - - def _guess_content_type(self, file: t.IO) -> str: # type:ignore[type-arg] - content = file.read(1024) - - if isinstance(content, str): - content = str.encode(content) - - file.seek(0) - - return magic_mime_from_buffer(content) - - def get_extra_file_initialization_context( - self, file: UploadFile - ) -> t.Dict[str, t.Any]: - return {} - - def convert_to_file_object(self, file: UploadFile) -> T: - unique_name = str(uuid.uuid4()) - - original_filename = file.filename or unique_name - - # use python magic to get the content type - content_type = self._guess_content_type(file.file) or "" - extension = guess_extension(content_type) - file_size = get_length(file.file) - if extension: - saved_filename = ( - f"{original_filename[:-len(extension)]}_{unique_name[:-8]}{extension}" - ) - else: - saved_filename = f"{unique_name[:-8]}_{original_filename}" - saved_filename = get_valid_filename(saved_filename) - - init_kwargs = self.get_extra_file_initialization_context(file) - init_kwargs.update( - storage=self.storage, - original_filename=original_filename, - uploaded_on=int(time.time()), - content_type=content_type, - extension=extension, - file_size=file_size, - saved_filename=saved_filename, - ) - return self.file_object_type(**init_kwargs) - - def process_bind_param_action( - self, value: t.Optional[t.Any], dialect: sa.Dialect - ) -> t.Optional[t.Union[str, t.Dict[str, t.Any]]]: - if value is None: - return value - - if isinstance(value, UploadFile): - value.file.seek(0) # make sure we are always at the beginning - file_obj = self.convert_to_file_object(value) - self.validate(file_obj) - - self.storage.put(file_obj.filename, value.file) - value = file_obj - - if isinstance(value, FileObject): - if dialect.name == "sqlite": - return json.dumps(value.to_dict()) - return value.to_dict() - - raise InvalidFileError(f"{value} is not supported") - - def process_result_value_action( - self, value: t.Optional[t.Any], dialect: sa.Dialect - ) -> t.Optional[t.Union[str, t.Dict[str, t.Any], t.Any]]: - if value is None: - return value - else: - if isinstance(value, str): - value = self.load_from_str(value) - elif isinstance(value, dict): - value = self.load(value) - return value diff --git a/ellar_sql/model/typeDecorator/file/field.py b/ellar_sql/model/typeDecorator/file/field.py deleted file mode 100644 index 6f7ea7f..0000000 --- a/ellar_sql/model/typeDecorator/file/field.py +++ /dev/null @@ -1,46 +0,0 @@ -import typing as t - -import sqlalchemy as sa - -from .base import FileFieldBase -from .file_info import FileObject - - -class FileField(FileFieldBase[FileObject], sa.TypeDecorator): # type: ignore[type-arg] - - """ - Provide SqlAlchemy TypeDecorator for saving files - ## Basic Usage - - fs = FileSystemStorage('path/to/save/files') - - class MyTable(Base): - image: FileField.FileObject = sa.Column( - ImageFileField(storage=fs, max_size=10*MB, allowed_content_type=["application/pdf"]), - nullable=True - ) - - def route(file: File[UploadFile]): - session = SessionLocal() - my_table_model = MyTable(image=file) - session.add(my_table_model) - session.commit() - return my_table_model.image.to_dict() - - """ - - @property - def file_object_type(self) -> t.Type[FileObject]: - return FileObject - - impl = sa.JSON - - def process_bind_param( - self, value: t.Optional[t.Any], dialect: sa.Dialect - ) -> t.Any: - return self.process_bind_param_action(value, dialect) - - def process_result_value( - self, value: t.Optional[t.Any], dialect: sa.Dialect - ) -> t.Any: - return self.process_result_value_action(value, dialect) diff --git a/ellar_sql/model/typeDecorator/file/file_info.py b/ellar_sql/model/typeDecorator/file/file_info.py deleted file mode 100644 index 84f4422..0000000 --- a/ellar_sql/model/typeDecorator/file/file_info.py +++ /dev/null @@ -1,46 +0,0 @@ -import typing as t - -from ellar.core.files.storages import BaseStorage - -T = t.TypeVar("T", bound="FileObject") - - -class FileObject: - def __init__( - self, - *, - storage: BaseStorage, - original_filename: str, - uploaded_on: int, - content_type: str, - saved_filename: str, - extension: str, - file_size: int, - ) -> None: - self._storage = storage - self.original_filename = original_filename - self.uploaded_on = uploaded_on - self.content_type = content_type - self.filename = saved_filename - self.extension = extension - self.file_size = file_size - - def locate(self) -> str: - return self._storage.locate(self.filename) - - def open(self) -> t.IO: # type:ignore[type-arg] - return self._storage.open(self.filename) - - def to_dict(self) -> t.Dict[str, t.Any]: - return { - "original_filename": self.original_filename, - "uploaded_on": self.uploaded_on, - "content_type": self.content_type, - "extension": self.extension, - "file_size": self.file_size, - "saved_filename": self.filename, - "service_name": self._storage.service_name(), - } - - def __repr__(self) -> str: - return f"<{self.__class__.__name__} filename={self.filename}, content_type={self.content_type}, file_size={self.file_size}>" diff --git a/ellar_sql/model/typeDecorator/image/__init__.py b/ellar_sql/model/typeDecorator/image/__init__.py deleted file mode 100644 index 5b2d233..0000000 --- a/ellar_sql/model/typeDecorator/image/__init__.py +++ /dev/null @@ -1,9 +0,0 @@ -from .crop_info import CroppingDetails -from .field import ImageFileField -from .file_info import ImageFileObject - -__all__ = [ - "ImageFileObject", - "ImageFileField", - "CroppingDetails", -] diff --git a/ellar_sql/model/typeDecorator/image/crop_info.py b/ellar_sql/model/typeDecorator/image/crop_info.py deleted file mode 100644 index 8f1025b..0000000 --- a/ellar_sql/model/typeDecorator/image/crop_info.py +++ /dev/null @@ -1,9 +0,0 @@ -from dataclasses import dataclass - - -@dataclass -class CroppingDetails: - x: int - y: int - height: int - width: int diff --git a/ellar_sql/model/typeDecorator/image/field.py b/ellar_sql/model/typeDecorator/image/field.py deleted file mode 100644 index 6f70027..0000000 --- a/ellar_sql/model/typeDecorator/image/field.py +++ /dev/null @@ -1,146 +0,0 @@ -import typing as t -from io import SEEK_END, BytesIO - -import sqlalchemy as sa -from ellar.common import UploadFile -from ellar.core.files.storages import BaseStorage -from PIL import Image - -from ..exceptions import InvalidImageOperationError -from ..file import FileFieldBase -from .crop_info import CroppingDetails -from .file_info import ImageFileObject - - -class ImageFileField(FileFieldBase[ImageFileObject], sa.TypeDecorator): # type: ignore[type-arg] - """ - Provide SqlAlchemy TypeDecorator for Image files - ## Basic Usage - - class MyTable(Base): - image: ImageFileField.FileObject = sa.Column( - ImageFileField(storage=FileSystemStorage('path/to/save/files', max_size=10*MB), - nullable=True - ) - - def route(file: File[UploadFile]): - session = SessionLocal() - my_table_model = MyTable(image=file) - session.add(my_table_model) - session.commit() - return my_table_model.image.to_dict() - - ## Cropping - Image file also provides cropping capabilities which can be defined in the column or when saving the image data. - - fs = FileSystemStorage('path/to/save/files') - class MyTable(Base): - image = sa.Column(ImageFileField(storage=fs, crop=CroppingDetails(x=100, y=200, height=400, width=400)), nullable=True) - - OR - def route(file: File[UploadFile]): - session = SessionLocal() - my_table_model = MyTable( - image=(file, CroppingDetails(x=100, y=200, height=400, width=400)), - ) - - """ - - impl = sa.JSON - - def __init__( - self, - *args: t.Any, - storage: BaseStorage, - max_size: t.Optional[int] = None, - crop: t.Optional[CroppingDetails] = None, - **kwargs: t.Any, - ): - kwargs.setdefault("allowed_content_types", ["image/jpeg", "image/png"]) - super().__init__(*args, storage=storage, max_size=max_size, **kwargs) - self.crop = crop - - @property - def file_object_type(self) -> t.Type[ImageFileObject]: - return ImageFileObject - - def process_bind_param( - self, value: t.Optional[t.Any], dialect: sa.Dialect - ) -> t.Any: - return self.process_bind_param_action(value, dialect) - - def process_result_value( - self, value: t.Optional[t.Any], dialect: sa.Dialect - ) -> t.Any: - return self.process_result_value_action(value, dialect) - - def get_extra_file_initialization_context( - self, file: UploadFile - ) -> t.Dict[str, t.Any]: - try: - with Image.open(file.file) as image: - width, height = image.size - return {"width": width, "height": height} - except Exception: - return {"width": None, "height": None} - - def crop_image_with_box_sizing( - self, file: UploadFile, crop: t.Optional[CroppingDetails] = None - ) -> UploadFile: - crop_info = crop or self.crop - img = Image.open(file.file) - ( - height, - width, - x, - y, - ) = ( - crop_info.height, - crop_info.width, - crop_info.x, - crop_info.y, - ) - left = x - top = y - right = x + width - bottom = y + height - - crop_box = (left, top, right, bottom) - - img_res = img.crop(box=crop_box) - temp_thumb = BytesIO() - img_res.save(temp_thumb, img.format) - # Go to the end of the stream. - temp_thumb.seek(0, SEEK_END) - - # Get the current position, which is now at the end. - # We can use this as the size. - size = temp_thumb.tell() - temp_thumb.seek(0) - - content = UploadFile( - file=temp_thumb, filename=file.filename, size=size, headers=file.headers - ) - return content - - def process_bind_param_action( - self, value: t.Optional[t.Any], dialect: sa.Dialect - ) -> t.Optional[t.Union[str, t.Dict[str, t.Any]]]: - if isinstance(value, tuple): - file, crop_data = value - if not isinstance(file, UploadFile) or not isinstance( - crop_data, CroppingDetails - ): - raise InvalidImageOperationError( - "Invalid data was provided for ImageFileField. " - "Accept values: UploadFile or (UploadFile, CroppingDetails)" - ) - new_file = self.crop_image_with_box_sizing(file=file, crop=crop_data) - return super().process_bind_param_action(new_file, dialect) - - if isinstance(value, UploadFile): - if self.crop: - return super().process_bind_param_action( - self.crop_image_with_box_sizing(value), dialect - ) - return super().process_bind_param_action(value, dialect) diff --git a/ellar_sql/model/typeDecorator/image/file_info.py b/ellar_sql/model/typeDecorator/image/file_info.py deleted file mode 100644 index 09e7f93..0000000 --- a/ellar_sql/model/typeDecorator/image/file_info.py +++ /dev/null @@ -1,15 +0,0 @@ -import typing as t - -from ..file import FileObject - - -class ImageFileObject(FileObject): - def __init__(self, *, height: float, width: float, **kwargs: t.Any) -> None: - super().__init__(**kwargs) - self.height = height - self.width = width - - def to_dict(self) -> t.Dict[str, t.Any]: - data = super().to_dict() - data.update(height=self.height, width=self.width) - return data diff --git a/ellar_sql/model/typeDecorator/mimetypes.py b/ellar_sql/model/typeDecorator/mimetypes.py deleted file mode 100644 index f287a76..0000000 --- a/ellar_sql/model/typeDecorator/mimetypes.py +++ /dev/null @@ -1,21 +0,0 @@ -import mimetypes as mdb -import typing - -import magic - - -def magic_mime_from_buffer(buffer: bytes) -> str: - return magic.from_buffer(buffer, mime=True) - - -def guess_extension(mimetype: str) -> typing.Optional[str]: - """ - Due to the python bugs 'image/jpeg' overridden: - - https://bugs.python.org/issue4963 - - https://bugs.python.org/issue1043134 - - https://bugs.python.org/issue6626#msg91205 - """ - - if mimetype == "image/jpeg": - return ".jpeg" - return mdb.guess_extension(mimetype) diff --git a/ellar_sql/model/utils.py b/ellar_sql/model/utils.py index 23d47c4..dbb836c 100644 --- a/ellar_sql/model/utils.py +++ b/ellar_sql/model/utils.py @@ -1,41 +1,25 @@ import re -import typing as t -from io import BytesIO import sqlalchemy as sa import sqlalchemy.orm as sa_orm from ellar_sql.constant import DATABASE_BIND_KEY, DEFAULT_KEY, NAMING_CONVERSION -from .database_binds import get_database_bind, has_database_bind, update_database_binds +from .database_binds import ( + DatabaseMetadata, + get_metadata, + has_metadata, + update_database_metadata, +) -KB = 1024 -MB = 1024 * KB - -def copy_stream(source: t.IO, target: t.IO, *, chunk_size: int = 16 * KB) -> int: # type:ignore[type-arg] - length = 0 - while 1: - buf = source.read(chunk_size) - if not buf: - break - length += len(buf) - target.write(buf) - return length - - -def get_length(source: t.IO) -> int: # type:ignore[type-arg] - buffer = BytesIO() - return copy_stream(source, buffer) - - -def make_metadata(database_key: str) -> sa.MetaData: - if has_database_bind(database_key): - return get_database_bind(database_key, certain=True) +def make_metadata(database_key: str) -> DatabaseMetadata: + if has_metadata(database_key): + return get_metadata(database_key, certain=True) if database_key != DEFAULT_KEY: # Copy the naming convention from the default metadata. - naming_convention = make_metadata(DEFAULT_KEY).naming_convention + naming_convention = make_metadata(DEFAULT_KEY).metadata.naming_convention else: naming_convention = NAMING_CONVERSION @@ -43,8 +27,10 @@ def make_metadata(database_key: str) -> sa.MetaData: metadata = sa.MetaData( naming_convention=naming_convention, info={DATABASE_BIND_KEY: database_key} ) - update_database_binds(database_key, metadata) - return metadata + update_database_metadata( + database_key, metadata, registry=sa_orm.registry(metadata=metadata) + ) + return get_metadata(database_key, certain=True) def camel_to_snake_case(name: str) -> str: diff --git a/ellar_sql/module.py b/ellar_sql/module.py index 958ee66..7fc8445 100644 --- a/ellar_sql/module.py +++ b/ellar_sql/module.py @@ -2,11 +2,11 @@ import typing as t import sqlalchemy as sa -from ellar.app import current_injector -from ellar.common import IApplicationShutdown, IModuleSetup, Module +from ellar.common import IExecutionContext, IModuleSetup, Module, middleware from ellar.common.utils.importer import get_main_directory_by_stack from ellar.core import Config, DynamicModule, ModuleBase, ModuleSetup from ellar.di import ProviderConfig, request_or_transient_scope +from ellar.events import app_context_teardown_events from sqlalchemy.ext.asyncio import ( AsyncEngine, AsyncSession, @@ -19,12 +19,37 @@ from .schemas import MigrationOption, SQLAlchemyConfig +def _invalid_configuration(message: str) -> t.Callable: + def _raise_exception(): + raise RuntimeError(message) + + return _raise_exception + + @Module(commands=[DBCommands]) -class EllarSQLModule(ModuleBase, IModuleSetup, IApplicationShutdown): - async def on_shutdown(self) -> None: - db_service = current_injector.get(EllarSQLService) - res = db_service.session_factory.remove() +class EllarSQLModule(ModuleBase, IModuleSetup): + @middleware() + async def session_middleware( + cls, context: IExecutionContext, call_next: t.Callable[..., t.Coroutine] + ): + connection = context.switch_to_http_connection().get_client() + + db_session = connection.service_provider.get(EllarSQLService) + session = db_session.session_factory() + connection.state.session = session + + try: + await call_next() + except Exception as ex: + res = session.rollback() + if isinstance(res, t.Coroutine): + await res + raise ex + + @classmethod + async def _on_application_tear_down(cls, db_service: EllarSQLService) -> None: + res = db_service.session_factory.remove() if isinstance(res, t.Coroutine): await res @@ -83,6 +108,24 @@ def __setup_module(cls, sql_alchemy_config: SQLAlchemyConfig) -> DynamicModule: scope=request_or_transient_scope, ) ) + providers.append( + ProviderConfig( + Session, + use_value=_invalid_configuration( + f"{Session} is not configured based on your database options. Please use {AsyncSession}" + ), + scope=request_or_transient_scope, + ) + ) + providers.append( + ProviderConfig( + sa.Engine, + use_value=_invalid_configuration( + f"{sa.Engine} is not configured based on your database options. Please use {AsyncEngine}" + ), + scope=request_or_transient_scope, + ) + ) else: providers.append(ProviderConfig(sa.Engine, use_value=db_service.engine)) providers.append( @@ -92,8 +135,30 @@ def __setup_module(cls, sql_alchemy_config: SQLAlchemyConfig) -> DynamicModule: scope=request_or_transient_scope, ) ) + providers.append( + ProviderConfig( + AsyncSession, + use_value=_invalid_configuration( + f"{AsyncSession} is not configured based on your database options. Please use {Session}" + ), + scope=request_or_transient_scope, + ) + ) + providers.append( + ProviderConfig( + AsyncEngine, + use_value=_invalid_configuration( + f"{AsyncEngine} is not configured based on your database options. Please use {sa.Engine}" + ), + scope=request_or_transient_scope, + ) + ) providers.append(ProviderConfig(EllarSQLService, use_value=db_service)) + app_context_teardown_events.connect( + functools.partial(cls._on_application_tear_down, db_service=db_service) + ) + return DynamicModule( cls, providers=providers, @@ -122,10 +187,8 @@ def __register_setup_factory( root_path: str, override_config: t.Dict[str, t.Any], ) -> DynamicModule: - if config.get("SQLALCHEMY_CONFIG") and isinstance( - config.SQLALCHEMY_CONFIG, dict - ): - defined_config = dict(config.SQLALCHEMY_CONFIG) + if config.get("ELLAR_SQL") and isinstance(config.ELLAR_SQL, dict): + defined_config = dict(config.ELLAR_SQL) defined_config.update(override_config) defined_config.setdefault("root_path", root_path) @@ -135,9 +198,9 @@ def __register_setup_factory( schema.migration_options.directory = get_main_directory_by_stack( schema.migration_options.directory, - stack_level=2, + stack_level=0, from_dir=defined_config["root_path"], ) return module.__setup_module(schema) - raise RuntimeError("Could not find `SQLALCHEMY_CONFIG` in application config.") + raise RuntimeError("Could not find `ELLAR_SQL` in application config.") diff --git a/ellar_sql/pagination/decorator.py b/ellar_sql/pagination/decorator.py index 5f06621..fb06ab0 100644 --- a/ellar_sql/pagination/decorator.py +++ b/ellar_sql/pagination/decorator.py @@ -16,8 +16,8 @@ def paginate( pagination_class: t.Optional[t.Type[PaginationBase]] = None, - model: t.Optional[t.Type[ModelBase]] = None, - template_context: bool = False, + model: t.Optional[t.Union[t.Type[ModelBase], sa.sql.Select[t.Any]]] = None, + as_template_context: bool = False, item_schema: t.Optional[t.Type[BaseModel]] = None, **paginator_options: t.Any, ) -> t.Callable: @@ -26,7 +26,7 @@ def paginate( :param pagination_class: Pagination Class of type PaginationBase :param model: SQLAlchemy Model or SQLAlchemy Select Statement - :param template_context: If True adds `paginator` object to templating context data + :param as_template_context: If True adds `paginator` object to templating context data :param item_schema: Pagination Object Schema for serializing object and creating response schema documentation :param paginator_options: Other keyword args for initializing `pagination_class` :return: TCallable @@ -42,7 +42,7 @@ def _wraps(func: t.Callable) -> t.Callable: operation = operation_class( route_function=func, pagination_class=pagination_class or PageNumberPagination, - template_context=template_context, + as_template_context=as_template_context, item_schema=item_schema, paginator_options=paginator_options, ) @@ -58,12 +58,12 @@ def __init__( route_function: t.Callable, pagination_class: t.Type[PaginationBase], paginator_options: t.Dict[str, t.Any], - template_context: bool = False, + as_template_context: bool = False, item_schema: t.Optional[t.Type[BaseModel]] = None, ) -> None: self._original_route_function = route_function self._pagination_view = pagination_class(**paginator_options) - _, _, view = self._get_route_function_wrapper(template_context, item_schema) + _, _, view = self._get_route_function_wrapper(as_template_context, item_schema) self.as_view = functools.wraps(route_function)(view) def _prepare_template_response( @@ -96,7 +96,7 @@ def _prepare_template_response( return filter_query, extra_context def _get_route_function_wrapper( - self, template_context: bool, item_schema: t.Type[BaseModel] + self, as_template_context: bool, item_schema: t.Type[BaseModel] ) -> t.Tuple[ecm.params.ExtraEndpointArg, ecm.params.ExtraEndpointArg, t.Callable]: unique_id = str(uuid.uuid4()) # use unique_id to make the kwargs difficult to collide with any route function parameter @@ -114,12 +114,12 @@ def _get_route_function_wrapper( self._original_route_function ) - if not template_context and not item_schema: + if not as_template_context and not item_schema: raise ecm.exceptions.ImproperConfiguration( "Must supply value for either `template_context` or `item_schema`" ) - if not template_context: + if not as_template_context: # if pagination is not for template context, then we create a response schema for the api response response_schema = self._pagination_view.get_output_schema(item_schema) ecm.set_metadata(RESPONSE_OVERRIDE_KEY, {200: response_schema})( @@ -133,7 +133,7 @@ def as_view(*args: t.Any, **kw: t.Any) -> t.Any: items = self._original_route_function(*args, **func_kwargs) - if not template_context: + if not as_template_context: return self._pagination_view.api_paginate( items, paginate_input, @@ -156,10 +156,10 @@ def as_view(*args: t.Any, **kw: t.Any) -> t.Any: class _AsyncPaginationOperation(_PaginationOperation): def _get_route_function_wrapper( - self, template_context: bool, item_schema: t.Type[BaseModel] + self, as_template_context: bool, item_schema: t.Type[BaseModel] ) -> t.Tuple[ecm.params.ExtraEndpointArg, ecm.params.ExtraEndpointArg, t.Callable]: _paginate_args, execution_context, _ = super()._get_route_function_wrapper( - template_context, item_schema + as_template_context, item_schema ) async def as_view(*args: t.Any, **kw: t.Any) -> t.Any: @@ -169,12 +169,13 @@ async def as_view(*args: t.Any, **kw: t.Any) -> t.Any: context: ecm.IExecutionContext = execution_context.resolve(func_kwargs) items = await self._original_route_function(*args, **func_kwargs) + request = context.switch_to_http_connection().get_request() - if not template_context: + if not as_template_context: return self._pagination_view.api_paginate( items, paginate_input, - context.switch_to_http_connection().get_request(), + request, ) filter_query, extra_context = self._prepare_template_response(items) @@ -182,7 +183,7 @@ async def as_view(*args: t.Any, **kw: t.Any) -> t.Any: pagination_context = self._pagination_view.pagination_context( filter_query, paginate_input, - context.switch_to_http_connection().get_request(), + request, ) extra_context.update(pagination_context) diff --git a/ellar_sql/pagination/view.py b/ellar_sql/pagination/view.py index d718212..3dbf250 100644 --- a/ellar_sql/pagination/view.py +++ b/ellar_sql/pagination/view.py @@ -34,11 +34,13 @@ def validate_model( model: t.Union[t.Type[ModelBase], sa.sql.Select[t.Any]], fallback: t.Optional[t.Union[t.Type[ModelBase], sa.sql.Select[t.Any]]], ) -> t.Union[t.Type[ModelBase], sa.sql.Select[t.Any]]: - if isinstance(model, sa.sql.Select): + if isinstance(model, sa.sql.Select) or ( + isinstance(model, type) and issubclass(model, ModelBase) + ): working_model = model else: - working_model = model or fallback # type:ignore[assignment] - assert working_model, "Model Can not be None" + working_model = fallback # type:ignore[assignment] + assert working_model is not None, "Model Can not be None" return working_model @abstractmethod diff --git a/ellar_sql/query/__init__.py b/ellar_sql/query/__init__.py index 8c8238a..3b95f4d 100644 --- a/ellar_sql/query/__init__.py +++ b/ellar_sql/query/__init__.py @@ -1,17 +1,11 @@ from .utils import ( first_or_404, - first_or_404_async, get_or_404, - get_or_404_async, one_or_404, - one_or_404_async, ) __all__ = [ - "get_or_404_async", "get_or_404", - "one_or_404_async", "one_or_404", - "first_or_404_async", "first_or_404", ] diff --git a/ellar_sql/query/utils.py b/ellar_sql/query/utils.py index af09365..9f131d9 100644 --- a/ellar_sql/query/utils.py +++ b/ellar_sql/query/utils.py @@ -10,7 +10,7 @@ _O = t.TypeVar("_O", bound=object) -def get_or_404( +async def get_or_404( entity: t.Type[_O], ident: t.Any, *, @@ -19,28 +19,12 @@ def get_or_404( ) -> _O: """ """ db_service = current_injector.get(EllarSQLService) - session = db_service.session_factory() + session = db_service.get_scoped_session()() value = session.get(entity, ident, **kwargs) - if value is None: - raise ecm.NotFound(detail=error_message) - - return t.cast(_O, value) - - -async def get_or_404_async( - entity: t.Type[_O], - ident: t.Any, - *, - error_message: t.Optional[str] = None, - **kwargs: t.Any, -) -> _O: - """ """ - db_service = current_injector.get(EllarSQLService) - session = db_service.session_factory() - - value = await session.get(entity, ident, **kwargs) + if isinstance(value, t.Coroutine): + value = await value if value is None: raise ecm.NotFound(detail=error_message) @@ -48,30 +32,18 @@ async def get_or_404_async( return t.cast(_O, value) -def first_or_404( +async def first_or_404( statement: sa.sql.Select[t.Any], *, error_message: t.Optional[str] = None ) -> t.Any: """ """ db_service = current_injector.get(EllarSQLService) session = db_service.session_factory() - value = session.execute(statement).scalar() - - if value is None: - raise ecm.NotFound(detail=error_message) - - return value - - -async def first_or_404_async( - statement: sa.sql.Select[t.Any], *, error_message: t.Optional[str] = None -) -> t.Any: - """ """ - db_service = current_injector.get(EllarSQLService) - session = db_service.session_factory() + result = session.execute(statement) + if isinstance(result, t.Coroutine): + result = await result - res = await session.execute(statement) - value = res.scalar() + value = result.scalar() if value is None: raise ecm.NotFound(detail=error_message) @@ -79,7 +51,7 @@ async def first_or_404_async( return value -def one_or_404( +async def one_or_404( statement: sa.sql.Select[t.Any], *, error_message: t.Optional[str] = None ) -> t.Any: """ """ @@ -87,20 +59,11 @@ def one_or_404( session = db_service.session_factory() try: - return session.execute(statement).scalar_one() - except (sa_exc.NoResultFound, sa_exc.MultipleResultsFound) as ex: - raise ecm.NotFound(detail=error_message) from ex + result = session.execute(statement) + if isinstance(result, t.Coroutine): + result = await result -async def one_or_404_async( - statement: sa.sql.Select[t.Any], *, error_message: t.Optional[str] = None -) -> t.Any: - """ """ - db_service = current_injector.get(EllarSQLService) - session = db_service.session_factory() - - try: - res = await session.execute(statement) - return res.scalar_one() + return result.scalar_one() except (sa_exc.NoResultFound, sa_exc.MultipleResultsFound) as ex: raise ecm.NotFound(detail=error_message) from ex diff --git a/ellar_sql/schemas.py b/ellar_sql/schemas.py index 6770be1..cf6ac85 100644 --- a/ellar_sql/schemas.py +++ b/ellar_sql/schemas.py @@ -35,6 +35,7 @@ class PageNumberPaginationSchema(BaseModel, t.Generic[T]): @dataclass class ModelBaseConfig: + # Will be used when creating SQLAlchemy Model as a base or standalone use_bases: t.Sequence[ t.Union[ t.Type[t.Union[sa_orm.DeclarativeBase, sa_orm.DeclarativeBaseNoMeta]], @@ -42,8 +43,8 @@ class ModelBaseConfig: t.Any, ] ] = field(default_factory=lambda: []) - - make_declarative_base: bool = False + # indicates whether a class should be created a base for other models to inherit + as_base: bool = False _use: t.Optional[t.Type[t.Any]] = None # def get_use_type(self) -> t.Type[t.Any]: @@ -82,3 +83,20 @@ class SQLAlchemyConfig(ecm.Serializer): engine_options: t.Optional[t.Dict[str, t.Any]] = None models: t.Optional[t.List[str]] = None + + +@dataclass +class ModelMetaStore: + base_config: ModelBaseConfig + pk_column: t.Optional[sa_orm.ColumnProperty] = None + columns: t.List[sa_orm.ColumnProperty] = field(default_factory=lambda: []) + + def __post_init__(self) -> None: + if self.columns: + self.pk_column = next(c for c in self.columns if c.primary_key) + + @property + def pk_name(self) -> t.Optional[str]: + if self.pk_column is not None: + return self.pk_column.key + return None diff --git a/ellar_sql/services/base.py b/ellar_sql/services/base.py index 647ea4c..095e389 100644 --- a/ellar_sql/services/base.py +++ b/ellar_sql/services/base.py @@ -11,7 +11,7 @@ get_main_directory_by_stack, module_import, ) -from ellar.events import app_context_teardown_events +from ellar.threading import execute_coroutine_with_sync_worker from sqlalchemy.ext.asyncio import ( AsyncSession, async_scoped_session, @@ -24,7 +24,7 @@ from ellar_sql.model import ( make_metadata, ) -from ellar_sql.model.database_binds import get_database_bind, get_database_binds +from ellar_sql.model.database_binds import get_all_metadata, get_metadata from ellar_sql.schemas import MigrationOption from ellar_sql.session import ModelSession @@ -62,12 +62,6 @@ def __init__( self._setup(databases, models=models, echo=echo) self.session_factory = self.get_scoped_session() - app_context_teardown_events.connect(self._on_application_tear_down) - - async def _on_application_tear_down(self) -> None: - res = self.session_factory.remove() - if isinstance(res, t.Coroutine): - await res @property def has_async_engine_driver(self) -> bool: @@ -155,12 +149,10 @@ def create_all(self, *databases: str) -> None: metadata_engines = self._get_metadata_and_engine(_databases) - if self.has_async_engine_driver and _databases == "__all__": - raise Exception( - "You are using asynchronous database configuration. Use `create_all_async` instead" - ) - for metadata_engine in metadata_engines: + if metadata_engine.is_async(): + execute_coroutine_with_sync_worker(metadata_engine.create_all_async()) + continue metadata_engine.create_all() def drop_all(self, *databases: str) -> None: @@ -168,12 +160,10 @@ def drop_all(self, *databases: str) -> None: metadata_engines = self._get_metadata_and_engine(_databases) - if self.has_async_engine_driver and _databases == "__all__": - raise Exception( - "You are using asynchronous database configuration. Use `drop_all_async` instead" - ) - for metadata_engine in metadata_engines: + if metadata_engine.is_async(): + execute_coroutine_with_sync_worker(metadata_engine.drop_all_async()) + continue metadata_engine.drop_all() def reflect(self, *databases: str) -> None: @@ -181,45 +171,11 @@ def reflect(self, *databases: str) -> None: metadata_engines = self._get_metadata_and_engine(_databases) - if self.has_async_engine_driver and _databases == "__all__": - raise Exception( - "You are using asynchronous database configuration. Use `reflect_async` instead" - ) for metadata_engine in metadata_engines: - metadata_engine.reflect() - - async def create_all_async(self, *databases: str) -> None: - _databases = self.__validate_databases_input(*databases) - - metadata_engines = self._get_metadata_and_engine(_databases) - - for metadata_engine in metadata_engines: - if not metadata_engine.is_async(): - metadata_engine.create_all() - continue - await metadata_engine.create_all_async() - - async def drop_all_async(self, *databases: str) -> None: - _databases = self.__validate_databases_input(*databases) - - metadata_engines = self._get_metadata_and_engine(_databases) - - for metadata_engine in metadata_engines: - if not metadata_engine.is_async(): - metadata_engine.drop_all() + if metadata_engine.is_async(): + execute_coroutine_with_sync_worker(metadata_engine.reflect_async()) continue - await metadata_engine.drop_all_async() - - async def reflect_async(self, *databases: str) -> None: - _databases = self.__validate_databases_input(*databases) - - metadata_engines = self._get_metadata_and_engine(_databases) - - for metadata_engine in metadata_engines: - if not metadata_engine.is_async(): - metadata_engine.reflect() - continue - await metadata_engine.reflect_async() + metadata_engine.reflect() def get_scoped_session( self, @@ -313,7 +269,7 @@ def _get_metadata_and_engine( engines = self._engines[self] if database == "__all__": - keys: t.List[str] = list(get_database_binds()) + keys: t.List[str] = list(get_all_metadata()) elif isinstance(database, str): keys = [database] else: @@ -328,6 +284,6 @@ def _get_metadata_and_engine( message = f"Bind key '{key}' is not in 'Database' config." raise sa_exc.UnboundExecutionError(message) from None - metadata = get_database_bind(key, certain=True) - result.append(MetaDataEngine(metadata=metadata, engine=engine)) + db_metadata = get_metadata(key, certain=True) + result.append(MetaDataEngine(metadata=db_metadata.metadata, engine=engine)) return result diff --git a/ellar_sql/templates/basic/README b/ellar_sql/templates/multiple/README similarity index 100% rename from ellar_sql/templates/basic/README rename to ellar_sql/templates/multiple/README diff --git a/ellar_sql/templates/basic/alembic.ini.mako b/ellar_sql/templates/multiple/alembic.ini.mako similarity index 100% rename from ellar_sql/templates/basic/alembic.ini.mako rename to ellar_sql/templates/multiple/alembic.ini.mako diff --git a/ellar_sql/templates/basic/env.py b/ellar_sql/templates/multiple/env.py similarity index 67% rename from ellar_sql/templates/basic/env.py rename to ellar_sql/templates/multiple/env.py index 08314ed..00dc594 100644 --- a/ellar_sql/templates/basic/env.py +++ b/ellar_sql/templates/multiple/env.py @@ -1,14 +1,10 @@ -import typing as t from logging.config import fileConfig from alembic import context from ellar.app import current_injector from ellar.threading import execute_coroutine_with_sync_worker -from ellar_sql.migrations import ( - MultipleDatabaseAlembicEnvMigration, - SingleDatabaseAlembicEnvMigration, -) +from ellar_sql.migrations import MultipleDatabaseAlembicEnvMigration from ellar_sql.services import EllarSQLService # this is the Alembic Config object, which provides @@ -29,17 +25,8 @@ async def main() -> None: db_service: EllarSQLService = current_injector.get(EllarSQLService) - alembic_env_migration_klass: t.Type[ - t.Union[MultipleDatabaseAlembicEnvMigration, SingleDatabaseAlembicEnvMigration] - ] - - if len(db_service.engines) > 1: - alembic_env_migration_klass = MultipleDatabaseAlembicEnvMigration - else: - alembic_env_migration_klass = SingleDatabaseAlembicEnvMigration - # initialize migration class - alembic_env_migration = alembic_env_migration_klass(db_service) + alembic_env_migration = MultipleDatabaseAlembicEnvMigration(db_service) if context.is_offline_mode(): alembic_env_migration.run_migrations_offline(context) # type:ignore[arg-type] diff --git a/ellar_sql/templates/basic/script.py.mako b/ellar_sql/templates/multiple/script.py.mako similarity index 86% rename from ellar_sql/templates/basic/script.py.mako rename to ellar_sql/templates/multiple/script.py.mako index efe9aa2..e15eb70 100644 --- a/ellar_sql/templates/basic/script.py.mako +++ b/ellar_sql/templates/multiple/script.py.mako @@ -26,8 +26,6 @@ depends_on = ${repr(depends_on)} db_names = list(db_service.engines.keys()) %> -% if len(db_names) > 1: - def upgrade(engine_name): globals()["upgrade_%s" % engine_name]() @@ -47,14 +45,3 @@ def downgrade_${db_name}(): ${context.get("%s_downgrades" % db_name, "pass")} % endfor - -% else: - -def upgrade(): - ${upgrades if upgrades else "pass"} - - -def downgrade(): - ${downgrades if downgrades else "pass"} - -% endif diff --git a/ellar_sql/templates/single/README b/ellar_sql/templates/single/README new file mode 100644 index 0000000..6e399f7 --- /dev/null +++ b/ellar_sql/templates/single/README @@ -0,0 +1 @@ +Migration Setup for EllarSQL Models diff --git a/ellar_sql/templates/single/alembic.ini.mako b/ellar_sql/templates/single/alembic.ini.mako new file mode 100644 index 0000000..2cf6f90 --- /dev/null +++ b/ellar_sql/templates/single/alembic.ini.mako @@ -0,0 +1,50 @@ +# A generic, single database configuration. + +[alembic] +# template used to generate migration files +file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s + +# set to 'true' to run the environment during +# the 'revision' command, regardless of autogenerate +# revision_environment = false + + +# Logging configuration +[loggers] +keys = root,sqlalchemy,alembic,ellar_sqlalchemy_migrate + +[handlers] +keys = console + +[formatters] +keys = generic + +[logger_root] +level = WARN +handlers = console +qualname = + +[logger_sqlalchemy] +level = WARN +handlers = +qualname = sqlalchemy.engine + +[logger_alembic] +level = INFO +handlers = +qualname = alembic + +[logger_ellar_sqlalchemy_migrate] +level = INFO +handlers = +qualname = flask_migrate + +[handler_console] +class = StreamHandler +args = (sys.stderr,) +level = NOTSET +formatter = generic + +[formatter_generic] +format = %(levelname)-5.5s [%(name)s] %(message)s +datefmt = %H:%M:%S diff --git a/ellar_sql/templates/single/env.py b/ellar_sql/templates/single/env.py new file mode 100644 index 0000000..41c6f37 --- /dev/null +++ b/ellar_sql/templates/single/env.py @@ -0,0 +1,37 @@ +from logging.config import fileConfig + +from alembic import context +from ellar.app import current_injector +from ellar.threading import execute_coroutine_with_sync_worker + +from ellar_sql.migrations import SingleDatabaseAlembicEnvMigration +from ellar_sql.services import EllarSQLService + +# this is the Alembic Config object, which provides +# access to the values within the .ini file in use. +config = context.config + +# Interpret the config file for Python logging. +# This line sets up loggers basically. +fileConfig(config.config_file_name) # type:ignore[arg-type] + +# logger = logging.getLogger("alembic.env") +# other values from the config, defined by the needs of env.py, +# can be acquired: +# my_important_option = config.get_main_option("my_important_option") +# ... etc. + + +async def main() -> None: + db_service: EllarSQLService = current_injector.get(EllarSQLService) + + # initialize migration class + alembic_env_migration = SingleDatabaseAlembicEnvMigration(db_service) + + if context.is_offline_mode(): + alembic_env_migration.run_migrations_offline(context) # type:ignore[arg-type] + else: + await alembic_env_migration.run_migrations_online(context) # type:ignore[arg-type] + + +execute_coroutine_with_sync_worker(main()) diff --git a/ellar_sql/templates/single/script.py.mako b/ellar_sql/templates/single/script.py.mako new file mode 100644 index 0000000..e1b8246 --- /dev/null +++ b/ellar_sql/templates/single/script.py.mako @@ -0,0 +1,27 @@ +<%! +import re + +%>"""${message} + +Revision ID: ${up_revision} +Revises: ${down_revision | comma,n} +Create Date: ${create_date} + +""" +from alembic import op +import sqlalchemy as sa +${imports if imports else ""} + +# revision identifiers, used by Alembic. +revision = ${repr(up_revision)} +down_revision = ${repr(down_revision)} +branch_labels = ${repr(branch_labels)} +depends_on = ${repr(depends_on)} + + +def upgrade(): + ${upgrades if upgrades else "pass"} + + +def downgrade(): + ${downgrades if downgrades else "pass"} diff --git a/examples/db-learning/README.md b/examples/db-learning/README.md new file mode 100644 index 0000000..ee49c90 --- /dev/null +++ b/examples/db-learning/README.md @@ -0,0 +1,24 @@ +## Introduction +This is EllarSQL documentation `db_learning` project source code with all the illustrations + +## Project setup +``` +pip install -r requirements.txt +``` + +## Important Quick Steps +After environment setup, kindly follow instruction below + +- apply migration `python manage.py db upgrade` +- seed user data `python manage.py seed` + +## Development Server +``` +python manage.py runserver --reload +``` + + +## Run Test +``` +pytest +``` diff --git a/examples/db-learning/db_learning/__init__.py b/examples/db-learning/db_learning/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/examples/db-learning/db_learning/command.py b/examples/db-learning/db_learning/command.py new file mode 100644 index 0000000..adc7ae0 --- /dev/null +++ b/examples/db-learning/db_learning/command.py @@ -0,0 +1,18 @@ +import ellar_cli.click as click +from ellar.app import current_injector + +from db_learning.models import User +from ellar_sql import EllarSQLService + + +@click.command("seed") +@click.with_app_context +def seed_user(): + db_service = current_injector.get(EllarSQLService) + session = db_service.session_factory() + + for i in range(300): + session.add(User(username=f"username-{i+1}", email=f"user{i+1}doe@example.com")) + + session.commit() + db_service.session_factory.remove() diff --git a/examples/db-learning/db_learning/config.py b/examples/db-learning/db_learning/config.py new file mode 100644 index 0000000..716ab46 --- /dev/null +++ b/examples/db-learning/db_learning/config.py @@ -0,0 +1,93 @@ +""" +Application Configurations +Default Ellar Configurations are exposed here through `ConfigDefaultTypesMixin` +Make changes and define your own configurations specific to your application + +export ELLAR_CONFIG_MODULE=db_learning.config:DevelopmentConfig +""" + +import typing as t + +from ellar.common import IExceptionHandler, JSONResponse +from ellar.core import ConfigDefaultTypesMixin +from ellar.core.versioning import BaseAPIVersioning, DefaultAPIVersioning +from ellar.pydantic import ENCODERS_BY_TYPE as encoders_by_type +from starlette.middleware import Middleware + + +class BaseConfig(ConfigDefaultTypesMixin): + DEBUG: bool = False + + DEFAULT_JSON_CLASS: t.Type[JSONResponse] = JSONResponse + SECRET_KEY: str = "ellar_QdZwHTfLkZQWQtAot-V6gbTHONMn3ekrl5jdcb5AOC8" + + # injector auto_bind = True allows you to resolve types that are not registered on the container + # For more info, read: https://injector.readthedocs.io/en/latest/index.html + INJECTOR_AUTO_BIND = False + + # jinja Environment options + # https://jinja.palletsprojects.com/en/3.0.x/api/#high-level-api + JINJA_TEMPLATES_OPTIONS: t.Dict[str, t.Any] = {} + + # Application route versioning scheme + VERSIONING_SCHEME: BaseAPIVersioning = DefaultAPIVersioning() + + # Enable or Disable Application Router route searching by appending backslash + REDIRECT_SLASHES: bool = False + + # Define references to static folders in python packages. + # eg STATIC_FOLDER_PACKAGES = [('boostrap4', 'statics')] + STATIC_FOLDER_PACKAGES: t.Optional[t.List[t.Union[str, t.Tuple[str, str]]]] = [] + + # Define references to static folders defined within the project + STATIC_DIRECTORIES: t.Optional[t.List[t.Union[str, t.Any]]] = [] + + # static route path + STATIC_MOUNT_PATH: str = "/static" + + CORS_ALLOW_ORIGINS: t.List[str] = ["*"] + CORS_ALLOW_METHODS: t.List[str] = ["*"] + CORS_ALLOW_HEADERS: t.List[str] = ["*"] + ALLOWED_HOSTS: t.List[str] = ["*"] + + # Application middlewares + MIDDLEWARE: t.Sequence[Middleware] = [] + + # A dictionary mapping either integer status codes, + # or exception class types onto callables which handle the exceptions. + # Exception handler callables should be of the form + # `handler(context:IExecutionContext, exc: Exception) -> response` + # and may be either standard functions, or async functions. + EXCEPTION_HANDLERS: t.List[IExceptionHandler] = [] + + # Object Serializer custom encoders + SERIALIZER_CUSTOM_ENCODER: t.Dict[ + t.Any, t.Callable[[t.Any], t.Any] + ] = encoders_by_type + + +class DevelopmentConfig(BaseConfig): + DEBUG: bool = True + + ELLAR_SQL: t.Dict[str, t.Any] = { + "databases": { + "default": "sqlite:///app.db", + }, + "echo": True, + "migration_options": { + "directory": "migrations" # root directory will be determined based on where the module is instantiated. + }, + "models": ["db_learning.models"], + } + + +class TestConfig(BaseConfig): + DEBUG = False + + ELLAR_SQL: t.Dict[str, t.Any] = { + **DevelopmentConfig.ELLAR_SQL, + "databases": { + "default": "sqlite:///test.db", + }, + "echo": False, + } diff --git a/examples/db-learning/db_learning/controller.py b/examples/db-learning/db_learning/controller.py new file mode 100644 index 0000000..142a0c8 --- /dev/null +++ b/examples/db-learning/db_learning/controller.py @@ -0,0 +1,41 @@ +import ellar.common as ecm +from ellar.pydantic import EmailStr + +from ellar_sql import get_or_404, model + +from .models import User + + +@ecm.Controller +class UsersController(ecm.ControllerBase): + @ecm.post("/") + def create_user( + self, + username: ecm.Body[str], + email: ecm.Body[EmailStr], + session: ecm.Inject[model.Session], + ): + user = User(username=username, email=email) + + session.add(user) + session.commit() + session.refresh(user) + + return user.dict() + + @ecm.get("/{user_id:int}") + async def user_by_id(self, user_id: int): + user: User = await get_or_404(User, user_id) + return user.dict() + + @ecm.get("/") + async def user_list(self, session: ecm.Inject[model.Session]): + stmt = model.select(User) + rows = session.execute(stmt.offset(0).limit(100)).scalars() + return [row.dict() for row in rows] + + @ecm.get("/{user_id:int}") + async def user_delete(self, user_id: int, session: ecm.Inject[model.Session]): + user = get_or_404(User, user_id) + session.delete(user) + return {"detail": f"User id={user_id} Deleted successfully"} diff --git a/examples/db-learning/db_learning/core/__init__.py b/examples/db-learning/db_learning/core/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/examples/db-learning/db_learning/domain/__init__.py b/examples/db-learning/db_learning/domain/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/examples/db-learning/db_learning/migrations/README b/examples/db-learning/db_learning/migrations/README new file mode 100644 index 0000000..6e399f7 --- /dev/null +++ b/examples/db-learning/db_learning/migrations/README @@ -0,0 +1 @@ +Migration Setup for EllarSQL Models diff --git a/examples/db-learning/db_learning/migrations/alembic.ini b/examples/db-learning/db_learning/migrations/alembic.ini new file mode 100644 index 0000000..2cf6f90 --- /dev/null +++ b/examples/db-learning/db_learning/migrations/alembic.ini @@ -0,0 +1,50 @@ +# A generic, single database configuration. + +[alembic] +# template used to generate migration files +file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s + +# set to 'true' to run the environment during +# the 'revision' command, regardless of autogenerate +# revision_environment = false + + +# Logging configuration +[loggers] +keys = root,sqlalchemy,alembic,ellar_sqlalchemy_migrate + +[handlers] +keys = console + +[formatters] +keys = generic + +[logger_root] +level = WARN +handlers = console +qualname = + +[logger_sqlalchemy] +level = WARN +handlers = +qualname = sqlalchemy.engine + +[logger_alembic] +level = INFO +handlers = +qualname = alembic + +[logger_ellar_sqlalchemy_migrate] +level = INFO +handlers = +qualname = flask_migrate + +[handler_console] +class = StreamHandler +args = (sys.stderr,) +level = NOTSET +formatter = generic + +[formatter_generic] +format = %(levelname)-5.5s [%(name)s] %(message)s +datefmt = %H:%M:%S diff --git a/examples/db-learning/db_learning/migrations/env.py b/examples/db-learning/db_learning/migrations/env.py new file mode 100644 index 0000000..41c6f37 --- /dev/null +++ b/examples/db-learning/db_learning/migrations/env.py @@ -0,0 +1,37 @@ +from logging.config import fileConfig + +from alembic import context +from ellar.app import current_injector +from ellar.threading import execute_coroutine_with_sync_worker + +from ellar_sql.migrations import SingleDatabaseAlembicEnvMigration +from ellar_sql.services import EllarSQLService + +# this is the Alembic Config object, which provides +# access to the values within the .ini file in use. +config = context.config + +# Interpret the config file for Python logging. +# This line sets up loggers basically. +fileConfig(config.config_file_name) # type:ignore[arg-type] + +# logger = logging.getLogger("alembic.env") +# other values from the config, defined by the needs of env.py, +# can be acquired: +# my_important_option = config.get_main_option("my_important_option") +# ... etc. + + +async def main() -> None: + db_service: EllarSQLService = current_injector.get(EllarSQLService) + + # initialize migration class + alembic_env_migration = SingleDatabaseAlembicEnvMigration(db_service) + + if context.is_offline_mode(): + alembic_env_migration.run_migrations_offline(context) # type:ignore[arg-type] + else: + await alembic_env_migration.run_migrations_online(context) # type:ignore[arg-type] + + +execute_coroutine_with_sync_worker(main()) diff --git a/examples/db-learning/db_learning/migrations/script.py.mako b/examples/db-learning/db_learning/migrations/script.py.mako new file mode 100644 index 0000000..e1b8246 --- /dev/null +++ b/examples/db-learning/db_learning/migrations/script.py.mako @@ -0,0 +1,27 @@ +<%! +import re + +%>"""${message} + +Revision ID: ${up_revision} +Revises: ${down_revision | comma,n} +Create Date: ${create_date} + +""" +from alembic import op +import sqlalchemy as sa +${imports if imports else ""} + +# revision identifiers, used by Alembic. +revision = ${repr(up_revision)} +down_revision = ${repr(down_revision)} +branch_labels = ${repr(branch_labels)} +depends_on = ${repr(depends_on)} + + +def upgrade(): + ${upgrades if upgrades else "pass"} + + +def downgrade(): + ${downgrades if downgrades else "pass"} diff --git a/examples/db-learning/db_learning/migrations/versions/2024_01_27_1031-aa924ee1b88a_initial_migration.py b/examples/db-learning/db_learning/migrations/versions/2024_01_27_1031-aa924ee1b88a_initial_migration.py new file mode 100644 index 0000000..5da3b82 --- /dev/null +++ b/examples/db-learning/db_learning/migrations/versions/2024_01_27_1031-aa924ee1b88a_initial_migration.py @@ -0,0 +1,34 @@ +"""initial migration + +Revision ID: aa924ee1b88a +Revises: +Create Date: 2024-01-27 10:31:22.187308 + +""" +import sqlalchemy as sa +from alembic import op + +# revision identifiers, used by Alembic. +revision = "aa924ee1b88a" +down_revision = None +branch_labels = None +depends_on = None + + +def upgrade(): + # ### commands auto generated by Alembic - please adjust! ### + op.create_table( + "user", + sa.Column("id", sa.Integer(), nullable=False), + sa.Column("username", sa.String(), nullable=False), + sa.Column("email", sa.String(), nullable=False), + sa.PrimaryKeyConstraint("id", name=op.f("pk_user")), + sa.UniqueConstraint("username", name=op.f("uq_user_username")), + ) + # ### end Alembic commands ### + + +def downgrade(): + # ### commands auto generated by Alembic - please adjust! ### + op.drop_table("user") + # ### end Alembic commands ### diff --git a/examples/db-learning/db_learning/models.py b/examples/db-learning/db_learning/models.py new file mode 100644 index 0000000..3763d3b --- /dev/null +++ b/examples/db-learning/db_learning/models.py @@ -0,0 +1,9 @@ +from ellar_sql import model + + +class User(model.Model): + id: model.Mapped[int] = model.mapped_column(model.Integer, primary_key=True) + username: model.Mapped[str] = model.mapped_column( + model.String, unique=True, nullable=False + ) + email: model.Mapped[str] = model.mapped_column(model.String) diff --git a/examples/db-learning/db_learning/pagination/__init__.py b/examples/db-learning/db_learning/pagination/__init__.py new file mode 100644 index 0000000..23316bc --- /dev/null +++ b/examples/db-learning/db_learning/pagination/__init__.py @@ -0,0 +1,7 @@ +from .api import list_api_router +from .template import list_template_router + +__all__ = [ + "list_api_router", + "list_template_router", +] diff --git a/examples/db-learning/db_learning/pagination/api.py b/examples/db-learning/db_learning/pagination/api.py new file mode 100644 index 0000000..da707ce --- /dev/null +++ b/examples/db-learning/db_learning/pagination/api.py @@ -0,0 +1,32 @@ +import ellar.common as ec +from ellar.openapi import ApiTags + +from db_learning.models import User +from ellar_sql import paginate + + +class UserSchema(ec.Serializer): + id: int + username: str + email: str + + +# @ec.get('/users') +# @paginate(item_schema=UserSchema, per_page=100) +# def list_users(): +# return User + +list_api_router = ec.ModuleRouter("/users-api") + + +@list_api_router.get() +@paginate(model=User, item_schema=UserSchema) +def list_users_api(): + pass + + +# openapi tag +ApiTags( + name="API Pagination", + external_doc_url="https://python-ellar.github.io/ellar-sql/pagination/#api-pagination", +)(list_api_router.get_control_type()) diff --git a/examples/db-learning/db_learning/pagination/template.py b/examples/db-learning/db_learning/pagination/template.py new file mode 100644 index 0000000..b629e52 --- /dev/null +++ b/examples/db-learning/db_learning/pagination/template.py @@ -0,0 +1,30 @@ +import ellar.common as ec +from ellar.openapi import ApiTags + +from db_learning.models import User +from ellar_sql import model, paginate + +list_template_router = ec.ModuleRouter("/users-template") + + +## CASE 1 +# @list_template_router.get('/users') +# @ec.render('list.html') +# @paginate(as_template_context=True) +# def list_users(): +# return model.select(User), {'name': 'Template Pagination'} # pagination model, template context + + +## CASE 2 +@list_template_router.get() +@ec.render("list.html") +@paginate(model=model.select(User), as_template_context=True) +def list_users_template(): + return {"name": "Template Pagination"} + + +# openapi tag +ApiTags( + name="Template Pagination", + external_doc_url="https://python-ellar.github.io/ellar-sql/pagination/#template-pagination", +)(list_template_router.get_control_type()) diff --git a/examples/db-learning/db_learning/root_module.py b/examples/db-learning/db_learning/root_module.py new file mode 100644 index 0000000..8c55cf1 --- /dev/null +++ b/examples/db-learning/db_learning/root_module.py @@ -0,0 +1,26 @@ +import ellar.common as ec +from ellar.app import App +from ellar.core import ModuleBase + +from db_learning.command import seed_user +from db_learning.controller import UsersController +from db_learning.pagination import list_api_router, list_template_router +from ellar_sql import EllarSQLModule, EllarSQLService + + +@ec.Module( + modules=[EllarSQLModule.register_setup()], + routers=[list_template_router, list_api_router], + controllers=[UsersController], + commands=[seed_user], +) +class ApplicationModule(ModuleBase, ec.IApplicationStartup): + async def on_startup(self, app: App) -> None: + db_service = app.injector.get(EllarSQLService) + db_service.create_all() + + @ec.exception_handler(404) + def exception_404_handler( + cls, ctx: ec.IExecutionContext, exc: Exception + ) -> ec.Response: + return ec.JSONResponse({"detail": "Resource not found."}, status_code=404) diff --git a/examples/db-learning/db_learning/server.py b/examples/db-learning/db_learning/server.py new file mode 100644 index 0000000..b0c2653 --- /dev/null +++ b/examples/db-learning/db_learning/server.py @@ -0,0 +1,32 @@ +import os + +from ellar.app import App, AppFactory +from ellar.common.constants import ELLAR_CONFIG_MODULE +from ellar.core import LazyModuleImport as lazyLoad +from ellar.openapi import OpenAPIDocumentBuilder, OpenAPIDocumentModule, SwaggerUI + + +def bootstrap() -> App: + application = AppFactory.create_from_app_module( + lazyLoad("db_learning.root_module:ApplicationModule"), + config_module=os.environ.get( + ELLAR_CONFIG_MODULE, "db_learning.config:DevelopmentConfig" + ), + global_guards=[], + ) + + # uncomment this section if you want API documentation + + document_builder = OpenAPIDocumentBuilder() + document_builder.set_title("Db_learning Title").set_version("1.0.2").set_contact( + name="Author Name", + url="https://www.author-name.com", + email="authorname@gmail.com", + ).set_license("MIT Licence", url="https://www.google.com") + + document = document_builder.build_document(application) + module = OpenAPIDocumentModule.setup( + document=document, docs_ui=SwaggerUI(), guards=[] + ) + application.install_module(module) + return application diff --git a/examples/db-learning/db_learning/templates/list.html b/examples/db-learning/db_learning/templates/list.html new file mode 100644 index 0000000..d252e5a --- /dev/null +++ b/examples/db-learning/db_learning/templates/list.html @@ -0,0 +1,29 @@ + + +

{{ name }}

+{% macro render_pagination(paginator, endpoint) %} +
+ {{ paginator.first }} - {{ paginator.last }} of {{ paginator.total }} +
+
+ {% for page in paginator.iter_pages() %} + {% if page %} + {% if page != paginator.page %} + {{ page }} + {% else %} + {{ page }} + {% endif %} + {% else %} + + {% endif %} + {% endfor %} +
+{% endmacro %} + +
    + {% for user in paginator %} +
  • {{ user.id }} - email:{{ user.email }} username: {{ user.username }} + {% endfor %} +
+{{render_pagination(paginator=paginator, endpoint="list_users_template") }} + diff --git a/examples/db-learning/manage.py b/examples/db-learning/manage.py new file mode 100644 index 0000000..7ac4bc1 --- /dev/null +++ b/examples/db-learning/manage.py @@ -0,0 +1,12 @@ +import os + +from ellar.common.constants import ELLAR_CONFIG_MODULE +from ellar_cli.main import create_ellar_cli + +if __name__ == "__main__": + os.environ.setdefault(ELLAR_CONFIG_MODULE, "db_learning.config:DevelopmentConfig") + + # initialize Commandline program + cli = create_ellar_cli("db_learning.server:bootstrap") + # start commandline execution + cli(prog_name="Ellar Web Framework CLI") diff --git a/examples/db-learning/requirements.txt b/examples/db-learning/requirements.txt new file mode 100644 index 0000000..c0a63cf --- /dev/null +++ b/examples/db-learning/requirements.txt @@ -0,0 +1,7 @@ +anyio[trio]>=3.2.1 +ellar-cli +ellar-sql +factory-boy==3.3.0 +pytest-asyncio +pytest-cov>=2.12.0 +pytest>=7.1.3 diff --git a/examples/db-learning/tests/__init__.py b/examples/db-learning/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/examples/db-learning/tests/common.py b/examples/db-learning/tests/common.py new file mode 100644 index 0000000..e4a5010 --- /dev/null +++ b/examples/db-learning/tests/common.py @@ -0,0 +1,3 @@ +from sqlalchemy import orm + +Session = orm.scoped_session(orm.sessionmaker()) diff --git a/examples/db-learning/tests/conftest.py b/examples/db-learning/tests/conftest.py new file mode 100644 index 0000000..1794c62 --- /dev/null +++ b/examples/db-learning/tests/conftest.py @@ -0,0 +1,48 @@ +import os + +import pytest +import sqlalchemy as sa +from db_learning.root_module import ApplicationModule +from ellar.common.constants import ELLAR_CONFIG_MODULE +from ellar.testing import Test + +from ellar_sql import EllarSQLService + +from . import common + +os.environ.setdefault(ELLAR_CONFIG_MODULE, "db_learning.config:TestConfig") + + +@pytest.fixture(scope="session") +def tm(): + test_module = Test.create_test_module(modules=[ApplicationModule]) + yield test_module + + +@pytest.fixture(scope="session") +def db(tm): + db_service = tm.get(EllarSQLService) + db_service.create_all() + + yield + + db_service.drop_all() + + +@pytest.fixture(scope="session") +def db_session(db, tm): + db_service = tm.get(EllarSQLService) + + yield db_service.session_factory() + + db_service.session_factory.remove() + + +@pytest.fixture +def factory_session(db, tm): + engine = tm.get(sa.Engine) + common.Session.configure(bind=engine) + + yield + + common.Session.remove() diff --git a/examples/db-learning/tests/factories.py b/examples/db-learning/tests/factories.py new file mode 100644 index 0000000..28b0324 --- /dev/null +++ b/examples/db-learning/tests/factories.py @@ -0,0 +1,16 @@ +import factory +from db_learning.models import User + +from ellar_sql.factory import SESSION_PERSISTENCE_FLUSH, EllarSQLFactory + +from . import common + + +class UserFactory(EllarSQLFactory): + class Meta: + model = User + sqlalchemy_session_persistence = SESSION_PERSISTENCE_FLUSH + sqlalchemy_session_factory = common.Session + + username = factory.Faker("user_name") + email = factory.Faker("email") diff --git a/examples/db-learning/tests/test_user_model.py b/examples/db-learning/tests/test_user_model.py new file mode 100644 index 0000000..84d6254 --- /dev/null +++ b/examples/db-learning/tests/test_user_model.py @@ -0,0 +1,26 @@ +import pytest +import sqlalchemy.exc as sa_exc + +from .factories import UserFactory + +# from db_learning.models import User +# +# def test_username_must_be_unique(db_session): +# # Creating and adding the first user +# user1 = User(username='ellarSQL', email='ellarsql@gmail.com') +# db_session.add(user1) +# db_session.commit() +# +# # Attempting to add a second user with the same username +# user2 = User(username='ellarSQL', email='ellarsql2@gmail.com') +# db_session.add(user2) +# +# # Expecting an IntegrityError due to unique constraint violation +# with pytest.raises(sa_exc.IntegrityError): +# db_session.commit() + + +def test_username_must_be_unique(factory_session): + user1 = UserFactory() + with pytest.raises(sa_exc.IntegrityError): + UserFactory(username=user1.username) diff --git a/examples/index-script/main.py b/examples/index-script/main.py new file mode 100644 index 0000000..c22f66b --- /dev/null +++ b/examples/index-script/main.py @@ -0,0 +1,35 @@ +from ellar_sql import EllarSQLService, model + + +class User(model.Model): + id: model.Mapped[int] = model.mapped_column(model.Integer, primary_key=True) + username: model.Mapped[str] = model.mapped_column( + model.String, unique=True, nullable=False + ) + email: model.Mapped[str] = model.mapped_column(model.String) + + +def main(): + db_service = EllarSQLService( + databases="sqlite:///app.db", + echo=True, + ) + + db_service.create_all() + + session = db_service.session_factory() + + for i in range(50): + session.add(User(username=f"username-{i+1}", email=f"user{i+1}doe@example.com")) + + session.commit() + rows = session.execute(model.select(User)).scalars() + + all_users = [row.dict() for row in rows] + assert len(all_users) == 50 + + session.close() + + +if __name__ == "__main__": + main() diff --git a/examples/single-db/db/controllers.py b/examples/single-db/db/controllers.py index 3c6b853..c109883 100644 --- a/examples/single-db/db/controllers.py +++ b/examples/single-db/db/controllers.py @@ -9,7 +9,7 @@ def index(self): return {'detail': "Welcome Dog's Resources"} """ -from ellar.common import Controller, ControllerBase, get, post, Body +from ellar.common import Body, Controller, ControllerBase, get, post from pydantic import EmailStr from sqlalchemy import select @@ -18,7 +18,6 @@ def index(self): @Controller class DbController(ControllerBase): - @get("/") def index(self): return {"detail": "Welcome Db Resource"} @@ -46,4 +45,3 @@ async def get_all_users(self): stmt = select(User) rows = session.execute(stmt.offset(0).limit(100)).scalars() return [row.dict() for row in rows] - diff --git a/examples/single-db/db/migrations/versions/2024_01_01_1016-cce418606c45_.py b/examples/single-db/db/migrations/versions/2024_01_01_1016-cce418606c45_.py index 1c44e60..2c7151c 100644 --- a/examples/single-db/db/migrations/versions/2024_01_01_1016-cce418606c45_.py +++ b/examples/single-db/db/migrations/versions/2024_01_01_1016-cce418606c45_.py @@ -1,39 +1,36 @@ """empty message Revision ID: cce418606c45 -Revises: +Revises: Create Date: 2024-01-01 10:16:49.511421 """ -from alembic import op import sqlalchemy as sa - +from alembic import op # revision identifiers, used by Alembic. -revision = 'cce418606c45' +revision = "cce418606c45" down_revision = None branch_labels = None depends_on = None - - def upgrade(): # ### commands auto generated by Alembic - please adjust! ### - op.create_table('user', - sa.Column('id', sa.Integer(), nullable=False), - sa.Column('username', sa.String(), nullable=False), - sa.Column('email', sa.String(), nullable=False), - sa.Column('created_date', sa.DateTime(), nullable=False), - sa.Column('time_updated', sa.DateTime(), nullable=False), - sa.PrimaryKeyConstraint('id', name=op.f('pk_user')), - sa.UniqueConstraint('username', name=op.f('uq_user_username')) + op.create_table( + "user", + sa.Column("id", sa.Integer(), nullable=False), + sa.Column("username", sa.String(), nullable=False), + sa.Column("email", sa.String(), nullable=False), + sa.Column("created_date", sa.DateTime(), nullable=False), + sa.Column("time_updated", sa.DateTime(), nullable=False), + sa.PrimaryKeyConstraint("id", name=op.f("pk_user")), + sa.UniqueConstraint("username", name=op.f("uq_user_username")), ) # ### end Alembic commands ### def downgrade(): # ### commands auto generated by Alembic - please adjust! ### - op.drop_table('user') + op.drop_table("user") # ### end Alembic commands ### - diff --git a/examples/single-db/db/models/__init__.py b/examples/single-db/db/models/__init__.py index 67c9337..0eceadc 100644 --- a/examples/single-db/db/models/__init__.py +++ b/examples/single-db/db/models/__init__.py @@ -1,5 +1,5 @@ from .users import User __all__ = [ - 'User', + "User", ] diff --git a/examples/single-db/db/models/base.py b/examples/single-db/db/models/base.py index fe93910..2314499 100644 --- a/examples/single-db/db/models/base.py +++ b/examples/single-db/db/models/base.py @@ -1,5 +1,6 @@ from datetime import datetime -from sqlalchemy import DateTime, func, MetaData + +from sqlalchemy import DateTime, MetaData, func from sqlalchemy.orm import Mapped, mapped_column from ellar_sql.model import Model @@ -14,15 +15,19 @@ class Base(Model): - __base_config__ = {'make_declarative_base': True} - __database__ = 'default' + __base_config__ = {"as_base": True} + __database__ = "default" - metadata = MetaData(naming_convention=convention) + metadata = MetaData(naming_convention=convention) - created_date: Mapped[datetime] = mapped_column( - "created_date", DateTime, default=datetime.utcnow, nullable=False - ) + created_date: Mapped[datetime] = mapped_column( + "created_date", DateTime, default=datetime.utcnow, nullable=False + ) - time_updated: Mapped[datetime] = mapped_column( - "time_updated", DateTime, nullable=False, default=datetime.utcnow, onupdate=func.now() - ) + time_updated: Mapped[datetime] = mapped_column( + "time_updated", + DateTime, + nullable=False, + default=datetime.utcnow, + onupdate=func.now(), + ) diff --git a/examples/single-db/db/models/users.py b/examples/single-db/db/models/users.py index 489a36d..b96551b 100644 --- a/examples/single-db/db/models/users.py +++ b/examples/single-db/db/models/users.py @@ -1,18 +1,13 @@ - from sqlalchemy import Integer, String from sqlalchemy.orm import Mapped, mapped_column + from .base import Base + class User(Base): id: Mapped[int] = mapped_column(Integer, primary_key=True) username: Mapped[str] = mapped_column(String, unique=True, nullable=False) email: Mapped[str] = mapped_column(String) - -assert getattr(User, '__dnd__', None) == 'Ellar' - # assert session - - - diff --git a/examples/single-db/db/module.py b/examples/single-db/db/module.py index e88bf79..0dfb75a 100644 --- a/examples/single-db/db/module.py +++ b/examples/single-db/db/module.py @@ -18,9 +18,10 @@ def register_providers(self, container: Container) -> None: """ from ellar.app import App -from ellar.common import Module, IApplicationStartup +from ellar.common import IApplicationStartup, Module from ellar.core import ModuleBase from ellar.di import Container + from ellar_sql import EllarSQLModule, EllarSQLService from .controllers import DbController @@ -30,9 +31,7 @@ def register_providers(self, container: Container) -> None: controllers=[DbController], providers=[], routers=[], - modules=[ - EllarSQLModule.register_setup() - ] + modules=[EllarSQLModule.register_setup()], ) class DbModule(ModuleBase, IApplicationStartup): """ @@ -41,7 +40,7 @@ class DbModule(ModuleBase, IApplicationStartup): async def on_startup(self, app: App) -> None: db_service = app.injector.get(EllarSQLService) - # db_service.create_all() + db_service.create_all() def register_providers(self, container: Container) -> None: - """for more complicated provider registrations, use container.register_instance(...) """ + """for more complicated provider registrations, use container.register_instance(...)""" diff --git a/examples/single-db/single_db/config.py b/examples/single-db/single_db/config.py index a15169e..95ea24d 100644 --- a/examples/single-db/single_db/config.py +++ b/examples/single-db/single_db/config.py @@ -8,11 +8,11 @@ import typing as t -from ellar.pydantic import ENCODERS_BY_TYPE as encoders_by_type -from starlette.middleware import Middleware from ellar.common import IExceptionHandler, JSONResponse from ellar.core import ConfigDefaultTypesMixin from ellar.core.versioning import BaseAPIVersioning, DefaultAPIVersioning +from ellar.pydantic import ENCODERS_BY_TYPE as encoders_by_type +from starlette.middleware import Middleware class BaseConfig(ConfigDefaultTypesMixin): @@ -56,7 +56,7 @@ class BaseConfig(ConfigDefaultTypesMixin): # A dictionary mapping either integer status codes, # or exception class types onto callables which handle the exceptions. # Exception handler callables should be of the form - # `handler(context:IExecutionContext, exc: Exception) -> response` + # `handler(context:IExecutionContext, exc: Exception) -> response` # and may be either standard functions, or async functions. EXCEPTION_HANDLERS: t.List[IExceptionHandler] = [] @@ -69,14 +69,14 @@ class BaseConfig(ConfigDefaultTypesMixin): class DevelopmentConfig(BaseConfig): DEBUG: bool = True # Configuration through Confog - SQLALCHEMY_CONFIG: t.Dict[str, t.Any] = { - 'databases': { - 'default': 'sqlite:///project.db', + ELLAR_SQL: t.Dict[str, t.Any] = { + "databases": { + "default": "sqlite:///project.db", # 'db2': 'sqlite+aiosqlite:///project2.db', }, - 'echo': True, - 'migration_options': { - 'directory': 'migrations' # root directory will be determined based on where the module is instantiated. + "echo": True, + "migration_options": { + "directory": "migrations" # root directory will be determined based on where the module is instantiated. }, - 'models': ['db.models'] - } \ No newline at end of file + "models": ["db.models"], + } diff --git a/examples/single-db/single_db/root_module.py b/examples/single-db/single_db/root_module.py index 710e4a2..99841bd 100644 --- a/examples/single-db/single_db/root_module.py +++ b/examples/single-db/single_db/root_module.py @@ -1,10 +1,17 @@ -from ellar.common import Module, exception_handler, IExecutionContext, JSONResponse, Response -from ellar.core import ModuleBase, LazyModuleImport as lazyLoad +from ellar.common import ( + IExecutionContext, + JSONResponse, + Module, + Response, + exception_handler, +) +from ellar.core import LazyModuleImport as lazyLoad +from ellar.core import ModuleBase from ellar.samples.modules import HomeModule -@Module(modules=[HomeModule, lazyLoad('db.module:DbModule')]) +@Module(modules=[HomeModule, lazyLoad("db.module:DbModule")]) class ApplicationModule(ModuleBase): @exception_handler(404) def exception_404_handler(cls, ctx: IExecutionContext, exc: Exception) -> Response: - return JSONResponse(dict(detail="Resource not found."), status_code=404) \ No newline at end of file + return JSONResponse({"detail": "Resource not found."}, status_code=404) diff --git a/examples/single-db/single_db/server.py b/examples/single-db/single_db/server.py index c0b5dd0..822708d 100644 --- a/examples/single-db/single_db/server.py +++ b/examples/single-db/single_db/server.py @@ -3,7 +3,7 @@ from ellar.app import AppFactory from ellar.common.constants import ELLAR_CONFIG_MODULE from ellar.core import LazyModuleImport as lazyLoad -from ellar.openapi import OpenAPIDocumentModule, OpenAPIDocumentBuilder, SwaggerUI +from ellar.openapi import OpenAPIDocumentBuilder, OpenAPIDocumentModule, SwaggerUI def bootstrap(): @@ -12,21 +12,22 @@ def bootstrap(): config_module=os.environ.get( ELLAR_CONFIG_MODULE, "single_db.config:DevelopmentConfig" ), - global_guards=[] + global_guards=[], ) document_builder = OpenAPIDocumentBuilder() - document_builder.set_title('Ellar SQLAlchemy Single DB Example') \ - .set_version('1.0.2') \ - .set_contact(name='Author Name', url='https://www.author-name.com', email='authorname@gmail.com') \ - .set_license('MIT Licence', url='https://www.google.com') + document_builder.set_title("Ellar SQLAlchemy Single DB Example").set_version( + "1.0.2" + ).set_contact( + name="Author Name", + url="https://www.author-name.com", + email="authorname@gmail.com", + ).set_license("MIT Licence", url="https://www.google.com") document = document_builder.build_document(application) application.install_module( OpenAPIDocumentModule.setup( - document=document, - docs_ui=SwaggerUI(dark_theme=True), - guards=[] + document=document, docs_ui=SwaggerUI(dark_theme=True), guards=[] ) ) return application diff --git a/examples/single-db/tests/conftest.py b/examples/single-db/tests/conftest.py index b18f954..e69de29 100644 --- a/examples/single-db/tests/conftest.py +++ b/examples/single-db/tests/conftest.py @@ -1 +0,0 @@ -from ellar.testing import Test, TestClient \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..17105ac --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,103 @@ +site_name: '' +site_description: Ellar-SQLAlchemy, designed for Ellar, enhances your application by integrating support for SQLAlchemy and Alembic. +site_url: https://github.com/python-ellar/ellar-sql +repo_name: python-ellar/ellar-sql +repo_url: https://github.com/python-ellar/ellar-sql +edit_uri: blob/master/docs +copyright: | + Copyright © 2024 Eadwin Ezeudoh + +docs_dir: docs +site_dir: site + +theme: + name: material + features: + - announce.dismiss + - content.action.edit + - content.action.view + - content.code.annotate + - content.code.copy + - content.tooltips + - search.highlight + - search.share + - search.suggest + - toc.follow + - navigation.footer + - navigation.indexes + - navigation.instant + - navigation.prune +# - navigation.tabs + - navigation.left + - navigation.tracking + palette: + - media: "(prefers-color-scheme: light)" + scheme: default + primary: custom + accent: cyan + toggle: + icon: material/lightbulb + name: Switch to dark mode + + - media: "(prefers-color-scheme: dark)" + scheme: slate + primary: custom + accent: cyan + toggle: + icon: material/lightbulb-outline + name: Switch to light mode + font: + text: Source Sans Pro + code: Fira Code + language: en + logo: img/EllarLogoB.png + favicon: img/Icon.svg + icon: + repo: fontawesome/brands/git-alt + +plugins: + - search: + separator: '[\s\-,:!=\[\]()"`/]+|\.(?!\d)|&[lg]t;|(?!\b)(?=[A-Z][a-z])' + - minify: + minify_html: true + - git-revision-date-localized: + enable_creation_date: false + +nav: + - Index: index.md + - Get Started: models/index.md + - Configuration: models/configuration.md + - Models: + - index: models/models.md + - Extra Fields: models/extra-fields.md + - Pagination: pagination/index.md + - Multiple Database: multiple/index.md + - Migrations: + - index: migrations/index.md + - customizing env: migrations/env.md + - Testing: + - index: testing/index.md + +markdown_extensions: + - attr_list + - toc: + permalink: true + - admonition + - def_list + - tables + - abbr + - footnotes + - md_in_html + - codehilite + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + - pymdownx.details + - pymdownx.tabbed: + alternate_style: true + - pymdownx.saneheaders + - pymdownx.tilde + +extra_css: + - stylesheets/extra.css diff --git a/pyproject.toml b/pyproject.toml index 386ca8b..84ca615 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -34,9 +34,7 @@ classifiers = [ dependencies = [ "ellar >= 0.6.7", "sqlalchemy >= 2.0.23", - "alembic >= 1.10.0", - "python-magic >= 0.4.27", - "pillow >= 10.1.0" + "alembic >= 1.10.0" ] dev = [ @@ -69,8 +67,8 @@ test = [ "ruff ==0.1.9", "mypy == 1.8.0", "autoflake", - "types-Pillow", "ellar-cli >= 0.3.3", + "factory-boy >= 3.3.0" ] [tool.ruff] @@ -102,6 +100,7 @@ pretty = true strict_optional = true disable_error_code = ["name-defined", 'union-attr'] disallow_subclassing_any = false +ignore_missing_imports = true [[tool.mypy.overrides]] module = "ellar_sql.cli.commands" ignore_errors = true diff --git a/tests/model_samples.py b/tests/model_samples.py index 09f42df..5e1d608 100644 --- a/tests/model_samples.py +++ b/tests/model_samples.py @@ -3,7 +3,7 @@ class Base(model.Model): - __base_config__ = ModelBaseConfig(make_declarative_base=True) + __base_config__ = ModelBaseConfig(as_base=True) metadata = model.MetaData( naming_convention={ diff --git a/tests/test_migrations/samples/models.py b/tests/test_migrations/samples/models.py index 06b6e8c..7bed05f 100644 --- a/tests/test_migrations/samples/models.py +++ b/tests/test_migrations/samples/models.py @@ -4,7 +4,7 @@ class Base(model.Model): - __base_config__ = {"make_declarative_base": True} + __base_config__ = {"as_base": True} class User(Base): diff --git a/tests/test_migrations/test_multiple_database.py b/tests/test_migrations/test_multiple_database.py index 1cbea4b..dc5fb7f 100644 --- a/tests/test_migrations/test_multiple_database.py +++ b/tests/test_migrations/test_multiple_database.py @@ -4,7 +4,7 @@ @clean_directory("multiple") def test_migrate_upgrade_for_multiple_database(): with set_env_variable("multiple_db", "true"): - result = run_command("multiple_database.py db init") + result = run_command("multiple_database.py db init -m") assert result.returncode == 0 assert ( b"tests/dumbs/multiple/migrations/alembic.ini' before proceeding." @@ -38,7 +38,7 @@ def test_migrate_upgrade_for_multiple_database(): @clean_directory("multiple") def test_migrate_upgrade_multiple_database_with_model_changes(): with set_env_variable("multiple_db", "true"): - result = run_command("multiple_database.py db init") + result = run_command("multiple_database.py db init -m") assert result.returncode == 0 result = run_command("multiple_database.py db migrate") @@ -59,7 +59,7 @@ def test_migrate_upgrade_multiple_database_with_model_changes(): @clean_directory("multiple_async") def test_migrate_upgrade_for_multiple_database_async(): with set_env_variable("multiple_db", "true"): - result = run_command("multiple_database_async.py db init") + result = run_command("multiple_database_async.py db init -m") assert result.returncode == 0 assert ( b"tests/dumbs/multiple_async/migrations/alembic.ini' before proceeding." diff --git a/tests/test_model.py b/tests/test_model.py index 0d14886..ea6afd1 100644 --- a/tests/test_model.py +++ b/tests/test_model.py @@ -31,7 +31,7 @@ class T2(Base): def test_create_model_with_another_base(): class AnotherBaseWithSameMetadata(model.Model): - __base_config__ = ModelBaseConfig(make_declarative_base=True) + __base_config__ = ModelBaseConfig(as_base=True) name = model.Column(model.String(128)) class T3(AnotherBaseWithSameMetadata): @@ -59,7 +59,7 @@ class T4(AnotherBaseWithSameMetadata): def test_metadata(): class Base2(model.Model): - __base_config__ = ModelBaseConfig(make_declarative_base=True) + __base_config__ = ModelBaseConfig(as_base=True) metadata = model.MetaData() __database__ = "db2" name = model.Column(model.String(128)) @@ -68,7 +68,7 @@ class Base2(model.Model): class Base3(model.Model): __base_config__ = ModelBaseConfig( - use_bases=[model.DeclarativeBase], make_declarative_base=True + use_bases=[model.DeclarativeBase], as_base=True ) __database__ = "db2" name = model.Column(model.String(128)) @@ -146,7 +146,7 @@ def test_abstractmodel_case_1(db_service, base_super_class_as_dataclass): bases, _ = base_super_class_as_dataclass class Base(model.Model): - __base_config__ = ModelBaseConfig(use_bases=bases, make_declarative_base=True) + __base_config__ = ModelBaseConfig(use_bases=bases, as_base=True) class TimestampModel(Base): __abstract__ = True @@ -184,7 +184,7 @@ def test_abstractmodel_case_2(db_service, base_super_class): # @model.as_base() class Base(model.Model): - __base_config__ = ModelBaseConfig(use_bases=bases, make_declarative_base=True) + __base_config__ = ModelBaseConfig(use_bases=bases, as_base=True) class TimestampModel(Base): # type: ignore[no-redef] __abstract__ = True @@ -245,7 +245,7 @@ class TimestampMixin: # type: ignore[no-redef] ) class Base(TimestampMixin, model.Model): - __base_config__ = ModelBaseConfig(make_declarative_base=True) + __base_config__ = ModelBaseConfig(as_base=True) class Post(Base): # type: ignore[no-redef] id: model.Mapped[int] = model.mapped_column(model.Integer, primary_key=True) @@ -268,7 +268,7 @@ def test_mixin_model_case_2(db_service, base_super_class_as_dataclass) -> None: bases, meta = base_super_class_as_dataclass class Base(model.Model): - __base_config__ = ModelBaseConfig(use_bases=bases, make_declarative_base=True) + __base_config__ = ModelBaseConfig(use_bases=bases, as_base=True) class TimestampMixin(model.MappedAsDataclass): created: model.Mapped[datetime] = model.mapped_column( diff --git a/tests/test_model_database_key.py b/tests/test_model_database_key.py index a884481..1b107a0 100644 --- a/tests/test_model_database_key.py +++ b/tests/test_model_database_key.py @@ -1,12 +1,13 @@ from ellar_sql import model -from ellar_sql.model.database_binds import get_database_bind +from ellar_sql.model.database_binds import get_metadata def test_bind_key_default(ignore_base): class User(model.Model): id = model.Column(model.Integer, primary_key=True) - assert User.metadata is get_database_bind("default", certain=True) + assert User.metadata is get_metadata("default", certain=True).metadata + assert User.registry is get_metadata("default", certain=True).registry def test_metadata_per_bind(ignore_base): @@ -14,7 +15,7 @@ class User(model.Model): __database__ = "other" id = model.Column(model.Integer, primary_key=True) - assert User.metadata is get_database_bind("other", certain=True) + assert User.metadata is get_metadata("other", certain=True).metadata def test_multiple_binds_same_table_name(ignore_base): @@ -27,8 +28,9 @@ class UserB(model.Model): __tablename__ = "user" id = model.Column(model.Integer, primary_key=True) - assert UserA.metadata is get_database_bind("default", certain=True) - assert UserB.metadata is get_database_bind("other", certain=True) + assert UserA.metadata is get_metadata("default", certain=True).metadata + assert UserB.metadata is get_metadata("other", certain=True).metadata + assert UserB.registry is get_metadata("other", certain=True).registry assert UserA.__table__.metadata is not UserB.__table__.metadata @@ -43,7 +45,7 @@ class Admin(User): id = model.Column(model.Integer, model.ForeignKey(User.id), primary_key=True) __mapper_args__ = {"polymorphic_identity": "admin"} - assert "admin" in get_database_bind("auth", certain=True).tables + assert "admin" in get_metadata("auth", certain=True).metadata.tables # inherits metadata, doesn't set it directly assert "metadata" not in Admin.__dict__ @@ -56,7 +58,7 @@ class AbstractUser(model.Model): class User(AbstractUser): id = model.Column(model.Integer, primary_key=True) - assert "user" in get_database_bind("auth", certain=True).tables + assert "user" in get_metadata("auth", certain=True).metadata.tables assert "metadata" not in User.__dict__ @@ -69,7 +71,7 @@ class User(model.Model): id = model.Column(model.Integer, primary_key=True) assert User.__table__.metadata is other_metadata - assert get_database_bind("other") is None + assert get_metadata("other") is None def test_explicit_table(ignore_base): @@ -83,5 +85,5 @@ class User(model.Model): __database__ = "other" __table__ = user_table - assert User.__table__.metadata is get_database_bind("auth") - assert get_database_bind("other") is None + assert User.__table__.metadata is get_metadata("auth").metadata + assert get_metadata("other") is None diff --git a/tests/test_model_export.py b/tests/test_model_export.py new file mode 100644 index 0000000..4042cf5 --- /dev/null +++ b/tests/test_model_export.py @@ -0,0 +1,148 @@ +import factory +import pytest +from factory.alchemy import SESSION_PERSISTENCE_COMMIT + +from ellar_sql import model +from ellar_sql.factory.base import EllarSQLFactory + + +def get_model_factory(db_service): + class User(model.Model): + id: model.Mapped[int] = model.Column(model.Integer, primary_key=True) + name: model.Mapped[str] = model.Column(model.String) + email: model.Mapped[str] = model.Column(model.String) + address: model.Mapped[str] = model.Column(model.String, nullable=True) + city: model.Mapped[str] = model.Column(model.String, nullable=True) + + class UserFactory(EllarSQLFactory): + class Meta: + model = User + sqlalchemy_session_factory = db_service.session_factory + sqlalchemy_session_persistence = SESSION_PERSISTENCE_COMMIT + + name = factory.Faker("name") + email = factory.Faker("email") + city = factory.Faker("city") + + return UserFactory + + +class TestModelExport: + def test_model_export_without_filter(self, db_service, ignore_base): + user_factory = get_model_factory(db_service) + + db_service.create_all() + + user = user_factory( + name="Ellar", email="ellar@support.com", city="Andersonchester" + ) + assert user.dict() == { + "address": None, + "city": "Andersonchester", + "email": "ellar@support.com", + "id": 1, + "name": "Ellar", + } + db_service.session_factory.close() + + def test_model_exclude_none(self, db_service, ignore_base): + user_factory = get_model_factory(db_service) + + db_service.create_all() + + user = user_factory( + name="Ellar", email="ellar@support.com", city="Andersonchester" + ) + assert user.dict(exclude_none=True) == { + "city": "Andersonchester", + "email": "ellar@support.com", + "id": 1, + "name": "Ellar", + } + db_service.session_factory.close() + + def test_model_export_include(self, db_service, ignore_base): + user_factory = get_model_factory(db_service) + + db_service.create_all() + + user = user_factory() + + assert user.dict(include={"email", "id", "name"}).keys() == { + "email", + "id", + "name", + } + db_service.session_factory.close() + + def test_model_export_exclude(self, db_service, ignore_base): + user_factory = get_model_factory(db_service) + + db_service.create_all() + + user = user_factory() + + assert user.dict(exclude={"email", "name"}).keys() == {"address", "city", "id"} + db_service.session_factory.close() + + +@pytest.mark.asyncio +class TestModelExportAsync: + async def test_model_export_without_filter_async( + self, db_service_async, ignore_base + ): + user_factory = get_model_factory(db_service_async) + + db_service_async.create_all() + + user = user_factory( + name="Ellar", email="ellar@support.com", city="Andersonchester" + ) + assert user.dict() == { + "address": None, + "city": "Andersonchester", + "email": "ellar@support.com", + "id": 1, + "name": "Ellar", + } + db_service_async.session_factory.close() + + async def test_model_exclude_none_async(self, db_service_async, ignore_base): + user_factory = get_model_factory(db_service_async) + + db_service_async.create_all() + + user = user_factory( + name="Ellar", email="ellar@support.com", city="Andersonchester" + ) + assert user.dict(exclude_none=True) == { + "city": "Andersonchester", + "email": "ellar@support.com", + "id": 1, + "name": "Ellar", + } + db_service_async.session_factory.close() + + async def test_model_export_include_async(self, db_service_async, ignore_base): + user_factory = get_model_factory(db_service_async) + + db_service_async.create_all() + + user = user_factory() + + assert user.dict(include={"email", "id", "name"}).keys() == { + "email", + "id", + "name", + } + db_service_async.session_factory.close() + + async def test_model_export_exclude_async(self, db_service_async, ignore_base): + user_factory = get_model_factory(db_service_async) + + db_service_async.create_all() + + user = user_factory() + + assert user.dict(exclude={"email", "name"}).keys() == {"address", "city", "id"} + db_service_async.session_factory.close() diff --git a/tests/test_model_factory.py b/tests/test_model_factory.py new file mode 100644 index 0000000..b8bbc07 --- /dev/null +++ b/tests/test_model_factory.py @@ -0,0 +1,299 @@ +import types +import typing + +import factory +import pytest +from factory import FactoryError +from factory.alchemy import SESSION_PERSISTENCE_COMMIT, SESSION_PERSISTENCE_FLUSH +from sqlalchemy.exc import IntegrityError +from sqlalchemy.orm import relationship + +from ellar_sql import model +from ellar_sql.factory.base import EllarSQLFactory + + +def create_group_model(**kwargs): + class User(model.Model): + id: model.Mapped[int] = model.Column(model.Integer, primary_key=True) + name: model.Mapped[str] = model.Column(model.String) + email: model.Mapped[str] = model.Column(model.String) + address: model.Mapped[str] = model.Column(model.String, nullable=True) + city: model.Mapped[str] = model.Column(model.String, nullable=True) + groups: model.Mapped[typing.List["Group"]] = relationship( + "Group", back_populates="user" + ) + + class Group(model.Model): + id: model.Mapped[int] = model.Column(model.Integer, primary_key=True) + name: model.Mapped[str] = model.Column(model.String) + user_id: model.Mapped[int] = model.Column( + model.ForeignKey("user.id"), unique=True + ) + + user: model.Mapped[User] = relationship( + "User", back_populates="groups", uselist=False + ) + + user_config = kwargs.pop("user", {}) + user_meta = types.new_class( + "UserMeta", (), {}, lambda ns: ns.update(model=User, **user_config) + ) + + class UserFactory(EllarSQLFactory): + class Meta(user_meta): + pass + + name = factory.Faker("name") + email = factory.Faker("email") + city = factory.Faker("city") + + group_config = kwargs.pop("group", {}) + group_meta = types.new_class( + "GroupMeta", (), {}, lambda ns: ns.update(model=Group, **group_config) + ) + + class GroupFactory(EllarSQLFactory): + class Meta(group_meta): + pass + + name = factory.Faker("name") + user = factory.SubFactory(UserFactory) + + return GroupFactory + + +class TestModelFactory: + def test_model_factory(self, db_service, ignore_base): + group_factory = create_group_model( + user={ + "sqlalchemy_session_factory": db_service.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_COMMIT, + }, + group={ + "sqlalchemy_session_factory": db_service.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_COMMIT, + }, + ) + + db_service.create_all() + + group = group_factory() + assert group.dict().keys() == {"name", "user_id", "id"} + db_service.session_factory.close() + + def test_model_factory_session_none(self, db_service, ignore_base): + group_factory = create_group_model( + user={ + "sqlalchemy_session_factory": db_service.session_factory, + }, + group={ + "sqlalchemy_session_factory": db_service.session_factory, + }, + ) + + db_service.create_all() + + group = group_factory() + assert f"" == repr(group) + assert group.dict().keys() != {"name", "user_id", "id"} + db_service.session_factory.close() + + def test_model_factory_session_flush(self, db_service, ignore_base): + group_factory = create_group_model( + user={ + "sqlalchemy_session_factory": db_service.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_FLUSH, + }, + group={ + "sqlalchemy_session_factory": db_service.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_FLUSH, + }, + ) + + db_service.create_all() + + group = group_factory() + assert group.dict().keys() == {"name", "user_id", "id"} + db_service.session_factory.close() + + def test_model_factory_get_or_create(self, db_service, ignore_base): + group_factory = create_group_model( + user={ + "sqlalchemy_session_factory": db_service.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_COMMIT, + }, + group={ + "sqlalchemy_session_factory": db_service.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_COMMIT, + "sqlalchemy_get_or_create": ("name",), + }, + ) + db_service.create_all() + + group = group_factory() + + group2 = group_factory(name=group.name) + + group3 = group_factory(name="new group") + + assert group.user_id == group2.user_id != group3.user_id + assert group.id == group2.id + + assert group.dict() == group2.dict() != group3.dict() + + db_service.session_factory.close() + + def test_model_factory_get_or_create_for_integrity_error( + self, db_service, ignore_base + ): + group_factory = create_group_model( + user={ + "sqlalchemy_session_factory": db_service.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_COMMIT, + }, + group={ + "sqlalchemy_session_factory": db_service.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_COMMIT, + "sqlalchemy_get_or_create": ("name",), + }, + ) + db_service.create_all() + + group = group_factory() + + with pytest.raises(IntegrityError): + group_factory(name="new group", user=group.user) + + db_service.session_factory.close() + + +@pytest.mark.asyncio +class TestModelFactoryAsync: + async def test_model_factory_async(self, db_service_async, ignore_base): + group_factory = create_group_model( + user={ + "sqlalchemy_session_factory": db_service_async.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_COMMIT, + }, + group={ + "sqlalchemy_session_factory": db_service_async.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_COMMIT, + }, + ) + + db_service_async.create_all() + + group = group_factory() + assert group.dict().keys() == {"name", "user_id", "id"} + await db_service_async.session_factory.close() + + async def test_model_factory_session_none_async( + self, db_service_async, ignore_base + ): + group_factory = create_group_model( + user={ + "sqlalchemy_session_factory": db_service_async.session_factory, + }, + group={ + "sqlalchemy_session_factory": db_service_async.session_factory, + }, + ) + + db_service_async.create_all() + + group = group_factory() + assert f"" == repr(group) + assert group.dict().keys() != {"name", "user_id", "id"} + await db_service_async.session_factory.close() + + async def test_model_factory_session_flush_async( + self, db_service_async, ignore_base + ): + group_factory = create_group_model( + user={ + "sqlalchemy_session_factory": db_service_async.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_FLUSH, + }, + group={ + "sqlalchemy_session_factory": db_service_async.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_FLUSH, + }, + ) + + db_service_async.create_all() + + group = group_factory() + assert group.dict().keys() == {"name", "user_id", "id"} + await db_service_async.session_factory.close() + + async def test_model_factory_get_or_create_async( + self, db_service_async, ignore_base + ): + group_factory = create_group_model( + user={ + "sqlalchemy_session_factory": db_service_async.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_FLUSH, + }, + group={ + "sqlalchemy_session_factory": db_service_async.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_FLUSH, + "sqlalchemy_get_or_create": ("name",), + }, + ) + db_service_async.create_all() + + group = group_factory() + + group2 = group_factory(name=group.name) + + group3 = group_factory(name="new group") + + assert group.user_id == group2.user_id != group3.user_id + assert group.id == group2.id + + assert group.dict() == group2.dict() != group3.dict() + + db_service_async.session_factory.close() + + async def test_model_factory_get_or_create_for_integrity_error_async( + self, db_service_async, ignore_base + ): + group_factory = create_group_model( + user={ + "sqlalchemy_session_factory": db_service_async.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_FLUSH, + }, + group={ + "sqlalchemy_session_factory": db_service_async.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_FLUSH, + "sqlalchemy_get_or_create": ("name",), + }, + ) + db_service_async.create_all() + + group = group_factory() + + with pytest.raises(IntegrityError): + group_factory(name="new group", user=group.user) + + db_service_async.session_factory.close() + + async def test_model_factory_get_or_create_raises_error_for_missing_field_async( + self, db_service_async, ignore_base + ): + group_factory = create_group_model( + user={ + "sqlalchemy_session_factory": db_service_async.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_FLUSH, + }, + group={ + "sqlalchemy_session_factory": db_service_async.session_factory, + "sqlalchemy_session_persistence": SESSION_PERSISTENCE_FLUSH, + "sqlalchemy_get_or_create": ("name", "user_id"), + }, + ) + db_service_async.create_all() + with pytest.raises(FactoryError): + group_factory() + + await db_service_async.session_factory.close() diff --git a/tests/test_module.py b/tests/test_module.py index e69de29..1a3d4cf 100644 --- a/tests/test_module.py +++ b/tests/test_module.py @@ -0,0 +1,35 @@ +import pytest +from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession + +from ellar_sql import model + + +async def test_invalid_session_engine_ioc_resolve_for_sync_setup( + app_setup, anyio_backend +): + app = app_setup() + + with pytest.raises(RuntimeError) as ex: + app.injector.get(AsyncEngine) + + assert "" in str(ex.value) + + with pytest.raises(RuntimeError) as ex1: + app.injector.get(AsyncSession) + + assert "" in str(ex1.value) + + +async def test_invalid_session_engine_ioc_resolve_for_async_setup( + app_setup_async, anyio_backend +): + app = app_setup_async() + + with pytest.raises(RuntimeError) as ex: + app.injector.get(model.Session) + assert "" in str(ex.value) + + with pytest.raises(RuntimeError) as ex1: + app.injector.get(model.Engine) + + assert "" in str(ex1.value) diff --git a/tests/test_pagination/seed.py b/tests/test_pagination/seed.py index 24f3501..115400c 100644 --- a/tests/test_pagination/seed.py +++ b/tests/test_pagination/seed.py @@ -20,10 +20,7 @@ def seed_100_users(app: App): session = db_service.session_factory() - if session.get_bind().dialect.is_async: - execute_coroutine_with_sync_worker(db_service.create_all_async()) - else: - db_service.create_all() + db_service.create_all() for i in range(100): session.add(user_model(name=f"User Number {i+1}")) diff --git a/tests/test_pagination/test_pagination_view.py b/tests/test_pagination/test_pagination_view.py index 549f24e..836a5fb 100644 --- a/tests/test_pagination/test_pagination_view.py +++ b/tests/test_pagination/test_pagination_view.py @@ -24,7 +24,7 @@ class UserSerializer(ecm.Serializer): def _get_route_test_route( user_model, pagination_class, case_2=False, case_3=False, invalid=False, **kw ): - kwargs = dict(kw, template_context=True, pagination_class=pagination_class) + kwargs = dict(kw, as_template_context=True, pagination_class=pagination_class) if case_2: kwargs.update(model=user_model) diff --git a/tests/test_pagination/test_pagination_view_async.py b/tests/test_pagination/test_pagination_view_async.py index 6e6d787..ab1ebbb 100644 --- a/tests/test_pagination/test_pagination_view_async.py +++ b/tests/test_pagination/test_pagination_view_async.py @@ -24,7 +24,7 @@ class UserSerializer(ecm.Serializer): def _get_route_test_route( user_model, pagination_class, case_2=False, case_3=False, invalid=False, **kw ): - kwargs = dict(kw, template_context=True, pagination_class=pagination_class) + kwargs = dict(kw, as_template_context=True, pagination_class=pagination_class) if case_2: kwargs.update(model=user_model) diff --git a/tests/test_query/test_utils.py b/tests/test_query/test_utils.py index 318d614..db65a20 100644 --- a/tests/test_query/test_utils.py +++ b/tests/test_query/test_utils.py @@ -8,12 +8,9 @@ from ellar_sql import ( EllarSQLService, first_or_404, - first_or_404_async, get_or_404, - get_or_404_async, model, one_or_404, - one_or_404_async, ) @@ -31,10 +28,7 @@ def _seed_model(app: App): session = db_service.session_factory() - if session.get_bind().dialect.is_async: - execute_coroutine_with_sync_worker(db_service.create_all_async()) - else: - db_service.create_all() + db_service.create_all() session.add(user_model(name="First User")) res = session.commit() @@ -48,65 +42,67 @@ def _seed_model(app: App): async def test_get_or_404_works(ignore_base, app_ctx, anyio_backend): user_model = _seed_model(app_ctx) - user_instance = get_or_404(user_model, 1) + user_instance = await get_or_404(user_model, 1) assert user_instance.name == "First User" with pytest.raises(NotFound): - get_or_404(user_model, 2) + await get_or_404(user_model, 2) async def test_get_or_404_async_works(ignore_base, app_ctx_async, anyio_backend): if anyio_backend == "asyncio": user_model = _seed_model(app_ctx_async) - user_instance = await get_or_404_async(user_model, 1) + user_instance = await get_or_404(user_model, 1) assert user_instance.name == "First User" with pytest.raises(NotFound): - await get_or_404_async(user_model, 2) + await get_or_404(user_model, 2) async def test_first_or_404_works(ignore_base, app_ctx, anyio_backend): user_model = _seed_model(app_ctx) - user_instance = first_or_404(model.select(user_model).where(user_model.id == 1)) + user_instance = await first_or_404( + model.select(user_model).where(user_model.id == 1) + ) assert user_instance.name == "First User" with pytest.raises(NotFound): - first_or_404(model.select(user_model).where(user_model.id == 2)) + await first_or_404(model.select(user_model).where(user_model.id == 2)) async def test_first_or_404_async_works(ignore_base, app_ctx_async, anyio_backend): if anyio_backend == "asyncio": user_model = _seed_model(app_ctx_async) - user_instance = await first_or_404_async( + user_instance = await first_or_404( model.select(user_model).where(user_model.id == 1) ) assert user_instance.name == "First User" with pytest.raises(NotFound): - await first_or_404_async(model.select(user_model).where(user_model.id == 2)) + await first_or_404(model.select(user_model).where(user_model.id == 2)) async def test_one_or_404_works(ignore_base, app_ctx, anyio_backend): user_model = _seed_model(app_ctx) - user_instance = one_or_404(model.select(user_model).where(user_model.id == 1)) + user_instance = await one_or_404(model.select(user_model).where(user_model.id == 1)) assert user_instance.name == "First User" with pytest.raises(NotFound): - one_or_404(model.select(user_model).where(user_model.id == 2)) + await one_or_404(model.select(user_model).where(user_model.id == 2)) async def test_one_or_404_async_works(ignore_base, app_ctx_async, anyio_backend): if anyio_backend == "asyncio": user_model = _seed_model(app_ctx_async) - user_instance = await one_or_404_async( + user_instance = await one_or_404( model.select(user_model).where(user_model.id == 1) ) assert user_instance.name == "First User" with pytest.raises(NotFound): - await one_or_404_async(model.select(user_model).where(user_model.id == 2)) + await one_or_404(model.select(user_model).where(user_model.id == 2)) diff --git a/tests/test_service.py b/tests/test_service.py index 923ee24..a3a1b8f 100644 --- a/tests/test_service.py +++ b/tests/test_service.py @@ -6,7 +6,7 @@ from ellar_sql.constant import DATABASE_BIND_KEY, DEFAULT_KEY from ellar_sql.model.database_binds import ( __model_database_metadata__, - get_database_bind, + get_metadata, ) from ellar_sql.schemas import ModelBaseConfig from ellar_sql.services import EllarSQLService @@ -56,7 +56,7 @@ def test_custom_metadata_2x(ignore_base): class Base(model.Model): __base_config__ = ModelBaseConfig( - use_bases=[model.DeclarativeBase], make_declarative_base=True + use_bases=[model.DeclarativeBase], as_base=True ) metadata = custom_metadata @@ -72,8 +72,8 @@ def test_metadata_per_bind(tmp_path, ignore_base): }, root_path=str(tmp_path), ) - assert get_database_bind("a").info[DATABASE_BIND_KEY] == "a" - assert get_database_bind("default").info[DATABASE_BIND_KEY] == "default" + assert get_metadata("a").metadata.info[DATABASE_BIND_KEY] == "a" + assert get_metadata("default").metadata.info[DATABASE_BIND_KEY] == "default" def test_setup_fails_when_default_database_is_not_configured(tmp_path, ignore_base): @@ -98,7 +98,7 @@ def test_setup_fails_when_default_database_is_not_configured(tmp_path, ignore_ba def test_copy_naming_convention(tmp_path, ignore_base): class Base(model.Model): __base_config__ = ModelBaseConfig( - use_bases=[model.DeclarativeBase], make_declarative_base=True + use_bases=[model.DeclarativeBase], as_base=True ) metadata = model.MetaData(naming_convention={"pk": "spk_%(table_name)s"}) @@ -111,8 +111,8 @@ class Base(model.Model): ) assert Base.metadata.naming_convention["pk"] == "spk_%(table_name)s" assert ( - get_database_bind("a").naming_convention - == get_database_bind("default").naming_convention + get_metadata("a").metadata.naming_convention + == get_metadata("default").metadata.naming_convention ) @@ -206,13 +206,13 @@ def test_reflect(tmp_path, ignore_base): }, root_path=str(tmp_path), ) - default_metadata = get_database_bind("default") + default_metadata = get_metadata("default").metadata assert not default_metadata.tables db_service.reflect() - assert "user" in __model_database_metadata__["default"].tables - assert "post" in __model_database_metadata__["post"].tables + assert "user" in __model_database_metadata__["default"].metadata.tables + assert "post" in __model_database_metadata__["post"].metadata.tables @pytest.mark.asyncio @@ -239,14 +239,14 @@ class Post(model.Model): with pytest.raises(sa_exc.OperationalError): await session.execute(model.select(Post)) - await db_service.create_all_async() + db_service.create_all() user_res = await session.execute(model.select(User)) user_res.scalars() post_res = await session.execute(model.select(Post)) post_res.scalars() - await db_service.drop_all_async() + db_service.drop_all() with pytest.raises(sa_exc.OperationalError): await session.execute(model.select(User)) @@ -268,7 +268,7 @@ async def test_service_reflect_async(tmp_path, ignore_base): model.Table( "post", model.Column("id", model.Integer, primary_key=True), __database__="post" ) - await db_service.create_all_async() + db_service.create_all() del db_service __model_database_metadata__.clear() @@ -279,28 +279,13 @@ async def test_service_reflect_async(tmp_path, ignore_base): }, root_path=str(tmp_path), ) - default_metadata = get_database_bind("default") + default_metadata = get_metadata("default").metadata assert not default_metadata.tables - await db_service.reflect_async() - - assert "user" in __model_database_metadata__["default"].tables - assert "post" in __model_database_metadata__["post"].tables - - -def test_using_create_drop_reflect_for_async_engine_fails( - db_service_async, ignore_base -): - model.Table("user", model.Column("id", model.Integer, primary_key=True)) - - with pytest.raises(Exception, match="Use `create_all_async` instead"): - db_service_async.create_all() - - with pytest.raises(Exception, match="Use `drop_all_async` instead"): - db_service_async.drop_all() + db_service.reflect() - with pytest.raises(Exception, match="Use `reflect_async` instead"): - db_service_async.reflect() + assert "user" in __model_database_metadata__["default"].metadata.tables + assert "post" in __model_database_metadata__["post"].metadata.tables @pytest.mark.asyncio @@ -310,13 +295,13 @@ async def test_using_create_drop_reflect_async_for_sync_engine_work( class User(model.Model): id = model.Column(model.Integer, primary_key=True) - await db_service.create_all_async() + db_service.create_all() session = db_service.session_factory() session.execute(model.select(User)).scalars() - await db_service.drop_all_async() + db_service.drop_all() with pytest.raises(sa_exc.OperationalError): session.execute(model.select(User)).scalars() - await db_service.reflect_async() + db_service.reflect() diff --git a/tests/test_session.py b/tests/test_session.py index 1a06a94..27b0ca8 100644 --- a/tests/test_session.py +++ b/tests/test_session.py @@ -165,7 +165,7 @@ def test_session_multiple_dbs_case_1( bases, _ = base_super_class_as_dataclass class Base(model.Model): - __base_config__ = ModelBaseConfig(use_bases=bases, make_declarative_base=True) + __base_config__ = ModelBaseConfig(use_bases=bases, as_base=True) class User(Base): id: model.Mapped[int] = model.mapped_column( diff --git a/tests/test_table_database.py b/tests/test_table_database.py index 728285b..0171a3a 100644 --- a/tests/test_table_database.py +++ b/tests/test_table_database.py @@ -1,12 +1,12 @@ from ellar_sql import model -from ellar_sql.model.database_binds import get_database_bind +from ellar_sql.model.database_binds import get_metadata def test_bind_key_default(ignore_base): user_table = model.Table( "user", model.Column("id", model.Integer, primary_key=True) ) - default_metadata = get_database_bind("default") + default_metadata = get_metadata("default").metadata assert user_table.metadata is default_metadata @@ -16,7 +16,7 @@ def test_metadata_per_bind(ignore_base): model.Column("id", model.Integer, primary_key=True), __database__="other", ) - other_metadata = get_database_bind("other") + other_metadata = get_metadata("other").metadata assert user_table.metadata is other_metadata @@ -29,8 +29,8 @@ def test_multiple_binds_same_table_name(ignore_base): model.Column("id", model.Integer, primary_key=True), __database__="other", ) - other_metadata = get_database_bind("other") - default_metadata = get_database_bind("default") + other_metadata = get_metadata("other").metadata + default_metadata = get_metadata("default").metadata assert user1_table.metadata is default_metadata assert user2_table.metadata is other_metadata @@ -44,5 +44,5 @@ def test_explicit_metadata(ignore_base): __database__="other", ) assert user_table.metadata is other_metadata - other_metadata = get_database_bind("other") + other_metadata = get_metadata("other") assert other_metadata is None diff --git a/tests/test_type_decorators/test_file_upload.py b/tests/test_type_decorators/test_file_upload.py index 65dee41..657d27f 100644 --- a/tests/test_type_decorators/test_file_upload.py +++ b/tests/test_type_decorators/test_file_upload.py @@ -1,165 +1,165 @@ -import os -import uuid -from io import BytesIO -from unittest.mock import patch - -import pytest -import sqlalchemy.exc as sa_exc -from ellar.common.datastructures import ContentFile, UploadFile -from ellar.core.files import storages -from starlette.datastructures import Headers - -from ellar_sql import model -from ellar_sql.model.utils import MB - - -def serialize_file_data(file): - keys = { - "original_filename", - "content_type", - "extension", - "file_size", - "service_name", - } - return {k: v for k, v in file.to_dict().items() if k in keys} - - -def test_file_column_type(db_service, ignore_base, tmp_path): - path = str(tmp_path / "files") - fs = storages.FileSystemStorage(path) - - class File(model.Model): - id: model.Mapped[uuid.uuid4] = model.mapped_column( - "id", model.Integer(), nullable=False, unique=True, primary_key=True - ) - file: model.Mapped[model.FileObject] = model.mapped_column( - "file", model.FileField(storage=fs), nullable=False - ) - - db_service.create_all() - session = db_service.session_factory() - session.add(File(file=ContentFile(b"Testing file column type", name="text.txt"))) - session.commit() - - file: File = session.execute(model.select(File)).scalar() - assert "content_type=text/plain" in repr(file.file) - - data = serialize_file_data(file.file) - assert data == { - "content_type": "text/plain", - "extension": ".txt", - "file_size": 24, - "original_filename": "text.txt", - "service_name": "local", - } - - assert os.listdir(path)[0].split(".")[1] == "txt" - - -def test_file_column_invalid_file_extension(db_service, ignore_base, tmp_path): - fs = storages.FileSystemStorage(str(tmp_path / "files")) - - class File(model.Model): - id: model.Mapped[uuid.uuid4] = model.mapped_column( - "id", model.Integer(), nullable=False, unique=True, primary_key=True - ) - file: model.Mapped[model.FileObject] = model.mapped_column( - "file", - model.FileField(storage=fs, allowed_content_types=["application/pdf"]), - nullable=False, - ) - - with pytest.raises(sa_exc.StatementError) as stmt_exc: - db_service.create_all() - session = db_service.session_factory() - session.add( - File(file=ContentFile(b"Testing file column type", name="text.txt")) - ) - session.commit() - assert ( - str(stmt_exc.value.orig) - == "Content type is not supported text/plain. Valid options are: application/pdf" - ) - - -@patch( - "ellar_sql.model.typeDecorator.file.base.magic_mime_from_buffer", - return_value=None, -) -def test_file_column_invalid_file_extension_case_2( - mock_buffer, db_service, ignore_base, tmp_path -): - fs = storages.FileSystemStorage(str(tmp_path / "files")) - - class File(model.Model): - id: model.Mapped[uuid.uuid4] = model.mapped_column( - "id", model.Integer(), nullable=False, unique=True, primary_key=True - ) - file: model.Mapped[model.FileObject] = model.mapped_column( - "file", - model.FileField(storage=fs, allowed_content_types=["application/pdf"]), - nullable=False, - ) - - with pytest.raises(sa_exc.StatementError) as stmt_exc: - db_service.create_all() - session = db_service.session_factory() - session.add( - File( - file=UploadFile( - BytesIO(b"Testing file column type"), - size=24, - filename="test.txt", - headers=Headers({"content-type": ""}), - ) - ) - ) - session.commit() - assert mock_buffer.called - assert ( - str(stmt_exc.value.orig) - == "Content type is not supported . Valid options are: application/pdf" - ) - - -@patch("ellar_sql.model.typeDecorator.file.base.get_length", return_value=MB * 7) -def test_file_column_invalid_file_size_case_2( - mock_buffer, db_service, ignore_base, tmp_path -): - fs = storages.FileSystemStorage(str(tmp_path / "files")) - - class File(model.Model): - id: model.Mapped[uuid.uuid4] = model.mapped_column( - "id", model.Integer(), nullable=False, unique=True, primary_key=True - ) - file: model.Mapped[model.FileObject] = model.mapped_column( - "file", model.FileField(storage=fs, max_size=MB * 6), nullable=False - ) - - with pytest.raises(sa_exc.StatementError) as stmt_exc: - db_service.create_all() - session = db_service.session_factory() - session.add(File(file=ContentFile(b"Testing File Size Validation"))) - session.commit() - assert mock_buffer.called - assert str(stmt_exc.value.orig) == "Cannot store files larger than: 6291456 bytes" - - -def test_file_column_invalid_set(db_service, ignore_base, tmp_path): - fs = storages.FileSystemStorage(str(tmp_path / "files")) - - class File(model.Model): - id: model.Mapped[uuid.uuid4] = model.mapped_column( - "id", model.Integer(), nullable=False, unique=True, primary_key=True - ) - file: model.Mapped[model.FileObject] = model.mapped_column( - "file", model.FileField(storage=fs, max_size=MB * 6), nullable=False - ) - - db_service.create_all() - session = db_service.session_factory() - with pytest.raises(sa_exc.StatementError) as stmt_exc: - session.add(File(file={})) - session.commit() - - assert str(stmt_exc.value.orig) == "{} is not supported" +# import os +# import uuid +# from io import BytesIO +# from unittest.mock import patch +# +# import pytest +# import sqlalchemy.exc as sa_exc +# from ellar.common.datastructures import ContentFile, UploadFile +# from ellar.core.files import storages +# from starlette.datastructures import Headers +# +# from ellar_sql import model +# from ellar_sql.model.utils import MB +# +# +# def serialize_file_data(file): +# keys = { +# "original_filename", +# "content_type", +# "extension", +# "file_size", +# "service_name", +# } +# return {k: v for k, v in file.to_dict().items() if k in keys} +# +# +# def test_file_column_type(db_service, ignore_base, tmp_path): +# path = str(tmp_path / "files") +# fs = storages.FileSystemStorage(path) +# +# class File(model.Model): +# id: model.Mapped[int] = model.mapped_column( +# "id", model.Integer(), nullable=False, unique=True, primary_key=True +# ) +# file: model.Mapped[model.FileObject] = model.mapped_column( +# "file", model.FileField(storage=fs), nullable=False +# ) +# +# db_service.create_all() +# session = db_service.session_factory() +# session.add(File(file=ContentFile(b"Testing file column type", name="text.txt"))) +# session.commit() +# +# file: File = session.execute(model.select(File)).scalar() +# assert "content_type=text/plain" in repr(file.file) +# +# data = serialize_file_data(file.file) +# assert data == { +# "content_type": "text/plain", +# "extension": ".txt", +# "file_size": 24, +# "original_filename": "text.txt", +# "service_name": "local", +# } +# +# assert os.listdir(path)[0].split(".")[1] == "txt" +# +# +# def test_file_column_invalid_file_extension(db_service, ignore_base, tmp_path): +# fs = storages.FileSystemStorage(str(tmp_path / "files")) +# +# class File(model.Model): +# id: model.Mapped[int] = model.mapped_column( +# "id", model.Integer(), nullable=False, unique=True, primary_key=True +# ) +# file: model.Mapped[model.FileObject] = model.mapped_column( +# "file", +# model.FileField(storage=fs, allowed_content_types=["application/pdf"]), +# nullable=False, +# ) +# +# with pytest.raises(sa_exc.StatementError) as stmt_exc: +# db_service.create_all() +# session = db_service.session_factory() +# session.add( +# File(file=ContentFile(b"Testing file column type", name="text.txt")) +# ) +# session.commit() +# assert ( +# str(stmt_exc.value.orig) +# == "Content type is not supported text/plain. Valid options are: application/pdf" +# ) +# +# +# @patch( +# "ellar_sql.model.typeDecorator.file.base.magic_mime_from_buffer", +# return_value=None, +# ) +# def test_file_column_invalid_file_extension_case_2( +# mock_buffer, db_service, ignore_base, tmp_path +# ): +# fs = storages.FileSystemStorage(str(tmp_path / "files")) +# +# class File(model.Model): +# id: model.Mapped[int] = model.mapped_column( +# "id", model.Integer(), nullable=False, unique=True, primary_key=True +# ) +# file: model.Mapped[model.FileObject] = model.mapped_column( +# "file", +# model.FileField(storage=fs, allowed_content_types=["application/pdf"]), +# nullable=False, +# ) +# +# with pytest.raises(sa_exc.StatementError) as stmt_exc: +# db_service.create_all() +# session = db_service.session_factory() +# session.add( +# File( +# file=UploadFile( +# BytesIO(b"Testing file column type"), +# size=24, +# filename="test.txt", +# headers=Headers({"content-type": ""}), +# ) +# ) +# ) +# session.commit() +# assert mock_buffer.called +# assert ( +# str(stmt_exc.value.orig) +# == "Content type is not supported . Valid options are: application/pdf" +# ) +# +# +# @patch("ellar_sql.model.typeDecorator.file.base.get_length", return_value=MB * 7) +# def test_file_column_invalid_file_size_case_2( +# mock_buffer, db_service, ignore_base, tmp_path +# ): +# fs = storages.FileSystemStorage(str(tmp_path / "files")) +# +# class File(model.Model): +# id: model.Mapped[int] = model.mapped_column( +# "id", model.Integer(), nullable=False, unique=True, primary_key=True +# ) +# file: model.Mapped[model.FileObject] = model.mapped_column( +# "file", model.FileField(storage=fs, max_size=MB * 6), nullable=False +# ) +# +# with pytest.raises(sa_exc.StatementError) as stmt_exc: +# db_service.create_all() +# session = db_service.session_factory() +# session.add(File(file=ContentFile(b"Testing File Size Validation"))) +# session.commit() +# assert mock_buffer.called +# assert str(stmt_exc.value.orig) == "Cannot store files larger than: 6291456 bytes" +# +# +# def test_file_column_invalid_set(db_service, ignore_base, tmp_path): +# fs = storages.FileSystemStorage(str(tmp_path / "files")) +# +# class File(model.Model): +# id: model.Mapped[int] = model.mapped_column( +# "id", model.Integer(), nullable=False, unique=True, primary_key=True +# ) +# file: model.Mapped[model.FileObject] = model.mapped_column( +# "file", model.FileField(storage=fs, max_size=MB * 6), nullable=False +# ) +# +# db_service.create_all() +# session = db_service.session_factory() +# with pytest.raises(sa_exc.StatementError) as stmt_exc: +# session.add(File(file={})) +# session.commit() +# +# assert str(stmt_exc.value.orig) == "{} is not supported" diff --git a/tests/test_type_decorators/test_image_upload.py b/tests/test_type_decorators/test_image_upload.py index a949541..a0cbd73 100644 --- a/tests/test_type_decorators/test_image_upload.py +++ b/tests/test_type_decorators/test_image_upload.py @@ -1,229 +1,229 @@ -import os -import uuid -from pathlib import Path - -import pytest -import sqlalchemy.exc as sa_exc -from ellar.common.datastructures import ContentFile, UploadFile -from ellar.core.files import storages -from starlette.datastructures import Headers - -from ellar_sql import model -from ellar_sql.model.utils import get_length - -fixtures_dir = Path(__file__).parent / "fixtures" - - -def get_image_upload(file, *, filename): - return UploadFile( - file, - filename=filename, - size=get_length(file), - headers=Headers({"content-type": ""}), - ) - - -def serialize_file_data(file): - keys = { - "original_filename", - "content_type", - "extension", - "file_size", - "service_name", - "height", - "width", - } - return {k: v for k, v in file.to_dict().items() if k in keys} - - -def test_image_column_type(db_service, ignore_base, tmp_path): - path = str(tmp_path / "images") - fs = storages.FileSystemStorage(path) - - class Image(model.Model): - id: model.Mapped[uuid.uuid4] = model.mapped_column( - "id", model.Integer(), nullable=False, unique=True, primary_key=True - ) - image: model.Mapped[model.ImageFileObject] = model.mapped_column( - "image", model.ImageFileField(storage=fs), nullable=False - ) - - with open(fixtures_dir / "image.png", mode="rb+") as fp: - db_service.create_all() - session = db_service.session_factory() - session.add(Image(image=get_image_upload(filename="image.png", file=fp))) - session.commit() - - file: Image = session.execute(model.select(Image)).scalar() - - data = serialize_file_data(file.image) - assert data == { - "content_type": "image/png", - "extension": ".png", - "file_size": 1590279, - "height": 1080, - "original_filename": "image.png", - "service_name": "local", - "width": 1080, - } - - assert os.listdir(path)[0].split(".")[1] == "png" - - -def test_image_file_with_cropping_details_set_on_column( - db_service, ignore_base, tmp_path -): - fs = storages.FileSystemStorage(str(tmp_path / "images")) - - class Image2(model.Model): - id: model.Mapped[uuid.uuid4] = model.mapped_column( - "id", model.Integer(), nullable=False, unique=True, primary_key=True - ) - image: model.Mapped[model.ImageFileObject] = model.mapped_column( - "image", - model.ImageFileField( - storage=fs, - crop=model.CroppingDetails(x=100, y=200, height=400, width=400), - ), - nullable=False, - ) - - with open(fixtures_dir / "image.png", mode="rb+") as fp: - db_service.create_all() - session = db_service.session_factory() - session.add(Image2(image=get_image_upload(filename="image.png", file=fp))) - session.commit() - - image_2: Image2 = session.execute(model.select(Image2)).scalar() - - data = serialize_file_data(image_2.image) - assert data == { - "content_type": "image/png", - "extension": ".png", - "file_size": 108477, - "height": 400, - "original_filename": "image.png", - "service_name": "local", - "width": 400, - } - - -def test_image_file_with_cropping_details_on_set(db_service, ignore_base, tmp_path): - fs = storages.FileSystemStorage(str(tmp_path / "images")) - - class Image3(model.Model): - id: model.Mapped[uuid.uuid4] = model.mapped_column( - "id", model.Integer(), nullable=False, unique=True, primary_key=True - ) - image: model.Mapped[model.ImageFileObject] = model.mapped_column( - "image", model.ImageFileField(storage=fs), nullable=False - ) - - db_service.create_all() - session = db_service.session_factory() - - with open(fixtures_dir / "image.png", mode="rb+") as fp: - file = get_image_upload(filename="image.png", file=fp) - image_input = (file, model.CroppingDetails(x=100, y=200, height=400, width=400)) - - session.add(Image3(image=image_input)) - session.commit() - - image_3: Image3 = session.execute(model.select(Image3)).scalar() - - data = serialize_file_data(image_3.image) - assert data == { - "content_type": "image/png", - "extension": ".png", - "file_size": 108477, - "height": 400, - "original_filename": "image.png", - "service_name": "local", - "width": 400, - } - - -def test_image_file_with_cropping_details_override(db_service, ignore_base, tmp_path): - fs = storages.FileSystemStorage(str(tmp_path / "images")) - - class Image4(model.Model): - id: model.Mapped[uuid.uuid4] = model.mapped_column( - "id", model.Integer(), nullable=False, unique=True, primary_key=True - ) - image: model.Mapped[model.ImageFileObject] = model.mapped_column( - "image", - model.ImageFileField( - storage=fs, - crop=model.CroppingDetails(x=100, y=200, height=400, width=400), - ), - nullable=False, - ) - - db_service.create_all() - session = db_service.session_factory() - - with open(fixtures_dir / "image.png", mode="rb+") as fp: - file = get_image_upload(filename="image.png", file=fp) - image_input = (file, model.CroppingDetails(x=100, y=200, height=300, width=300)) - - session.add(Image4(image=image_input)) - session.commit() - - image_4: Image4 = session.execute(model.select(Image4)).scalar() - - data = serialize_file_data(image_4.image) - assert data == { - "content_type": "image/png", - "extension": ".png", - "file_size": 54508, - "height": 300, - "original_filename": "image.png", - "service_name": "local", - "width": 300, - } - - -def test_image_column_invalid_set(db_service, ignore_base, tmp_path): - fs = storages.FileSystemStorage(str(tmp_path / "files")) - - class Image3(model.Model): - id: model.Mapped[uuid.uuid4] = model.mapped_column( - "id", model.Integer(), nullable=False, unique=True, primary_key=True - ) - image: model.Mapped[model.ImageFileObject] = model.mapped_column( - "image", model.ImageFileField(storage=fs), nullable=False - ) - - db_service.create_all() - session = db_service.session_factory() - - with pytest.raises(sa_exc.StatementError) as stmt_exc: - invalid_input = (ContentFile(b"Invalid Set"), {}) - session.add(Image3(image=invalid_input)) - session.commit() - assert str(stmt_exc.value.orig) == ( - "Invalid data was provided for ImageFileField. " - "Accept values: UploadFile or (UploadFile, CroppingDetails)" - ) - - -def test_image_column_invalid_set_case_2(db_service, ignore_base, tmp_path): - fs = storages.FileSystemStorage(str(tmp_path / "files")) - - class Image3(model.Model): - id: model.Mapped[uuid.uuid4] = model.mapped_column( - "id", model.Integer(), nullable=False, unique=True, primary_key=True - ) - image: model.Mapped[model.ImageFileObject] = model.mapped_column( - "image", model.ImageFileField(storage=fs), nullable=False - ) - - db_service.create_all() - session = db_service.session_factory() - - with pytest.raises(sa_exc.StatementError) as stmt_exc: - session.add(Image3(image=ContentFile(b"Not an image"))) - session.commit() - assert str(stmt_exc.value.orig) == ( - "Content type is not supported text/plain. Valid options are: image/jpeg, image/png" - ) +# import os +# import uuid +# from pathlib import Path +# +# import pytest +# import sqlalchemy.exc as sa_exc +# from ellar.common.datastructures import ContentFile, UploadFile +# from ellar.core.files import storages +# from starlette.datastructures import Headers +# +# from ellar_sql import model +# from ellar_sql.model.utils import get_length +# +# fixtures_dir = Path(__file__).parent / "fixtures" +# +# +# def get_image_upload(file, *, filename): +# return UploadFile( +# file, +# filename=filename, +# size=get_length(file), +# headers=Headers({"content-type": ""}), +# ) +# +# +# def serialize_file_data(file): +# keys = { +# "original_filename", +# "content_type", +# "extension", +# "file_size", +# "service_name", +# "height", +# "width", +# } +# return {k: v for k, v in file.to_dict().items() if k in keys} +# +# +# def test_image_column_type(db_service, ignore_base, tmp_path): +# path = str(tmp_path / "images") +# fs = storages.FileSystemStorage(path) +# +# class Image(model.Model): +# id: model.Mapped[int] = model.mapped_column( +# "id", model.Integer(), nullable=False, unique=True, primary_key=True +# ) +# image: model.Mapped[model.ImageFileObject] = model.mapped_column( +# "image", model.ImageFileField(storage=fs), nullable=False +# ) +# +# with open(fixtures_dir / "image.png", mode="rb+") as fp: +# db_service.create_all() +# session = db_service.session_factory() +# session.add(Image(image=get_image_upload(filename="image.png", file=fp))) +# session.commit() +# +# file: Image = session.execute(model.select(Image)).scalar() +# +# data = serialize_file_data(file.image) +# assert data == { +# "content_type": "image/png", +# "extension": ".png", +# "file_size": 1590279, +# "height": 1080, +# "original_filename": "image.png", +# "service_name": "local", +# "width": 1080, +# } +# +# assert os.listdir(path)[0].split(".")[1] == "png" +# +# +# def test_image_file_with_cropping_details_set_on_column( +# db_service, ignore_base, tmp_path +# ): +# fs = storages.FileSystemStorage(str(tmp_path / "images")) +# +# class Image2(model.Model): +# id: model.Mapped[int] = model.mapped_column( +# "id", model.Integer(), nullable=False, unique=True, primary_key=True +# ) +# image: model.Mapped[model.ImageFileObject] = model.mapped_column( +# "image", +# model.ImageFileField( +# storage=fs, +# crop=model.CroppingDetails(x=100, y=200, height=400, width=400), +# ), +# nullable=False, +# ) +# +# with open(fixtures_dir / "image.png", mode="rb+") as fp: +# db_service.create_all() +# session = db_service.session_factory() +# session.add(Image2(image=get_image_upload(filename="image.png", file=fp))) +# session.commit() +# +# image_2: Image2 = session.execute(model.select(Image2)).scalar() +# +# data = serialize_file_data(image_2.image) +# assert data == { +# "content_type": "image/png", +# "extension": ".png", +# "file_size": 108477, +# "height": 400, +# "original_filename": "image.png", +# "service_name": "local", +# "width": 400, +# } +# +# +# def test_image_file_with_cropping_details_on_set(db_service, ignore_base, tmp_path): +# fs = storages.FileSystemStorage(str(tmp_path / "images")) +# +# class Image3(model.Model): +# id: model.Mapped[int] = model.mapped_column( +# "id", model.Integer(), nullable=False, unique=True, primary_key=True +# ) +# image: model.Mapped[model.ImageFileObject] = model.mapped_column( +# "image", model.ImageFileField(storage=fs), nullable=False +# ) +# +# db_service.create_all() +# session = db_service.session_factory() +# +# with open(fixtures_dir / "image.png", mode="rb+") as fp: +# file = get_image_upload(filename="image.png", file=fp) +# image_input = (file, model.CroppingDetails(x=100, y=200, height=400, width=400)) +# +# session.add(Image3(image=image_input)) +# session.commit() +# +# image_3: Image3 = session.execute(model.select(Image3)).scalar() +# +# data = serialize_file_data(image_3.image) +# assert data == { +# "content_type": "image/png", +# "extension": ".png", +# "file_size": 108477, +# "height": 400, +# "original_filename": "image.png", +# "service_name": "local", +# "width": 400, +# } +# +# +# def test_image_file_with_cropping_details_override(db_service, ignore_base, tmp_path): +# fs = storages.FileSystemStorage(str(tmp_path / "images")) +# +# class Image4(model.Model): +# id: model.Mapped[int] = model.mapped_column( +# "id", model.Integer(), nullable=False, unique=True, primary_key=True +# ) +# image: model.Mapped[model.ImageFileObject] = model.mapped_column( +# "image", +# model.ImageFileField( +# storage=fs, +# crop=model.CroppingDetails(x=100, y=200, height=400, width=400), +# ), +# nullable=False, +# ) +# +# db_service.create_all() +# session = db_service.session_factory() +# +# with open(fixtures_dir / "image.png", mode="rb+") as fp: +# file = get_image_upload(filename="image.png", file=fp) +# image_input = (file, model.CroppingDetails(x=100, y=200, height=300, width=300)) +# +# session.add(Image4(image=image_input)) +# session.commit() +# +# image_4: Image4 = session.execute(model.select(Image4)).scalar() +# +# data = serialize_file_data(image_4.image) +# assert data == { +# "content_type": "image/png", +# "extension": ".png", +# "file_size": 54508, +# "height": 300, +# "original_filename": "image.png", +# "service_name": "local", +# "width": 300, +# } +# +# +# def test_image_column_invalid_set(db_service, ignore_base, tmp_path): +# fs = storages.FileSystemStorage(str(tmp_path / "files")) +# +# class Image3(model.Model): +# id: model.Mapped[int] = model.mapped_column( +# "id", model.Integer(), nullable=False, unique=True, primary_key=True +# ) +# image: model.Mapped[model.ImageFileObject] = model.mapped_column( +# "image", model.ImageFileField(storage=fs), nullable=False +# ) +# +# db_service.create_all() +# session = db_service.session_factory() +# +# with pytest.raises(sa_exc.StatementError) as stmt_exc: +# invalid_input = (ContentFile(b"Invalid Set"), {}) +# session.add(Image3(image=invalid_input)) +# session.commit() +# assert str(stmt_exc.value.orig) == ( +# "Invalid data was provided for ImageFileField. " +# "Accept values: UploadFile or (UploadFile, CroppingDetails)" +# ) +# +# +# def test_image_column_invalid_set_case_2(db_service, ignore_base, tmp_path): +# fs = storages.FileSystemStorage(str(tmp_path / "files")) +# +# class Image3(model.Model): +# id: model.Mapped[int] = model.mapped_column( +# "id", model.Integer(), nullable=False, unique=True, primary_key=True +# ) +# image: model.Mapped[model.ImageFileObject] = model.mapped_column( +# "image", model.ImageFileField(storage=fs), nullable=False +# ) +# +# db_service.create_all() +# session = db_service.session_factory() +# +# with pytest.raises(sa_exc.StatementError) as stmt_exc: +# session.add(Image3(image=ContentFile(b"Not an image"))) +# session.commit() +# assert str(stmt_exc.value.orig) == ( +# "Content type is not supported text/plain. Valid options are: image/jpeg, image/png" +# ) pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy