* [Pooler](deployment/self-hosting/pooler.md)

* [Replication](deployment/self-hosting/replication.md)

PostgresML is a Postgres extension written in Rust, so it can be built and installed on any system that supported by PostgreSQL and the Rust compiler. If you're planning on using GPU acceleration for Large Language Models or for XGBoost / LightGBM supervised learning, we would recommend you use an operating system well supported by Nvidia drivers and Cuda. Thankfully, that list is pretty large these days, including popular distributions like Ubuntu, Debian, RHEL, Centos, Fedora and OpenSuse.

PostgresML depends on a few system packages and libraries that should be installed separately in order to successfully compile it from source. The names of the packages vary based on the Linux distribution you're using, but in most cases you should be able to find all of them verbatim in your package manger repositories:

This guide assumes that you're using PostgreSQL 14, so if your Postgres version is different, replace `14` in `postgresql-server-dev-14` with the correct version of Postgres. PostgresML supports all Postgres versions supported by `pgrx` and the PostgreSQL community (as of this writing, versions 12 through 16).

All of our source code is open source and hosted in GitHub. You can download it with git:

git clone https://github.com/postgresml/postgresml

The repository contains the extension, the dashboard, SDKs, and all apps we've written that are powered by PostgresML.

For a typical deployment in production, you would need to compile and install the extension into your system PostgreSQL installation. PostgresML is using the `pgrx` Rust extension toolkit, so this is pretty easy and straight forward to do.

`pgrx` is open source and available from crates.io. We are currently using the `0.10.0` version. It's important that your `pgrx` version matches what we're using, since there are some hard dependencies between our code and theirs.

To install `pgrx`, simply run:

Before using `pgrx`, it needs to be initialized against the installed version of PostgreSQL. In this example, we'll be using the Ubuntu 22.04 default PostgreSQL 14 installation:

Now that `pgrx` is initialized, you can compile and install the extension:

This will produce a number of artifacts in `target/release/pg14-pgml` which you can then copy to their respective folders in `/usr` using `sudo cp`. At the time of this writing, `pgrx` is working on a command that does this automatically, but it has not been released yet.

Once the files are copied to their respective folders in `/usr`, you need to make sure the `pgml` extension is loaded in `shared_preload_libraries`. We use shared memory to control model versioning and other cool things that make PostgresML "just work". In `/etc/postgresql/14/main/postgresql.conf`, change or add the following line:

Restart Postgres for this change to take effect:

To make sure PostgresML is installed, you can create the extension in a database of your choice:

A pooler is a piece of software that is placed in front of a PostgreSQL cluster in order to load balance client connections and minimize the load placed on the database servers. Clients connect to the pooler, which pretends to be a Postgres database, and the pooler in turn connects to Postgres servers and forward clients' requests in an efficient manner.

Postgres is a process-based database server (as opposed to threads), and each client connection forks the primary process to operate in its own memory space. A fork is generally more expensive than a thread because of extra memory allocation and OS scheduling overhead, but with a properly configured pooler, Postgres achieves a high degree of concurrency at massive scale in production.

PostgresML caches machine learning models in the connection process memory space. For XGBoost/LightGBM/Scikit-learn models, which are typically only a few MBs in size, this is not a major concern, but for LLMs like Llama2 and Mistral, which are tens of gigabytes, the system memory and GPU memory usage is considerable. In order to be able to run these models effectively in production, the usage of a pooler running in transaction mode is essential. A pooler will route thousands of clients to the same Postgres server connection, reusing the same cached model, allowing for high concurrency and efficient use of resources.

The PostgreSQL open source community has developed many poolers over the years: PgBouncer, Odyssey, and PgPool. Each one has its pros and cons, but most of them can scale a PostgresML server effectively. At PostgresML, we developed our own pooler called PgCat, which supports many enterprise-grade features not available elsewhere that we needed to provide a seamless experience using Postgres in production, like load balancing, failover and sharding.

This guide will use PgCat as the pooler of choice.

If you have followed our [Self-hosting](./) guide, you can just install PgCat for Ubuntu 22.04 from our APT repository:

sudo apt install -y pgcat

If not, you can easily install it from source.

Download the source code from Github:

git clone https://github.com/postgresml/pgcat

If you don't have it already, install the Rust compiler from rust-lang.org:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Finally, compile PgCat in release mode and install it into your system folders:

<pre class="language-bash"><code class="lang-bash">cd pgcat &#x26;&#x26; \

cargo build --release &#x26;&#x26; \

<strong>sudo cp target/release/pgcat /usr/local/bin/pgcat &#x26;&#x26; \

</strong>sudo cp pgcat.toml /etc/pgcat.toml.example

</code></pre>

PgCat uses the TOML configuration language and, if installed from APT, will use the configuration file stored in `/etc/pgcat.toml`. If installed from source, you'll have to pass the configuration file path as an argument when launching.

This example will assume that you have a database called `postgresml` with a user `postgresml_user` already configured. You can create and use as many databases and users as you need. That being said, each database/user combination will be a separate connection pool in PgCat and create its own PostgreSQL server connections.

For a primary-only setup used to serve Large Language Models, the pooler configuration is pretty basic:

[general]

host = "0.0.0.0"

port = 6432

admin_username = "pgcat"

admin_password = "<secure password>"

server_lifetime = 86400000

idle_timeout = 86400000

[pools.postgresml]

pool_mode = "transaction"

[pools.postgresml.shards.0]

servers = [

["<primary hostname or IP address>", 5432, "primary"].

]

database = "postgresml"

[pools.postgresml.users.0]

username = "postgresml_user"

password = "<secure password>"

pool_size = 1

Important considerations here are the `pool_size` of only `1` which will create and maintain only one PostgreSQL connection loaded with the LLM. Both `idle_timeout` and `server_lifetime` settings are set to 24 hours, so every 24 hours a new PostgreSQL connection will be created and the old one closed. This may not be desirable since loading a LLM into the GPU can take several seconds. To avoid this, this value can be set to be arbitrarily large, e.g. 100 years. In that case, the connection will basically never be closed.

Having only one server connection is not mandatory. If your hardware allows to load more than one LLM into your GPUs, you can increase the `pool_size` to a larger value. Our dedicated databases currently support up to 256GB GPU-powered LLMs, so we allow considerably more connections than would be otherwise supported by say just a GeForce RTX 4080.

Once configured, the pooler is ready to go. If you installed it from our APT repository, you can just run:

sudo service pgcat start

If you compiled it from source, you can run it directly:

To validate that the pooler is running correctly, you can connect to it with `psql`:

PGPASSWORD="<secure password>" psql \

-h "127.0.0.1" \

-p 6432 \

-U postgresml_user \

-d postgresml