postgresml · montanalow · Aug 12, 2024 · Aug 10, 2024 · Aug 11, 2024 · Aug 11, 2024
diff --git a/README.md b/README.md
@@ -97,10 +97,10 @@ SELECT pgml.transform(
 ```
 
 ## Tabular data
-- [47+ classification and regression algorithms](https://postgresml.org/docs/api/sql-extension/pgml.train/)
+- [47+ classification and regression algorithms](https://postgresml.org/docs/open-source/pgml/api/pgml.train)
 - [8 - 40X faster inference than HTTP based model serving](https://postgresml.org/blog/postgresml-is-8x-faster-than-python-http-microservices)
 - [Millions of transactions per second](https://postgresml.org/blog/scaling-postgresml-to-one-million-requests-per-second)
-- [Horizontal scalability](https://github.com/postgresml/pgcat)
+- [Horizontal scalability](https://postgresml.org/docs/open-source/pgcat/)
 
 **Training a classification model**
 
@@ -142,7 +142,7 @@ docker run \
     sudo -u postgresml psql -d postgresml
 ```
 
-For more details, take a look at our [Quick Start with Docker](https://postgresml.org/docs/resources/developer-docs/quick-start-with-docker) documentation.
+For more details, take a look at our [Quick Start with Docker](https://postgresml.org/docs/open-source/pgml/developers/quick-start-with-docker) documentation.
 
 # Getting Started
 
@@ -1105,7 +1105,7 @@ pgml: SELECT logs->>'epoch' AS epoch, logs->>'step' AS step, logs->>'loss' AS lo
 During training, model is periodically uploaded to Hugging Face Hub. You will find the model at `https://huggingface.co/<username>/<project_name>`. An example model that was automatically pushed to Hugging Face Hub is [here](https://huggingface.co/santiadavani/imdb_review_sentiement).
 
 ### 6. Inference using fine-tuned model
-Now, that we have fine-tuned model on Hugging Face Hub, we can use [`pgml.transform`](https://postgresml.org/docs/introduction/apis/sql-extensions/pgml.transform/text-classification) to perform real-time predictions as well as batch predictions. 
+Now, that we have fine-tuned model on Hugging Face Hub, we can use [`pgml.transform`](/docs/open-source/pgml/api/pgml.transform) to perform real-time predictions as well as batch predictions. 
 
 **Real-time predictions**
 

diff --git a/pgml-cms/blog/semantic-search-in-postgres-in-15-minutes.md b/pgml-cms/blog/semantic-search-in-postgres-in-15-minutes.md
@@ -56,7 +56,7 @@ SELECT pgml.embed('mixedbread-ai/mxbai-embed-large-v1', 'Generating embeddings i
 
 !!!
 
-We used the [pgml.embed](/docs/api/sql-extension/pgml.embed) PostresML function to generate an embedding of the sentence "Generating embeddings in Postgres is fun!" using the [mixedbread-ai/mxbai-embed-large-v1](https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1) model from mixedbread.ai.
+We used the [pgml.embed](/docs/open-source/pgml/api/pgml.embed) PostresML function to generate an embedding of the sentence "Generating embeddings in Postgres is fun!" using the [mixedbread-ai/mxbai-embed-large-v1](https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1) model from mixedbread.ai.
 
 The output size of the vector varies per model, and in `mxbai-embed-large-v1` outputs vectors with 1024 dimensions: each vector contains 1024 floating point numbers. 
 

diff --git a/pgml-cms/blog/sentiment-analysis-using-express-js-and-postgresml.md b/pgml-cms/blog/sentiment-analysis-using-express-js-and-postgresml.md
@@ -24,7 +24,7 @@ Express is a mature JS backend framework touted as being fast and flexible. It i
 
 Sentiment analysis is a valuable tool for understanding the emotional polarity of text. You can determine if the text is positive, negative, or neutral. Common use cases include understanding product reviews, survey questions, and social media posts.
 
-In this application, we'll be applying sentiment analysis to note taking. Note taking and journaling can be an excellent practice for work efficiency and self improvement. However, if you are like me, it quickly becomes impossible to find and make use of anything I've written down. Notes that are useful must be easy to navigate. With this motivation, let's create a demo that can record notes throughout the day. Each day will have a summary and sentiment score. That way, if I'm looking for that time a few weeks ago when we were frustrated with our old MLOps platform — it will be easy to find.&#x20;
+In this application, we'll be applying sentiment analysis to note taking. Note taking and journaling can be an excellent practice for work efficiency and self improvement. However, if you are like me, it quickly becomes impossible to find and make use of anything I've written down. Notes that are useful must be easy to navigate. With this motivation, let's create a demo that can record notes throughout the day. Each day will have a summary and sentiment score. That way, if I'm looking for that time a few weeks ago when we were frustrated with our old MLOps platform — it will be easy to find.
 
 We will perform all the Machine Learning heavy lifting with the pgml extension function `pgml.transform()`. This brings Hugging Face Transformers into our data layer.
 
@@ -36,7 +36,7 @@ You can see the full code on [GitHub](https://github.com/postgresml/example-expr
 
 This app is composed of three main parts, reading and writing to a database, performing sentiment analysis on entries, and creating a summary.
 
-We are going to use [postgresql-client](https://www.npmjs.com/package/postgresql-client) to connect to our DB.&#x20;
+We are going to use [postgresql-client](https://www.npmjs.com/package/postgresql-client) to connect to our DB.
 
 When the application builds we ensure we have two tables, one for notes and one for the the daily summary and sentiment score.
 
@@ -62,7 +62,7 @@ const day = await connection.execute(`
 
 We also have three endpoints to hit:
 
-* `app.get(“/", async (req, res, next)` which returns all the notes for that day and the daily summary.&#x20;
+* `app.get(“/", async (req, res, next)` which returns all the notes for that day and the daily summary.
 * `app.post(“/add", async (req, res, next)` which accepts a new note entry and performs a sentiment analysis. We simplify the score by converting it to 1, 0, -1 for positive, neutral, negative and save it in our notes table.
 
 ```postgresql
@@ -146,8 +146,8 @@ not bad for less than an hour of coding.
 
 ### Final Thoughts
 
-This app is far from complete but does show an easy and scalable way to get started with ML in Express. From here I encourage you to head over to our [docs](https://postgresml.org/docs/api/sql-extension/) and see what other features could be added.
+This app is far from complete but does show an easy and scalable way to get started with ML in Express. From here I encourage you to head over to our [docs](https://postgresml.org/docs) and see what other features could be added.
 
-If SQL is not your thing, no worries. Check out or [JS SDK](https://postgresml.org/docs/api/client-sdk/getting-started) to streamline all our best practices with simple JavaScript.&#x20;
+If SQL is not your thing, no worries. Check out or [JS SDK](https://postgresml.org/docs/open-source/korvus/) to streamline all our best practices with simple JavaScript.
 
-We love hearing from you — please reach out to us on [Discord ](https://discord.gg/DmyJP3qJ7U)or simply [Contact Us](https://postgresml.org/contact) here if you have any questions or feedback.&#x20;
+We love hearing from you — please reach out to us on [Discord ](https://discord.gg/DmyJP3qJ7U)or simply [Contact Us](https://postgresml.org/contact) here if you have any questions or feedback.
diff --git a/pgml-cms/blog/using-postgresml-with-django-and-embedding-search.md b/pgml-cms/blog/using-postgresml-with-django-and-embedding-search.md
@@ -28,7 +28,7 @@ PostgresML allows anyone to integrate advanced AI capabilities into their applic
 
 Advanced search engines like Google use this technique to extract the meaning of search queries and rank the results based on what the user actually _wants_, unlike simple keyword matches which can easily give irrelevant results.
 
-To accomplish this, for each document in our app, we include an embedding column stored as a vector. A vector is just an array of floating point numbers. For each item in our to-do list, we automatically generate the embedding using the PostgresML [`pgml.embed()`](https://postgresml.org/docs/introduction/apis/sql-extensions/pgml.embed) function. This function runs inside the database and doesn't require the Django app to install the model locally.
+To accomplish this, for each document in our app, we include an embedding column stored as a vector. A vector is just an array of floating point numbers. For each item in our to-do list, we automatically generate the embedding using the PostgresML [`pgml.embed()`](/docs/open-source/pgml/api/pgml.embed) function. This function runs inside the database and doesn't require the Django app to install the model locally.
 
 An embedding model running inside PostgresML is able to extract the meaning of search queries & compare it to the meaning of the documents it stores, just like a human being would if they were able to search millions of documents in just a few milliseconds.
 

diff --git a/pgml-cms/docs/README.md b/pgml-cms/docs/README.md
@@ -23,16 +23,14 @@ PostgresML allows you to take advantage of the fundamental relationship between
 
 These capabilities are primarily provided by two open-source software projects, that may be used independently, but are designed to be used together with the rest of the Postgres ecosystem:
 
-* [**pgml**](/docs/api/sql-extension/) - an open source extension for PostgreSQL. It adds support for GPUs and the latest ML & AI algorithms _inside_ the database with a SQL API and no additional infrastructure, networking latency, or reliability costs.
-* [**PgCat**](/docs/product/pgcat/) - an open source connection pooler for PostgreSQL. It abstracts the scalability and reliability concerns of managing a distributed cluster of Postgres databases. Client applications connect only to the pooler, which handles load balancing, sharding, and failover, outside of any single database server.
+* [**pgml**](/docs/open-source/pgml/) - an open source extension for PostgreSQL. It adds support for GPUs and the latest ML & AI algorithms _inside_ the database with a SQL API and no additional infrastructure, networking latency, or reliability costs.
+* [**PgCat**](/docs/open-source/pgcat/) - an open source connection pooler for PostgreSQL. It abstracts the scalability and reliability concerns of managing a distributed cluster of Postgres databases. Client applications connect only to the pooler, which handles load balancing, sharding, and failover, outside of any single database server.
 
 <figure><img src=".gitbook/assets/architecture.png" alt="PostgresML architectural diagram"><figcaption></figcaption></figure>
 
-To learn more about how we designed PostgresML, take a look at our [architecture overview](/docs/resources/architecture/).
-
 ## Client SDK
 
-The PostgresML team also provides [native language SDKs](/docs/api/client-sdk/) which implement best practices for common ML & AI applications. The JavaScript and Python SDKs are generated from the a core Rust library, which provides a uniform API, correctness and efficiency across all environments.
+The PostgresML team also provides [native language SDKs](/docs/open-source/korvus/) which implement best practices for common ML & AI applications. The JavaScript and Python SDKs are generated from the a core Rust library, which provides a uniform API, correctness and efficiency across all environments.
 
 While using the SDK is completely optional, SDK clients can perform advanced machine learning tasks in a single SQL request, without having to transfer additional data, models, hardware or dependencies to the client application.
 

diff --git a/pgml-cms/docs/SUMMARY.md b/pgml-cms/docs/SUMMARY.md
@@ -23,31 +23,15 @@
 * [PGML](open-source/pgml/README.md)
   * [API](open-source/pgml/api/README.md)
     * [pgml.embed()](open-source/pgml/api/pgml.embed.md)
-    * [pgml.transform()](open-source/pgml/api/pgml.transform/README.md)
-      * [Fill-Mask](open-source/pgml/api/pgml.transform/fill-mask.md)
-      * [Question answering](open-source/pgml/api/pgml.transform/question-answering.md)
-      * [Summarization](open-source/pgml/api/pgml.transform/summarization.md)
-      * [Text classification](open-source/pgml/api/pgml.transform/text-classification.md)
-      * [Text Generation](open-source/pgml/api/pgml.transform/text-generation.md)
-      * [Text-to-Text Generation](open-source/pgml/api/pgml.transform/text-to-text-generation.md)
-      * [Token Classification](open-source/pgml/api/pgml.transform/token-classification.md)
-      * [Translation](open-source/pgml/api/pgml.transform/translation.md)
-      * [Zero-shot Classification](open-source/pgml/api/pgml.transform/zero-shot-classification.md)
+    * [pgml.transform()](open-source/pgml/api/pgml.transform.md)
     * [pgml.transform_stream()](open-source/pgml/api/pgml.transform_stream.md)
     * [pgml.deploy()](open-source/pgml/api/pgml.deploy.md)
     * [pgml.decompose()](open-source/pgml/api/pgml.decompose.md)
     * [pgml.chunk()](open-source/pgml/api/pgml.chunk.md)
     * [pgml.generate()](open-source/pgml/api/pgml.generate.md)
     * [pgml.predict()](open-source/pgml/api/pgml.predict/README.md)
       * [Batch Predictions](open-source/pgml/api/pgml.predict/batch-predictions.md)
-    * [pgml.train()](open-source/pgml/api/pgml.train/README.md)
-      * [Regression](open-source/pgml/api/pgml.train/regression.md)
-      * [Classification](open-source/pgml/api/pgml.train/classification.md)
-      * [Clustering](open-source/pgml/api/pgml.train/clustering.md)
-      * [Decomposition](open-source/pgml/api/pgml.train/decomposition.md)
-      * [Data Pre-processing](open-source/pgml/api/pgml.train/data-pre-processing.md)
-      * [Hyperparameter Search](open-source/pgml/api/pgml.train/hyperparameter-search.md)
-      * [Joint Optimization](open-source/pgml/api/pgml.train/joint-optimization.md)
+    * [pgml.train()](open-source/pgml/api/pgml.train.md)
     * [pgml.tune()](open-source/pgml/api/pgml.tune.md)
   * [Guides](open-source/pgml/guides/README.md)
     * [Embeddings](open-source/pgml/guides/embeddings/README.md)
@@ -56,11 +40,27 @@
       * [Aggregation](open-source/pgml/guides/embeddings/vector-aggregation.md)
       * [Similarity](open-source/pgml/guides/embeddings/vector-similarity.md)
       * [Normalization](open-source/pgml/guides/embeddings/vector-normalization.md)
+    * [LLMs](open-source/pgml/guides/llms/README.md)
+      * [Fill-Mask](open-source/pgml/guides/llms/fill-mask.md)
+      * [Question answering](open-source/pgml/guides/llms/question-answering.md)
+      * [Summarization](open-source/pgml/guides/llms/summarization.md)
+      * [Text classification](open-source/pgml/guides/llms/text-classification.md)
+      * [Text Generation](open-source/pgml/guides/llms/text-generation.md)
+      * [Text-to-Text Generation](open-source/pgml/guides/llms/text-to-text-generation.md)
+      * [Token Classification](open-source/pgml/guides/llms/token-classification.md)
+      * [Translation](open-source/pgml/guides/llms/translation.md)
+      * [Zero-shot Classification](open-source/pgml/guides/llms/zero-shot-classification.md)
+    * [Supervised Learning](open-source/pgml/guides/supervised-learning/README.md)
+      * [Regression](open-source/pgml/guides/supervised-learning/regression.md)
+      * [Classification](open-source/pgml/guides/supervised-learning/classification.md)
+      * [Clustering](open-source/pgml/guides/supervised-learning/clustering.md)
+      * [Decomposition](open-source/pgml/guides/supervised-learning/decomposition.md)
+      * [Data Pre-processing](open-source/pgml/guides/supervised-learning/data-pre-processing.md)
+      * [Hyperparameter Search](open-source/pgml/guides/supervised-learning/hyperparameter-search.md)
+      * [Joint Optimization](open-source/pgml/guides/supervised-learning/joint-optimization.md)
     * [Search](open-source/pgml/guides/improve-search-results-with-machine-learning.md)
     * [Chatbots](open-source/pgml/guides/chatbots/README.md)
-    * [Supervised Learning](open-source/pgml/guides/supervised-learning.md)
     * [Unified RAG](open-source/pgml/guides/unified-rag.md)
-    * [Natural Language Processing](open-source/pgml/guides/natural-language-processing.md)
     * [Vector database](open-source/pgml/guides/vector-database.md)
     <!--
     * [Search]()

diff --git a/pgml-cms/docs/introduction/getting-started/README.md b/pgml-cms/docs/introduction/getting-started/README.md
@@ -6,14 +6,14 @@ description: Getting starting with PostgresML, a GPU powered machine learning da
 
 A PostgresML deployment consists of multiple components working in concert to provide a complete Machine Learning platform:
 
-* PostgreSQL database, with [_pgml_](/docs/api/sql-extension/), _pgvector_ and many other extensions that add features useful in day-to-day and machine learning use cases
-* [PgCat pooler](/docs/product/pgcat/) to load balance thousands of concurrenct client requests across several database instances
+* PostgreSQL database, with `pgml`, `pgvector` and many other extensions that add features useful in day-to-day and machine learning use cases
+* [PgCat pooler](/docs/open-source/pgcat/) to load balance thousands of concurrenct client requests across several database instances
 * A web application to manage deployed models and share experiments analysis with SQL notebooks
 
-We provide a fully managed solution in [our cloud](create-your-database), and document a self-hosted installation in the [Developer Docs](/docs/resources/developer-docs/quick-start-with-docker).
+We provide a fully managed solution in [our cloud](/docs/cloud/overview), and document a self-hosted installation in the [Developer Docs](/docs/open-source/pgml/developers/quick-start-with-docker).
 
 <figure class="my-4"><img src="../../.gitbook/assets/architecture.png" alt="PostgresML architecture"><figcaption></figcaption></figure>
 
 By building PostgresML on top of a mature database, we get reliable backups for model inputs and proven scalability without reinventing the wheel, so that we can focus on providing access to the latest developments in open source machine learning and artificial intelligence.
 
-This guide will help you get started with [$100 credits](create-your-database), which includes access to GPU accelerated models and 5 GB of storage, or you can skip to our [Developer Docs](/docs/resources/developer-docs/quick-start-with-docker) to see how to run PostgresML locally with our Docker image.
+This guide will help you get started with [$100 credits](create-your-database), which includes access to GPU accelerated models and 5 GB of storage, or you can skip to our [Developer Docs](/docs/open-source/pgml/developers/quick-start-with-docker) to see how to run PostgresML locally with our Docker image.
-Original file line number
+Diff line change
@@ Expand Up @@
     !!!
-    We used the [pgml.embed](/docs/api/sql-extension/pgml.embed) PostresML function to generate an embedding of the sentence "Generating embeddings in Postgres is fun!" using the [mixedbread-ai/mxbai-embed-large-v1](https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1) model from mixedbread.ai.
+    We used the [pgml.embed](/docs/open-source/pgml/api/pgml.embed) PostresML function to generate an embedding of the sentence "Generating embeddings in Postgres is fun!" using the [mixedbread-ai/mxbai-embed-large-v1](https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1) model from mixedbread.ai.
     The output size of the vector varies per model, and in `mxbai-embed-large-v1` outputs vectors with 1024 dimensions: each vector contains 1024 floating point numbers.
@@ Expand Down @@