GITBOOK-31: Announcing the 1.0 SDK

Silas Marvin · gitbook-bot · commit 0b34be2115d7 · 2024-03-04T20:16:42.000Z
diff --git a/pgml-cms/blog/SUMMARY.md b/pgml-cms/blog/SUMMARY.md
@@ -28,3 +28,4 @@
 * [Postgres Full Text Search is Awesome!](postgres-full-text-search-is-awesome.md)
 * [Oxidizing Machine Learning](oxidizing-machine-learning.md)
 * [Data is Living and Relational](data-is-living-and-relational.md)
+* [The 1.0 SDK is Here](the-1.0-sdk-is-here.md)
diff --git a/pgml-cms/blog/announcing-gptq-and-ggml-quantized-llm-support-for-huggingface-transformers.md b/pgml-cms/blog/announcing-gptq-and-ggml-quantized-llm-support-for-huggingface-transformers.md
@@ -3,9 +3,6 @@ description: >-
   GPTQ & GGML allow PostgresML to fit larger models in less RAM. These
   algorithms perform inference significantly faster on NVIDIA, Apple and Intel
   hardware.
-featured: false
-tags: [engineering]
-image: ".gitbook/assets/image (14).png"
 ---
 
 # Announcing GPTQ & GGML Quantized LLM support for Huggingface Transformers
diff --git a/pgml-cms/blog/announcing-support-for-aws-us-east-1-region.md b/pgml-cms/blog/announcing-support-for-aws-us-east-1-region.md
@@ -1,10 +1,3 @@
----
-description: >-
-  We added aws us east 1 to our list of support aws regions.
-featured: false
-tags: [product]
----
-
 # Announcing Support for AWS us-east-1 Region
 
 <div align="left">
diff --git a/pgml-cms/blog/data-is-living-and-relational.md b/pgml-cms/blog/data-is-living-and-relational.md
@@ -3,8 +3,6 @@ description: >-
   A common problem with data science and machine learning tutorials is the
   published and studied datasets are often nothing like what you’ll find in
   industry.
-featured: false
-tags: [engineering]
 ---
 
 # Data is Living and Relational
diff --git a/pgml-cms/blog/generating-llm-embeddings-with-open-source-models-in-postgresml.md b/pgml-cms/blog/generating-llm-embeddings-with-open-source-models-in-postgresml.md
@@ -2,8 +2,6 @@
 description: >-
   How to use the pgml.embed(...) function to generate embeddings with free and
   open source models in your own database.
-image: ".gitbook/assets/blog_image_generating_llm_embeddings.png"
-features: true
 ---
 
 # Generating LLM embeddings with open source models in PostgresML
diff --git a/pgml-cms/blog/how-to-improve-search-results-with-machine-learning.md b/pgml-cms/blog/how-to-improve-search-results-with-machine-learning.md
@@ -3,9 +3,6 @@ description: >-
   PostgresML makes it easy to use machine learning on your data and scale
   workloads horizontally in our cloud. One of the most common use cases is to
   improve search results.
-featured: true
-image: ".gitbook/assets/image (2) (2).png"
-tags: ["Engineering"]
 ---
 
 # How-to Improve Search Results with Machine Learning
diff --git a/pgml-cms/blog/introducing-the-openai-switch-kit-move-from-closed-to-open-source-ai-in-minutes.md b/pgml-cms/blog/introducing-the-openai-switch-kit-move-from-closed-to-open-source-ai-in-minutes.md
@@ -1,11 +1,8 @@
 ---
-featured: true
-tags: [engineering, product]
 image: https://postgresml.org/dashboard/static/images/open_source_ai_social_share.png
 description: >-
   Quickly and easily transition from the confines of the OpenAI APIs to higher
   quality embeddings and unrestricted text generation models.
-image: ".gitbook/assets/blog_image_switch_kit.png"
 ---
 
 # Introducing the OpenAI Switch Kit: Move from closed to open-source AI in minutes
diff --git a/pgml-cms/blog/postgres-full-text-search-is-awesome.md b/pgml-cms/blog/postgres-full-text-search-is-awesome.md
@@ -2,7 +2,6 @@
 description: >-
   If you want to improve your search results, don't rely on expensive O(n*m)
   word frequency statistics. Get new sources of data instead.
-image: ".gitbook/assets/image (53).png"
 ---
 
 # Postgres Full Text Search is Awesome!
diff --git a/pgml-cms/blog/postgresml-is-going-multicloud.md b/pgml-cms/blog/postgresml-is-going-multicloud.md
@@ -1,16 +1,5 @@
 # PostgresML is going multicloud
 
-<div align="left">
-
-<figure><img src=".gitbook/assets/lev.jpg" alt="Author" width="100"><figcaption></figcaption></figure>
-
-</div>
-
-Lev Kokotov
-
-Jan 18, 2024
-
-
 We started PostgresML two years ago with the goal of making machine learning and AI accessible and easy for everyone.  To make this a reality, we needed to deploy PostgresML as closely as possible to our end users. With that goal mind, today we're proud to announce support for a new cloud provider: Azure.
 
 ### How we got here
diff --git a/pgml-cms/blog/speeding-up-vector-recall-5x-with-hnsw.md b/pgml-cms/blog/speeding-up-vector-recall-5x-with-hnsw.md
@@ -3,9 +3,6 @@ description: >-
   HNSW indexing is the latest upgrade in vector recall performance. In this post
   we announce our updated SDK that utilizes HNSW indexing to give world class
   performance in vector search.
-tags: [engineering]
-featured: true
-image: ".gitbook/assets/blog_image_hnsw.png"
 ---
 
 # Speeding up vector recall 5x with HNSW
diff --git a/pgml-cms/blog/the-1.0-sdk-is-here.md b/pgml-cms/blog/the-1.0-sdk-is-here.md
@@ -0,0 +1,194 @@
+---
+description: >-
+  Our official pgml SDK has been stabilized and released for Python and
+  JavaScript.
+---
+
+# The 1.0 SDK is Here
+
+## Announcing the Release of our Official PGML 1.0 SDK
+
+We have spent the last few months stabilizing and finalizing the 1.0 version of our SDK in both JavaScript and Python.
+
+This release comes with a bunch of performance improvements and new features. To highlight a few of the capabilities of our new SDK:
+
+* Create Collections for storing, searching over, and managing groups of documents
+* Define powerful and flexible Pipelines to dictate ingesting, splitting, embedding, and indexing of documents
+* Search over documents and document chunks using semantic search, full text search, or hybrid semantic and full text search with extensive options for filtering on additional metadata
+* Utilize almost any of the powerful embedding models available on HuggingFace
+* Its all SQL! Get hands on with an ER diagram of your Collection and query from it however you want
+
+Our SDK has been built specifically with the task of searching in mind. [We use it power the search on our own website](https://github.com/postgresml/postgresml/blob/6ba605d67016a1177d410d1eb91ae8763b4784c4/pgml-dashboard/src/utils/markdown.rs#L1243), [and to perform RAG with our ChatBot demo](https://github.com/postgresml/postgresml/blob/b3b5f03eb6c54bec88120617d5175279273d81d1/pgml-dashboard/src/api/chatbot.rs#L527).
+
+### Why It's Exciting
+
+Our SDK is no different from any other companies. It abstracts away some complexities of managing SQL tables, building complex queries, and other boring and repetitive tasks, but the SDK itself is not groundbreaking.&#x20;
+
+We think our SDK release is exciting because the underlying technology we use is something worth being excited about. Our SDK relies on our open source postgres extension to perform machine learning tasks using SQL. The lighting fast document embedding and magic-like hybrid search are all relatively simple SQL queries utilizing our postgres extension. Everything happens locally in your database without using any network calls.
+
+What does it actually look like? Given some Collection and Pipeline defined below:
+
+{% tabs %}
+{% tab title="JavaScript" %}
+```javascript
+// Create Collection and Pipeline
+const collection = pgml.newCollection("my_collection");
+const pipeline = pgml.newPipeline("my_pipeline", {
+  text: {
+    splitter: { model: "recursive_character" },
+    semantic_search: {
+      model: "intfloat/e5-small",
+    },
+  },
+});
+await collection.add_pipeline(pipeline);
+
+// Upsert a document
+const documents = [
+  { id: "document_one", text: "Here is some hidden value 1000" }
+];
+await collection.upsert_documents(documents);
+
+// Search over our collection
+const results = await collection.vector_search(
+  {
+    query: {
+      fields: {
+        text: {
+          query: "What is the hidden value?"
+        },
+      },
+    },
+    limit: 5,
+  },
+  pipeline,
+);
+console.log(results);
+```
+{% endtab %}
+
+{% tab title="Python" %}
+```python
+# Create Collection and Pipeline
+collection = Collection("my_collection")
+pipeline = Pipeline(
+    "my_pipeline",
+    {
+        "text": {
+            "splitter": {"model": "recursive_character"},
+            "semantic_search": {
+                "model": "intfloat/e5-small",
+            },
+        },
+    },
+)
+
+# Upsert a document
+documents = [{"id": "document_one", "text": "Here is some hidden value 1000"}]
+await collection.upsert_documents(documents)
+
+# Search over our collection
+results = await collection.vector_search(
+    {
+        "query": {
+            "fields": {
+                "text": {"query": "What is the hidden value?"},
+            },
+        },
+        "limit": 5,
+    },
+    pipeline,
+)
+print(results)
+```
+{% endtab %}
+{% endtabs %}
+
+The SQL for the vector\_search is actually just:
+
+```sql
+WITH "pipeline" (
+    "schema"
+) AS (
+    SELECT
+        "schema"
+    FROM
+        "my_collection"."pipelines"
+    WHERE
+        "name" = 'my_pipeline'
+),
+"text_embedding" (
+    "embedding"
+) AS (
+    SELECT
+        pgml.embed (transformer => (
+                SELECT
+                    SCHEMA #>> '{text,semantic_search,model}'
+                FROM pipeline), text => 'What is the hidden value?', kwargs => '{}') AS "embedding"
+)
+SELECT
+    "document",
+    "chunk",
+    "score"
+FROM (
+    SELECT
+        1 - (embeddings.embedding <=> (
+                SELECT
+                    embedding
+                FROM "text_embedding")::vector) AS score,
+        "documents"."id",
+        "chunks"."chunk",
+        "documents"."document"
+    FROM
+        "my_collection_my_pipeline"."text_embeddings" AS "embeddings"
+        INNER JOIN "my_collection_my_pipeline"."text_chunks" AS "chunks" ON "chunks"."id" = "embeddings"."chunk_id"
+        INNER JOIN "my_collection"."documents" AS "documents" ON "documents"."id" = "chunks"."document_id"
+    ORDER BY
+        embeddings.embedding <=> (
+            SELECT
+                embedding
+            FROM "text_embedding")::vector ASC
+    LIMIT 5) AS "s"
+ORDER BY
+    "score" DESC
+LIMIT 5
+
+```
+
+> NOTE: This SQL is programmatically generated and built to work in situations where the query is searching over more than one field. That is why you see a redundant limit and sort. It doesn't tangibly affect the speed of the query in this case
+
+In fact, you can see every SQL query the SDK runs if you enable debug logging.
+
+{% tabs %}
+{% tab title="JavaScript" %}
+```javascript
+pgml.init_logger("DEBUG");
+```
+{% endtab %}
+
+{% tab title="Python" %}
+```python
+pgml.init_logger("DEBUG");
+```
+{% endtab %}
+{% endtabs %}
+
+Want to see an ER diagram of your collection?
+
+{% tabs %}
+{% tab title="JavaScript" %}
+```javascript
+console.log(await collection.generate_er_diagram(pipeline));
+```
+{% endtab %}
+
+{% tab title="Python" %}
+```python
+print(await collection.generate_er_diagram(pipeline))
+```
+{% endtab %}
+{% endtabs %}
+
+The above code prints out PlantUML script. Paste it into their online interpreter and checkout [the resulting diagram](https://www.plantuml.com/plantuml/uml/lPD1hjiW48Rtd6BqDbqz7w2hTnE4OMgJ08DWS9B6lNinbaELjceNqSk6\_F-WcUz7uu\_CAd7nJdo1sHe4dX5o93wqjaax55MgXQo1c6Xqw3DSBC-WmkJGW4vqoV0DaKK-sn1LKXwS3SYtY429Pn820rk-mLkSl1iqEOUQBONy1Yh3Pcgu2wY\_EkKhZ7QoWPj-Vs-7JgWOZLHSosmzLdGV6mSLRWvyfu3jSb0UjsjuvQPLdRLipaZaK8LcrYod2Y6V1sPpbWkcNEcE7Zywlx\_9JZyOqiNNqXxZeLuO9LD96cKfhTbsDFiOLRrJfZ3-7J7QYCu6t14VwhDVE-iPlVedhgpgO1osZbBF9Pnt-AvVXj-VylT5Q9Ea3GQlVoWSYVy\_2VeHZR5Xwccwzwf47VovqsDKjPVAI6bZtp-zTHs6TUtR8KJVvLQx\_\_huelzlvNLz3YC-C9ZYtKy0)[.](https://www.plantuml.com/plantuml/uml/lPD1hjiW48Rtd6BqDbqz7w2hTnE4OMgJ08DWS9B6lNinbaELjceNqSk6\_F-WcUz7uu\_CAd7nJdo1sHe4dX5o93wqjaax55MgXQo1c6Xqw3DSBC-WmkJGW4vqoV0DaKK-sn1LKXwS3SYtY429Pn820rk-mLkSl1iqEOUQBONy1Yh3Pcgu2wY\_EkKhZ7QoWPj-Vs-7JgWOZLHSosmzLdGV6mSLRWvyfu3jSb0UjsjuvQPLdRLipaZaK8LcrYod2Y6V1sPpbWkcNEcE7Zywlx\_9JZyOqiNNqXxZeLuO9LD96cKfhTbsDFiOLRrJfZ3-7J7QYCu6t14VwhDVE-iPlVedhgpgO1osZbBF9Pnt-AvVXj-VylT5Q9Ea3GQlVoWSYVy\_2VeHZR5Xwccwzwf47VovqsDKjPVAI6bZtp-zTHs6TUtR8KJVvLQx\_\_huelzlvNLz3YC-C9ZYtKy0)
+
+Thanks for reading about the release of our 1.0 SDK we hope you are as excited about it as we are!
diff --git a/pgml-cms/blog/using-postgresml-with-django-and-embedding-search.md b/pgml-cms/blog/using-postgresml-with-django-and-embedding-search.md
@@ -1,21 +1,10 @@
 ---
-description: >-
-  An example application using PostgresML and Django to build embedding based search.
-tags: [engineering]
+tags:
+  - engineering
 ---
 
 # Using PostgresML with Django and embedding search
 
-<div align="left">
-
-<figure><img src=".gitbook/assets/lev.jpg" alt="Author" width="100"><figcaption></figcaption></figure>
-
-</div>
-
-Lev Kokotov
-
-Feb 15, 2024
-
 Building web apps on top of PostgresML allows anyone to integrate advanced machine learning and AI features into their products without much work or needing to understand how it really works. In this blog post, we'll talk about building a classic to-do Django app, with the spicy addition of semantic search powered by embedding models running inside your PostgreSQL database.
 
 ### Getting the code
@@ -67,14 +56,13 @@ And that's it! In just a few lines of code, we're generating and storing high qu
 
 Djago Rest Framework provides the bulk of the implementation. We just added a `ModelViewSet` for the `TodoItem` model, with just one addition: a search endpoint. The search endpoint required us to write a bit of SQL to embed the search query and accept a few filters, but the core of it can be summarized in a single annotation on the query set:
 
-```python
-results = TodoItem.objects.annotate(
-    similarity=RawSQL(
+<pre class="language-python"><code class="lang-python"><strong>results = TodoItem.objects.annotate(
+</strong>    similarity=RawSQL(
         "pgml.embed('intfloat/e5-small', %s)::vector(384) &#x3C;=> embedding",
         [query],
     )
 ).order_by("similarity")
-```
+</code></pre>
 
 This single line of SQL does quite a bit:
 
@@ -88,7 +76,7 @@ All of this happens inside PostgresML. Our Django app doesn't need to implement
 
 Before going forward, make sure you have the app running either locally or in a cloud provider of your choice. If hosting it somewhere, replace `localhost:8000` with the URL and port of your service.
 
-The simplest way to interact with it is to use cURL or your preferred HTTP client. If running in debug mode locally, the Rest Framework provides a nice web UI which you can access on [http://localhost:8000/api/todo/](http://localhost:8000/api/todo/) using a browser.
+The simplest way to interact with it is to use cURL or an HTTP client of your preference. If running in debug mode locally, the Rest Framework provides a nice web UI which you can access on [http://localhost:8000/api/todos/](http://localhost:8000/api/todo/) using a browser.
 
 To create a to-do item with cURL, you can just run this:
 
@@ -115,13 +103,13 @@ In return, you'll get your to-do item alongside the embedding of the `descriptio
 
 The embedding contains 384 floating point numbers; we removed most of them in this blog post to make sure it fits on the page.
 
-You can try creating multiple to-do items for fun and profit. If the description is changed, so will the embedding, demonstrating how the `intfloat/e5-small` model understands the semantic meaning of your text.
+You can try creating multiple to-do items for fun and profit. If the description is changed, so will the embedding, demonstrating how the `intfloat/e5-small` model is understanding the semantic meaning of your text.
 
 ### Searching
 
 Once you have a few embeddings and to-dos stored in your database, the fun part of searching can begin. In a typical search example with PostgreSQL, you'd now be using `tsvector` to keyword match your to-dos to the search query with term frequency. That's a good technique, but semantic search is better.
 
-Our search endpoint accepts a query, a completed to-do filter, and a limit. To use it, you can just run this:
+We've created a simple search endpoint that accepts a query, a completed to-do filter, and a limit. To use it, you can just do this:
 
 ```bash
 curl \
@@ -134,7 +122,7 @@ curl \
 If you've created a bunch of different to-do items, you should get only one search result back, and exactly the one you were expecting:
 
 ```json
-"Make a New Year resolution"
+"Make a New Year resolution list"
 ```
 
 You can increase the `limit` to something larger and you should get more documents, in decreasing order of relevance.
