update

HerringtonDarkholme · HerringtonDarkholme · commit e19547638896 · 2024-12-20T01:28:00.000-05:00
diff --git a/website/blog/typed-napi.md b/website/blog/typed-napi.md
@@ -1,9 +1,14 @@
+---
+sidebar: false
+---
+
 # Improve Napi Typing
 
-What's type safety? Why?
+I'm thrilled to announce that [@ast-grep/napi] now supports typed, solving a [long standing issue](https://github.com/ast-grep/ast-grep/issues/48) in our feature request.
+
+In this blog post, we will walk through the problem and the [design](https://github.com/ast-grep/ast-grep/issues/1669) of the new feature. It will also be a valuable resource to write a good TypeScript type in general.
 
-https://github.com/ast-grep/ast-grep/issues/1669
-https://github.com/ast-grep/ast-grep/issues/48
+## What's type safety? Why?
 
 Writing AST manipulation code is hard. Even if we have a lot of helpful interactive tool, it's still hard to handle all edge cases.
 
@@ -25,74 +30,201 @@ Designing a good library in the modern JavaScript world is not only about provid
 * Performant: fast to compile. complex types can slow down the compiler
 
 
-It is really hard to provide a type system that is both Sound and Complete. This is similar to provide a good typing API.
+It is really hard to provide a type system that is both [Sound and Complete](https://logan.tw/posts/2014/11/12/soundness-and-completeness-of-the-type-system/#:~:text=A%20type%2Dsystem%20is%20sound,any%20false%20positive%20%5B2%5D.). This is similar to provide a good typing API.
 
-https://logan.tw/posts/2014/11/12/soundness-and-completeness-of-the-type-system/#:~:text=A%20type%2Dsystem%20is%20sound,any%20false%20positive%20%5B2%5D.
 
 TS libs nowaday probably pay too much attention to correctness IMHO.
 Having a type to check your path parameter in your routing is cool, but what's the cost?
 
 Designing a good TypeScript type is essentially a trade-off of these four aspects.
 
-# TreeSitter's types
+## TreeSitter's types
 
 Let's come back to ast-grep's problem. ast-grep is based on Tree-Sitter.
 
-Tree-Sitter's official API is untyped. It provies a uniform API to access the syntax tree across different languages. A node in Tree-Sitter has several methods like `kind`, `field`, `parent`, `children`, `range`, `text`, etc.
+Tree-Sitter's official API is untyped. It provies a uniform API to access the syntax tree across different languages. A node in Tree-Sitter has several common methods to access its node type, children, parent, and text content.
+
+```TypeScript
+class Node {
+  kind(): string // get the node type
+  field(name: string): Node // get a child node by field name
+  parent(): Node
+  children(): Node[]
+  text(): string
+}
+```
+The API is simple and easy to use, but it lacks type information.
 
-However, a specific language's syntax tree has a specific structure. For example, a function declaration in JavaScript has a `function` keyword, a name, a list of parameters, and a body. Other AST parser libraries encode this structure in their AST object types. For example, a `function_declaration` has fields like `parameters` and `body`.
+In contrast, a specific language's syntax tree has a specific structure. For example, a function declaration in JavaScript has a `function` keyword, a name, a list of parameters, and a body. Other AST parser libraries encode this structure in their AST object types. For example, a `function_declaration` has fields like `parameters` and `body`.
 
 Fortunately tree-sitter provides static node types in json.
 There are several challenges to generate TypeScript types from tree-sitter's static node types.
 
 1. json is hosted by parser library repo
-We needs type generation (it is like layman's type provider)
+We needs type generation (it is like F-sharp's type provider)
 2. json contains a lot unnamed kinds
 You are writing a compiler plugin, not elementary school math homework
 3. json has alias type
-
+For example, `declaration` is an alias of `function_declaration`, `class_declaration` and other declaration kinds.
 
 ## Design Type
 
-1. lenient type check if no information
-2. more strict checking if refined
+The design principle of the new API is to progressively provide a more strict code checking and completion when the user gives more type information.
+
+1. **Allow untyped AST access if no type information is provided**
+
+Existing untyped API is still available and it is the default behavior.
+The new feature should not break the existing code.
+
+2. **Allow user to type AST node and enjoy more type safety**
+
+The user can give types to AST nodes either manually or automatically.
+Both approaches should refine the general untyped AST nodes to typed AST nodes and bring type check and intelligent completion to the user.
 
 
 ## Define Type
 
-### Prune unnamed kinds
-For example `+`/`-`/`*`/`/` is too noisy for a general AST library
+## TreeSitter's `TypeMap`
+The new typed API will consume TreeSitte's [static node types](https://tree-sitter.github.io/tree-sitter/using-parsers#static-node-types) like below:
+
+```typescript
+interface TypeMpa {
+  [kind: string]: {
+    type: string
+    named: boolean
+    fields?: {
+      [field: string]: {
+        types: { type: string, named: boolean }[]
+      }
+    }
+    subtypes?: { type: string, named: boolean }[]
+  }
+}
+```
+What is `TypeMaps`? It is a type that contains all static node types. It is a map from kind to the static type of the kind.
+
+```typescript
+type TypeScriptMap = {
+  // AST node type definition
+  function_declaration: {
+    type: "function_declaration", // kind
+    named: true,                  // is named
+    fields: {
+      body: {
+        types: [ { type: "statement_block", named: true } ]
+      },
+      ...
+    }
+  },
+  // node type alias
+  declaration: {
+    type: "declaration",
+    subtypes: [
+      { type: "class_declaration", named: true },
+      { type: "function_declaration", named: true },
+    ]
+  },
+  ...
+}
+```
+
+The type information is encoded in a JSON object. Syntax node's static type contains the kind, whether it is named, and the fields of the node.
+`fields` is a map from field name to the type of the field, which encodes the structure of the AST like other parser libraries.
+
+Tree-sitter also provides alias types where a kind is an alias of a list of other kinds. For example, `declaration` is an alias of `function_declaration`, `class_declaration` and other kinds. The alias type is used to reduce the number of kinds in the static type.
+
+We want to both type a node's kind and its fields.
+
+## `SgNode<K>`
+
+`SgNode<M, K>` is the main type in the new API. It is a generic type that represents a node with kind `K`. It is a union of all possible kinds of nodes.
+
+```typescript
+class SgNode<M extends TypesMap, K extends keyof M> {
+  kind: K
+  fields: M[K]['fields'] // for simplicity
+}
+```
+
+
 
 ### `ResolveType<M, T>`
 
-Use type script to resolve type alias
+TreeSitter's type alias is helpful to reduce the generated JSON file size but it is not useful to users because the alias is never directly used as a node's kind nor is used as `kind` in ast-grep rule.
+
+We need to use a type alias to resolve the alias type to its concrete type.
+
+```typescript
+type ResolveType<M, T extends keyof M> =
+  M[T] extends {subtypes: infer S extends {type: string}[] }
+    ? ResolveType<M, S[number]['type']>
+    : T
+```
 
 ### `Kinds<M>`
 
-1. string literal completion with `LowPriorityString`
-2. lenient
+Having a collection of possible AST node kinds is awesome, but it is sometime too clumsy to use a big string literal union type.
+Also, TreeSitter's static type contains a lot of unnamed kinds, which are not useful to users. Including them in the union type is too noisy. We need to allow users to opt-in to use the kind, and fallback to a plain `string` type.
+
+```typescript
+type Kinds<M> = keyof M & LowPriorityString
+type LowPriorityString = string & {}
+```
+
+The above type is a linient string type that is compatible with any string type. But it also uses a well-known trick to take advantage of TypeScript's type priority to prefer the `keyof M` type in completion over the `string & {}` type. To make it more self-explanatory, the `stirng & {}` type is aliased to `LowPriorityString`.
 
 Problem? open-ended union is not well supported in TypeScript
 
 https://github.com/microsoft/TypeScript/issues/33471
 https://github.com/microsoft/TypeScript/issues/26277
 
 
-### Distinguish general `string`ly kinds and specific kinds
+###  Bridging general nodes and specific nodes via `RefineNode`
+
+There are two categories of nodes:
+* general `string`ly typed SgNode
+* precisely typed SgNode
+
+general node is like the untyped old API (but with better completion)
+precisely typed node is a union type of all possible kinds of nodes
+
+The previous general node is typed as `SgNode<M, Kinds<M>>`, the later is typed as `SgNode<M, 'specific_kind'>`.
+
+when it comes to a node that can have several specific kinds, it is better to use a union type of all possible kinds of nodes.
+
+Which kind of union should we use?
 
 Note `SgNode<'expression' | 'type'>` is different from `SgNode<'expression'> | SgNode<'type'>`
+TypeScript has difficulty in narrowing the previous type, because it not safe to assume the former is equivalent to the later.
+However, `SgNode` is covariant in the kind parameter and this means it is okay.
+it is general okay to distribute the type constructor over union type if the parameter is covariant.
+but TypeScript does not support this feature.
 
-ast-grep uses a trick via the type `RefineNode<>` to let you switch between the two
+So ast-grep uses a trick via the type `RefineNode<M, K>` to let you refine the former one to  the later one.
+
+If the uniont type `K` contains a constituent of `string` type, it is equivalent to `SgNode<M, Kinds<M>>`.
+Otherwise, we can refine the node to a union type of all possible kinds of nodes.
+
+```typescript
+type RefineNode<M, K> = string extends K ? SgNode<M, K> :
+  K extends keyof M ? SgNode<M, K> : never // this conditional type unpack the string union to Node union
+```
+it is like biome / rowan's API where you can refine the node to a specific kind.
 
 
 ## Refine Type
 
+Now let's talk about how to refine the general node to a specific node in ast-grep/napi
+
 ### Refine Node, Manually
 
-1.via `sgNode.find<"KIND">`
-2.via `sgNode.is<"KIND">`, One time type narrowing
+Most AST traversal methods in ast-grep now can take a new type parameter to refine the node to a specific kind.
 
-Using the intersting overloading feature of TypeScript
+This is like the `document.querySelector<T>` method in the DOM API. It returns a general `Element` type, but you can refine it to a specific type like `HTMLDivElement` by providing generic argument.
+
+For example `sgNode.find<"KIND">()`
+
+This uses the intersting overloading feature of TypeScript
 
 ```typescript
 interface NodeMethod<K> {
@@ -101,14 +233,37 @@ interface NodeMethod<K> {
 }
 ```
 
+If no type is provided, it returns a general node. If a type is provided, it returns a specific node.
+
+ another way to do runtime checking is via `sgNode.is("kind")`, One time type narrowing
+
+```typescript
+if (sgNode.is("function_declaration")) {
+  sgNode.kind // narrow to 'function_declaration'
+}
+```
+
 ### Refine Node, Automatically
 
-`sgNode.field("kind")` will
+The key feature of the new API is to automatically refine the node to a specific kind when the user gives more type information.
+
+This is done by using the `field` method
+
+`sgNode.field("kind")` will automatically check the field name and its corresponding types in the static type, and refine the node to the specific kind.
 
 
 ### Exhaustive Checking via `sgNode.kindToRefine`
 
-Only available for node with specific kinds
+Why do we need the `kindToRefine` property given that we already have a `kind()` method?
+
+TypeScript cannot narrow type via a method call. It can only narrow type via a property access.
+
+Also `kindToRefine` is a getter under the hood powered by napi. It is less efficient thant JavaScript's object property access.
+Actually, it will call Rust function from JavaScript, which is as expensive as the `kind()` method.
+
+To bring user's awareness to this performance implication and to make a backward compatible API change, we introduce the `kindToRefine` property.
+
+It is mostly useful for a union type of nodes with specific kinds
 
 ```typescript
 const func: SgNode<'function_declaration'> | SgNode<'arrow_function'>
@@ -127,8 +282,28 @@ switch (func.kindToRefine) {
 
 ## Confine Types
 
+Be austere of type level programming.
+
+### Prune unnamed kinds
+For example `+`/`-`/`*`/`/` is too noisy for a general AST library
+
+This is also the reason why we need to include `string` in the `Kinds`.
+
+### Opt-in refinement for better compile time performance
+
+The new API is designed to provide a better type checking and completion experience to the user. But it comes with a cost of performance. The more type information the user provides, the slower the compile time.
+
+```typescript
+import { parse } from '@ast-grep/napi'
+import TS from '@ast-grep/napi/lang/TypeScript'
+const untyped = parse(Lang.TypeScript, code)
+const typed = parse<TS>(Lang.TypeScript, code)
+```
+
 ### Typed Rule!
 
+The last feature worth mentioning is the typed rule! You can even type the `kind` in rule JSON!
+
 ```typescript
 sgNode.find({
   rule: {
@@ -138,15 +313,11 @@ sgNode.find({
 })
 ```
 
-
-### Opt-in refinement for better compile time performance
-
 ## Ending
 
 I'm very thrilled to see the future of AST manipulation in TypeScript.
 This feature enables users to switch freely between untyped AST and typed AST.
 
-it is like biome / rowan
 
 https://x.com/hd_nvim/status/1868453729940500924
 There are very few devs that understands Rust deeply enough and compiler deeply enough that also care about TypeScript in web dev enough to build something for web devs in Rust
