feat(cli): integrate dynamiclink for tree-sitter to reduce CLI size and improve language support #1580

amirabbas-gh · 2025-07-05T00:35:56Z

📚 Description

This PR introduces the dynamiclink mechanism for tree-sitter language support in the ast-grep Codemods functionality. By using dynamic linking instead of bundling static binaries, we significantly reduce the CLI binary size—approximately 50% smaller—while also enabling easier updates and broader language support.

Key changes:

feat: Added dynamiclink logic
feat: Introduced support for tsx, css, html, and kotlin
refactor: Migrated native YAML parser to dynamiclink method

This change not only improves performance and maintainability but also makes future language support integration easier and cleaner.

🧪 Test Plan

All existing tests execution pass successfully.
Verified language-specific tree-sitter parsing via CLI for supported languages.
Manually tested the CLI binary to confirm reduced size and functional correctness.

vercel · 2025-07-05T00:36:01Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
codemod	⬜️ Ignored (Inspect)	Visit Preview		Jul 21, 2025 6:16pm

pkg-pr-new · 2025-07-05T00:37:23Z

Open in StackBlitz

npm i https://pkg.pr.new/codemod@1580

commit: 2fd7aa3

mohebifar · 2025-07-05T00:37:41Z

Cargo.toml

@@ -30,8 +30,8 @@ butterflow-runners = { path = "crates/runners" }
 butterflow-scheduler = { path = "crates/scheduler" }
 codemod-sandbox = { path = "crates/codemod-sandbox" }

+ast-grep-language = "0.38.5"


Do we even need ast-grep-language anymore?

mohebifar · 2025-07-05T00:39:07Z

crates/core/tests/engine_tests.rs

@@ -1544,6 +1544,8 @@ message: "Found console.log statement"
        )
        .await;

+    println!("result 123: {result:?}");


mohebifar · 2025-07-05T00:40:12Z

crates/codemod-sandbox/Cargo.toml

  "llrt_modules",
  "rquickjs-git",
  "rquickjs-git/full-async",
  "tokio",
  "serde_yaml",
+  "ast-grep-language",


Is it needed?

mohebifar · 2025-07-05T00:40:46Z

crates/codemod-sandbox/Cargo.toml

+dirs.workspace = true
+reqwest.workspace = true
+ast-grep-language = { workspace = true, default-features = true, optional = true }
+futures.workspace = true


Is futures used in codemod-sandbox?

Copilot

Pull Request Overview

This PR introduces a dynamic linking mechanism for tree-sitter languages in the ast-grep CLI, transitioning from statically bundled parsers to dynamically loaded ones. This change reduces the CLI binary size by approximately 50% while improving maintainability and enabling easier language support expansion.

Implements a new dynamiclink system for tree-sitter parsers that downloads and caches language libraries on-demand
Adds support for four new languages: TSX, CSS, HTML, and Kotlin
Migrates the native YAML parser implementation to use the new dynamic linking approach

Reviewed Changes

Copilot reviewed 34 out of 35 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
crates/ast-grep-codemod-dynamic-lang/	New crate implementing dynamic language loading with runtime parser registration
crates/codemod-sandbox/src/tree_sitter/mod.rs	Core tree-sitter dynamic loading logic with S3-based parser downloads
crates/core/src/engine.rs	Updated to use async AST grep execution and new SupportedLanguage enum
crates/codemod-sandbox/src/ast_grep/	Migrated from static SupportLang to dynamic DynamicLang throughout
crates/cli/src/templates/	Added template files for new supported languages (TSX, CSS, HTML, Kotlin)
crates/core/tests/engine_tests.rs	Added serial test execution to prevent race conditions

Copilot · 2025-07-10T16:36:36Z

crates/codemod-sandbox/src/sandbox/engine/language_data.rs

            vec![".ts", ".mts", ".cts", ".js", ".mjs", ".cjs"],
        );
        map.insert(
-            Tsx,
+            "typescript",
            vec![".tsx", ".jsx", ".ts", ".js", ".mjs", ".cjs", ".mts", ".cts"],
        );


Two different language configurations are mapped to the same "typescript" key on lines 13 and 17. This will cause the first mapping to be overwritten, losing the TypeScript-specific extensions.

Copilot · 2025-07-10T16:36:36Z

crates/codemod-sandbox/src/tree_sitter/mod.rs

+            extensions: lang.extensions.iter().map(|s| s.to_string()).collect(),
+        })
+        .collect();
+


The unsafe block lacks documentation explaining why unsafe operations are necessary and what safety invariants are maintained.

Suggested change

// SAFETY: The `DynamicLang::register` function requires unsafe because it performs

// operations that bypass Rust's safety guarantees. We ensure safety by:

// 1. Providing valid `registrations` data, which is constructed from trusted inputs

// (the `languages` parameter and the `ReadyLang` struct).

// 2. Ensuring that the `lib_path` and other fields in `registrations` are valid and

// point to existing files or directories.

// 3. Verifying that the `DynamicLang::register` function is used as intended and

// does not cause undefined behavior.

Copilot · 2025-07-10T16:36:36Z

crates/codemod-sandbox/src/tree_sitter/mod.rs

+                std::fs::create_dir_all(parent)
+                    .map_err(|e| format!("Failed to create directory: {e}"))?;
+            }
+            let url = format!("https://tree-sitter-parsers.s3.us-east-1.amazonaws.com/tree-sitter/parsers/tree-sitter-{language}/latest/{os}-{arch}.{extension}");


The hardcoded S3 URL should be configurable to allow for different hosting providers or mirror sites. Consider extracting this as a constant or configuration parameter.

Suggested change

let url = format!("https://tree-sitter-parsers.s3.us-east-1.amazonaws.com/tree-sitter/parsers/tree-sitter-{language}/latest/{os}-{arch}.{extension}");

let base_url = std::env::var("TREE_SITTER_BASE_URL").unwrap_or_else(|_| "https://tree-sitter-parsers.s3.us-east-1.amazonaws.com".to_string());

let url = format!("{base_url}/tree-sitter/parsers/tree-sitter-{language}/latest/{os}-{arch}.{extension}");

Copilot · 2025-07-10T16:36:37Z

crates/ast-grep-codemod-dynamic-lang/src/lib.rs

+static mut DYNAMIC_LANG: Vec<Inner> = vec![];
+static mut LANG_INDEX: Vec<(String, u32)> = vec![];


Global mutable static variables are not thread-safe and can lead to data races. Consider using thread-safe alternatives like Mutex<Vec<...>> or RwLock<Vec<...>>.

Suggested change

static mut DYNAMIC_LANG: Vec<Inner> = vec![];

static mut LANG_INDEX: Vec<(String, u32)> = vec![];

static DYNAMIC_LANG: LazyLock<RwLock<Vec<Inner>>> = LazyLock::new(|| RwLock::new(vec![]));

static LANG_INDEX: LazyLock<RwLock<Vec<(String, u32)>>> = LazyLock::new(|| RwLock::new(vec![]));

Copilot · 2025-07-10T16:36:37Z

crates/ast-grep-codemod-dynamic-lang/src/lib.rs

+}
+
+// both use vec since lang will be small
+static mut DYNAMIC_LANG: Vec<Inner> = vec![];


Global mutable static variables are not thread-safe and can lead to data races. Consider using thread-safe alternatives like Mutex<Vec<...>> or RwLock<Vec<...>>.

Copilot · 2025-07-10T16:36:37Z

crates/codemod-sandbox/src/tree_sitter/mod.rs

+            let response = reqwest::get(url)
+                .await
+                .map_err(|e| format!("Failed to download: {e}"))?;
+            let body = response
+                .bytes()
+                .await
+                .map_err(|e| format!("Failed to read response: {e}"))?;


Network requests should include timeout configurations and better error handling. The current error message from reqwest may not provide sufficient context for debugging download failures.

Suggested change

let response = reqwest::get(url)

.await

.map_err(|e| format!("Failed to download: {e}"))?;

let body = response

.bytes()

.await

.map_err(|e| format!("Failed to read response: {e}"))?;

let client = reqwest::Client::builder()

.timeout(std::time::Duration::from_secs(30))

.build()

.map_err(|e| format!("Failed to build HTTP client: {e}"))?;

let response = client

.get(&url)

.send()

.await

.map_err(|e| format!("Failed to download from {url}: {e}"))?;

let body = response

.bytes()

.await

.map_err(|e| format!("Failed to read response from {url}: {e}"))?;

Copilot · 2025-07-10T16:36:38Z

crates/core/src/engine.rs

-            parsed_lang
+        if let Some(lang_str) = &js_ast_grep.language {
+            config = config.with_language(
+                SupportedLanguage::from_str(lang_str).unwrap_or(SupportedLanguage::Typescript),


Using unwrap_or to silently default to TypeScript when language parsing fails can mask configuration errors. Consider logging a warning when the fallback is used.

Suggested change

SupportedLanguage::from_str(lang_str).unwrap_or(SupportedLanguage::Typescript),

SupportedLanguage::from_str(lang_str).unwrap_or_else(|_| {

warn!("Failed to parse language '{}', falling back to TypeScript.", lang_str);

SupportedLanguage::Typescript

}),

mohebifar · 2025-07-10T16:46:09Z

@amirabbas-gh Copilot left some good comments here. Could you please address these comments? In the meantime, I'll review it.

.github/workflows/rust-ci.yml

mohebifar · 2025-07-17T17:24:55Z

crates/codemod-sandbox/src/tree_sitter/mod.rs

+                    .map_err(|e| format!("Failed to create directory: {e}"))?;
+            }
+            let base_url = std::env::var("TREE_SITTER_BASE_URL").unwrap_or_else(|_| {
+                "https://tree-sitter-parsers.s3.us-east-1.amazonaws.com".to_string()


mohebifar · 2025-07-17T17:26:02Z

crates/codemod-sandbox/Cargo.toml

 llrt_modules = { path = "../../submodules/llrt/llrt_modules", default-features = true, optional = true }
 ignore = { workspace = true, optional = true }
 serde_yaml = { workspace = true, optional = true }
+dirs.workspace = true
+reqwest.workspace = true
+serial_test = "3.2.0"


Why not a dev dependency?

@mohebifar done

…unctionality

…insted if codemod-sandbox

mohebifar reviewed Jul 5, 2025

View reviewed changes

amirabbas-gh changed the title ~~Dynamiclink~~ feat(cli): integrate dynamiclink for tree-sitter to reduce CLI size and improve language support Jul 5, 2025

amirabbas-gh requested a review from mohebifar July 5, 2025 10:58

amirabbas-gh force-pushed the dynamiclink branch 2 times, most recently from bf1d74d to 491dee9 Compare July 9, 2025 09:47

mohebifar requested a review from Copilot July 10, 2025 16:35

Copilot AI reviewed Jul 10, 2025

View reviewed changes

amirabbas-gh force-pushed the dynamiclink branch 2 times, most recently from 50b5cf2 to f5fc84a Compare July 14, 2025 20:12

mohebifar reviewed Jul 17, 2025

View reviewed changes

amirabbas-gh force-pushed the dynamiclink branch 2 times, most recently from 6c4d959 to 5e172b0 Compare July 18, 2025 12:10

amirabbas-gh requested a review from mohebifar July 18, 2025 13:03

amirabbas-gh added 14 commits July 21, 2025 21:39

feat: add dynamiclink logic for tree sitter at js ast-grep codemods f…

81ea8c7

…unctionality

refactor: use dynamiclink method for native yaml parser

3711d25

fix: fix failure test about dynamiclink method

a2fa76c

feat: add tsx, css, html, kotlin support

8646ccd

fix: fix lint and format

995a380

refactor: clean up deps

a05f03a

refactor: customize ast-grep dynamic in own crate

5f60890

refactor: clean up codebase from test prints

2980f22

feat: add tsx langauge

5527a64

refactor: make tree sitter url more clean

0dd9a94

fix: fix data races and make dynamic-lang thread-safe

52e3223

feat: add timeout logic for tree sitter fetch request

8b6df24

feat: add warning when language is not supported

5c8613c

feat: add progress bar for downloading tree sitter

b8945b4

amirabbas-gh added 6 commits July 21, 2025 21:40

fix: fix lint

8cf0cf0

refactor: use static base url for tree sitter

d6ed997

fix: move serial_test into dev deps

e8ca1de

refactor: move download progress bar for tree sitter loader into CLI …

b75231f

…insted if codemod-sandbox

fix: resolve lint issues at ast-grep/native.rs

dfc6564

fix: fix conflicts fixing bugs

2fd7aa3

amirabbas-gh force-pushed the dynamiclink branch from 5e172b0 to 2fd7aa3 Compare July 21, 2025 18:15

+    // SAFETY: The `DynamicLang::register` function requires unsafe because it performs
+    // operations that bypass Rust's safety guarantees. We ensure safety by:
+    // 1. Providing valid `registrations` data, which is constructed from trusted inputs
+    //    (the `languages` parameter and the `ReadyLang` struct).
+    // 2. Ensuring that the `lib_path` and other fields in `registrations` are valid and
+    //    point to existing files or directories.
+    // 3. Verifying that the `DynamicLang::register` function is used as intended and
+    //    does not cause undefined behavior.

	let url = format!("https://tree-sitter-parsers.s3.us-east-1.amazonaws.com/tree-sitter/parsers/tree-sitter-{language}/latest/{os}-{arch}.{extension}");
	let base_url = std::env::var("TREE_SITTER_BASE_URL").unwrap_or_else(\|_\| "https://tree-sitter-parsers.s3.us-east-1.amazonaws.com".to_string());
	let url = format!("{base_url}/tree-sitter/parsers/tree-sitter-{language}/latest/{os}-{arch}.{extension}");

		static mut DYNAMIC_LANG: Vec<Inner> = vec![];
		static mut LANG_INDEX: Vec<(String, u32)> = vec![];

-            let response = reqwest::get(url)
-                .await
-                .map_err(|e| format!("Failed to download: {e}"))?;
-            let body = response
-                .bytes()
-                .await
-                .map_err(|e| format!("Failed to read response: {e}"))?;
+            let client = reqwest::Client::builder()
+                .timeout(std::time::Duration::from_secs(30))
+                .build()
+                .map_err(|e| format!("Failed to build HTTP client: {e}"))?;
+            let response = client
+                .get(&url)
+                .send()
+                .await
+                .map_err(|e| format!("Failed to download from {url}: {e}"))?;
+            let body = response
+                .bytes()
+                .await
+                .map_err(|e| format!("Failed to read response from {url}: {e}"))?;

feat(cli): integrate dynamiclink for tree-sitter to reduce CLI size and improve language support #1580

Are you sure you want to change the base?

feat(cli): integrate dynamiclink for tree-sitter to reduce CLI size and improve language support #1580

Uh oh!

Conversation

amirabbas-gh commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📚 Description

🧪 Test Plan

Uh oh!

vercel bot commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pkg-pr-new bot commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

mohebifar commented Jul 10, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

amirabbas-gh commented Jul 5, 2025 •

edited

Loading

vercel bot commented Jul 5, 2025 •

edited

Loading

pkg-pr-new bot commented Jul 5, 2025 •

edited

Loading