How do you rewrite a pattern recursively? #1566

steinuil · 2024-10-31T13:58:37Z

steinuil
Oct 31, 2024

I wrote two rules to transform Python's Union[T1, T2] -> T1 | T2 and Optional[T] -> T | None.

Here are rules in question.

id: optional
language: Python
rule:
  any:
    - pattern:
        context: 'arg: Optional[$TYPE]'
        selector: generic_type
    - pattern: Optional[$TYPE]
fix: $TYPE | None
---
id: unions
language: Python
rule:
  pattern:
    context: 'a: Union[$$$TYPES]'
    selector: generic_type
rewriters:
  - id: rewrite-unions
    rule:
      pattern: $TYPE
      kind: type
    fix: $TYPE
transform:
  UNIONS:
    rewrite:
      rewriters:
        - rewrite-unions
      source: $$$TYPES
      joinBy: " | "
fix: $UNIONS

Let's say I have a type that looks like this in my codebase:

results:  Optional[Union[List[Union[str, dict]], str]]

What I've observed is that ast-grep will only fix the outermost node that matches a pattern and will not recurse into it, so that to reach a fixpoint I have to run it multiple times.

First run:

Optional[Union[List[Union[str, dict]], str]] -> Union[List[Union[str, dict]], str] | None

Second run:

Union[List[Union[str, dict]], str] | None -> List[Union[str, dict]] | str | None

Third run:

List[Union[str, dict]] | str | None -> List[str | dict] | str | None

Is there any way to tell ast-grep to recurse into an already matched node and apply these fixes recursively without having to run it multiple times?

Answered by HerringtonDarkholme

Feb 6, 2025

Hi, sorry for the late reply.

Is there any way to tell ast-grep to recurse into an already matched node

Yes, it is possible but verbose.

See the link

See the rule

rewriters:
- id: optional
  language: Python
  rule:
    any:
    - pattern:
        context: 'arg: Optional[$TYPE]'
        selector: generic_type
    - pattern: Optional[$TYPE]
  transform:
    NT:
      rewrite: 
        rewriters: [optional, unions]
        source: $TYPE
  fix: $NT | None
- id: unions
  language: Python
  rule:
    pattern:
      context: 'a: Union[$$$TYPES]'
      selector: generic_type
  transform:
    UNIONS:
      rewrite:
        rewriters:
          - rewrite-unions
        source: $$$TYPES
        j…

View full answer

HerringtonDarkholme · 2025-02-06T19:27:08Z

HerringtonDarkholme
Feb 6, 2025
Maintainer

Hi, sorry for the late reply.

Is there any way to tell ast-grep to recurse into an already matched node

Yes, it is possible but verbose.

See the link

See the rule

rewriters:
- id: optional
  language: Python
  rule:
    any:
    - pattern:
        context: 'arg: Optional[$TYPE]'
        selector: generic_type
    - pattern: Optional[$TYPE]
  transform:
    NT:
      rewrite: 
        rewriters: [optional, unions]
        source: $TYPE
  fix: $NT | None
- id: unions
  language: Python
  rule:
    pattern:
      context: 'a: Union[$$$TYPES]'
      selector: generic_type
  transform:
    UNIONS:
      rewrite:
        rewriters:
          - rewrite-unions
        source: $$$TYPES
        joinBy: " | "
  fix: $UNIONS
- id: rewrite-unions
  rule:
    pattern: $TYPE
    kind: type
  transform:
    NT:
      rewrite: 
        rewriters: [optional, unions]
        source: $TYPE
  fix: $NT
rule:
  kind: type
  pattern: $TPE
transform:
  NEW_TYPE:
    rewrite: 
      rewriters: [optional, unions]
      source: $TPE
fix: $NEW_TYPE

The trick here is to apply rewriters to inner types recursively.

2 replies

AClon314 Apr 16, 2025

Hi, I know this is rather a headache problem.

I use this recursive-rewrite-type to rewrite Union.

Problem: Let ast-grep scan --interactive skip if pattern==fix (pattern equals fix) ?

I run ast-grep scan and get lots of help[...] lines, meaning that so much substitutes are happening:

❯ ast-grep scan
app.py
help[recursive-rewrite-types]: 
app.py
help[recursive-rewrite-types]: 
app.py
help[recursive-rewrite-types]: 
@@ -711,7 +711,7 @@
712 712│     instance = None
713 713│ 
714 714│     @classmethod
715    │-    def get(cls, value: Union[Callable, Any] = None):
    715│+    def get(cls, value: Callable | Any = None):
716 716│         if Single.instance is None:
717 717│             if callable(value):
718 718│                 Single.instance = value()
app.py
help[recursive-rewrite-types]: 
app.py
help[recursive-rewrite-types]: 
app.py
help[recursive-rewrite-types]:

So ast-grep scan -i have lots of blank fix and it's hard to skip:

app.py
help[recursive-rewrite-types]: 
Accept change? (Yes[y], No[n], Accept All[a], Quit[q], Edit[e]) n

I try this playground and found that those substitutes happens like TYPE_A → TYPE_A when last one recursive layer.

You can see this in the playground ↘️bottom right corner.

MetaVar Name	Matched Node(s)
NEW_TYPE	str
TPE	Union[str, dict]
❮4/7 match(es)❯

MetaVar Name	Matched Node(s)
NEW_TYPE	str
TPE	str
❮5/7 match(es)❯

HerringtonDarkholme Apr 16, 2025
Maintainer

I recommend using utils to filter out function that does not include Union

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

How do you rewrite a pattern recursively? #1566

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Uh oh!

How do you rewrite a pattern recursively? #1566

Uh oh!

steinuil Oct 31, 2024

Replies: 1 comment · 2 replies

Uh oh!

Uh oh!

HerringtonDarkholme Feb 6, 2025 Maintainer

Uh oh!

Uh oh!

AClon314 Apr 16, 2025

Uh oh!

HerringtonDarkholme Apr 16, 2025 Maintainer

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

steinuil
Oct 31, 2024

Replies: 1 comment 2 replies

HerringtonDarkholme
Feb 6, 2025
Maintainer

HerringtonDarkholme Apr 16, 2025
Maintainer