Precise code search and transformation for Raku
Raku’s RakuAST opens up a powerful way to analyze and transform code by working directly with its Abstract Syntax Tree (AST). ASTQuery builds on that by offering a compact, expressive query language to find the nodes you care about,
capture them, and even drive compile-time rewrites.
This guide explains:
- What ASTs are and why they matter
- What RakuAST provides
- How to search ASTs and build macro-like passes
- How ASTQuery’s selector language works
- Practical examples: queries, captures, attribute filters, and rewrites
ASTs, Briefly
- What: An AST is a structured, typed tree that represents your code after parsing (e.g., “call”, “operator application”, “variable”).
- Why: Compilers, linters, and refactoring tools operate on ASTs because they capture code semantics, not just text. This enables robust search and safe transformations.
What Is RakuAST?
- Raku’s AST: RakuAST is the new, structured representation of Raku code. It exposes node types like
RakuAST::Call,RakuAST::ApplyInfix,RakuAST::Var, and more. - Access:
my $ast = $code.AST;for strings, or$*CUfor the current compilation unit in aCHECKphaser. - Status: RakuAST is still experimental. Some node fields may not be
rwon your Rakudo; rebuild/replace enclosing nodes when needed.
Why Search ASTs?
- Beyond grep: Find “function calls with an Int somewhere under args”, not just text matches.
- Safer refactors: Target particular node shapes and attributes to avoid false positives.
- Automated upgrades: Write codemods that transform legacy patterns into new APIs.
Macro-Like Passes (Compiler-Time Rewrites)
- Use a
CHECKphaser withuse experimental :rakuast;to inspect/modify$*CUbefore runtime. - Typical flow:
1) Fetch
$*CU2) Query nodes with ASTQuery 3) Mutate nodes (or rebuild if fields aren’trw) (How mutable RakuAST needs to be is still being discussed)
Example: Add '!!!' at the end of every say call.
use experimental :rakuast;
use ASTQuery;
CHECK {
my $ast = $*CU;
for $ast.&ast-query(Q|.call#say|).list {
.args.push: RakuAST::StrLiteral.new: "!!!";
}
}
say "some text"; # prints "some text!!!"
ASTQuery, at a Glance
• Query language: Describe node kinds, relationships (child/descendant/ancestor), and attributes succinctly.
• Captures: Name nodes you want to retrieve with $name.
• Functions: Reusable predicates referenced with &name.
• Programmatic API: ast-query and ast-matcher.
• CLI: Query files and print results in a readable form.
Quickstart
use ASTQuery;
my $code = q:to/CODE/;
sub f($x) { }
f 42;
say 1 * 3;
CODE
my $ast = $code.AST;
# Find Apply operator nodes where left=1 and right=3
my $ops = $ast.&ast-query('.apply-operator[left=1, right=3]');
say $ops.list;
# Find calls that have an Int somewhere under args
my $calls = $ast.&ast-query('&is-call[args=>>>.int]');
say $calls.list;
Selector Language
Node description format:
RakuAST::Class::Name.group#id[attr1, attr2=attrvalue]$name&function
Components:
• RakuAST::Class::Name: Optional full class name.
• .group: Optional node group (alias to multiple classes).
• #id: Optional id value compared against the node’s id field (per-type mapping).
• [attributes]: Optional attribute matchers (see below).
• $name: Optional capture name (one per node part).
• &function: Optional function matcher (compose with AND when multiple).
Relationship operators:
• >: Left has right as a child.
• >>: Left has right as a descendant, skipping only ignorable nodes.
• >>>: Left has right as a descendant (any nodes allowed between).
• <: Right is the parent of left.
• <<: Right is an ancestor of left, skipping only ignorable nodes.
• <<<: Right is an ancestor of left (any nodes allowed between).
• Note: The space operator is no longer used.
Ignorable nodes (skipped by >>/<<):
• RakuAST::Block, RakuAST::Blockoid, RakuAST::StatementList,
RakuAST::Statement::Expression, RakuAST::ArgList
Attribute relation operators (start traversal from attribute value when it is a RakuAST node):
• [attr=>MATCH] child
• [attr=>>MATCH] descendant via ignorable nodes
• [attr=>>>MATCH] descendant (any nodes)
Attribute value operators (compare against a literal, identifier, or regex literal):
• [attr~=value] contains (substring) or regex match
• [attr^=value] starts-with
• [attr$=value] ends-with
• [attr*=/regex/] regex literal
Notes:
• When an attribute holds a RakuAST node, the matcher walks nested nodes via configured id fields to reach a comparable
leaf (e.g., .call[name] → Name’s identifier).
• Non-existent attributes never match.
Captures:
• Append $name to capture the current node part, e.g., .call#say$call then access with $match<call>.
Functions:
• Use &name to apply reusable predicates; multiple functions compose with AND.
Built-in Groups and Functions
Common groups:
• .call → RakuAST::Call
• .apply-operator → RakuAST::ApplyInfix|ApplyListInfix|ApplyPostfix|Ternary
• .operator → RakuAST::Infixish|Prefixish|Postfixish
• .conditional → RakuAST::Statement::IfWith|Unless|Without
• .variable, .variable-usage, .variable-declaration
• .statement, .expression, .int, .str, .ignorable
Built-in &functions:
• &is-call, &is-operator, &is-apply-operator
• &is-assignment, &is-conditional
• &has-var, &has-call, &has-int
See REFERENCE.md for the full, authoritative list of groups, functions, and id fields.
ID Fields (#id) and How Matching Works
• Each RakuAST type maps to an “id field” used for #id comparisons (e.g., RakuAST::Call uses name, RakuAST::Infix uses
operator, literals use value).
• When comparing attributes whose value is a RakuAST node, ASTQuery walks down by id fields until reaching a leaf value
to compare.
• For variable declarations, bare ids strip sigils for comparison:
• .variable-declaration#x matches my $x, even though the declaration’s name includes the sigil internally (if needed, you can always use [name="$x"]).
Examples
Find specific infix applications (left=1, right=3):
my $code = q{
for ^10 {
if $_ %% 2 {
say 1 * 3;
}
}
};
my $ast = $code.AST;
my $result = $ast.&ast-query: Q|.apply-operator[left=1, right=3]|;
# ast-query returns a ASTQuery::Match object
say $result.list;
If you print the object itself, instead of getting the list of matched nodes, it will print something like this:
Use ancestor operator <<< with captures:
my $result = $ast.&ast-query('RakuAST::Infix <<< .conditional$cond .int#2$int');
say $result.list; # infix nodes
say $result.hash; # captured 'cond' and 'int'
```
Parent operator `<` and capturing:
```raku
my $result = $ast.&ast-query('RakuAST::Infix < .apply-operator[right=2]$op');
say $result<op>; # ApplyInfix nodes with right=2
```
Descendant operator `>>>` and capturing a variable:
```raku
my $result = $ast.&ast-query('.call >>> RakuAST::Var$var');
say $result.list; # call nodes
say $result.hash; # captured 'var'
```
Attribute relation traversal (from attribute node):
```raku
# Calls that have an Int somewhere under args:
my $calls = $ast.&ast-query('&is-call[args=>>>.int]');
```
Attribute value operators:
```raku
# Calls whose name contains "sa" (e.g., say)
my $q1 = $ast.&ast-query('.call[name~= "sa"]');
# Calls whose name starts with "s"
my $q2 = $ast.&ast-query('.call[name^= "s"]');
# Calls whose name ends with "y"
my $q3 = $ast.&ast-query('.call[name$= "y"]');
# Calls whose name matches /sa.*/
my $q4 = $ast.&ast-query('.call[name*=/sa.*/]');
Capturing and retrieving nodes:
my $m = $ast.&ast-query('.call#say$call');
my $call-node = $m<call>;
my @matched = $m.list;
```
## Reusable Function Matchers
Register a function and use it via &name:
• From a compiled matcher:
```raku
my $m = ast-matcher('.call#f');
new-function('&f-call', $m);
$ast.&ast-query('&f-call');
```
• From a callable:
```raku
new-function('&single-argument-call' => -> $n {
$n.^name.starts-with('RakuAST::Call')
&& $n.args.defined
&& $n.args.args.defined
&& $n.args.args.elems == 1
});
$ast.&ast-query('&single-argument-call');
```
• From a selector string:
```raku
new-function('&var-decl' => '.variable-declaration');
$ast.&ast-query('&var-decl');
```
## Programmatic API
• `ast-query($ast, Str $selector) / ast-query($ast, $matcher)`: Run a query and get an ASTQuery::Match (acts like
Positional + Associative).
• `ast-matcher(Str $selector)`: Compile a selector once and reuse it.
• `new-function($name, $callable|$matcher|$selector)`: Register &name.
• `add-ast-group($name, @classes)` / `add-to-ast-group($name, *@classes)`: Define/extend group aliases.
• `set-ast-id($class, $id-method)`: Configure which attribute is used as the id for #id and nested value matching.
## CLI Usage
• Run against a directory or single file:
• `ast-query.raku 'SELECTOR' [path]`
• If path is omitted, it scans the current directory recursively.
• Extensions scanned: raku, rakumod, rakutest, rakuconfig, p6, pl6, pm6.
• Example:
• `ast-query.raku '.call#say >>> .int' lib/`
## Debugging Selectors
• Set ASTQUERY_DEBUG=1 to print a colored tree of matcher decisions, including deparsed node snippets and pass/fail per
validator step. This helps understand why a node matched—or didn’t.
## Notes and Caveats
• RakuAST is experimental. It's still being discussed how mutable it will be.
• Regex flags in /.../ literals aren’t supported in attribute value operators yet.
• The old “space operator” for relationships is deprecated; use the explicit operators (`>`, `>>`, `>>>`, `<`, `<<`, `<<<`).
## Conclusion
ASTQuery lets you describe meaningful shapes in RakuAST—calls, operators, variables, and more—compose those
descriptions, capture the nodes you want, and apply them to everything from precise code search to automated
compiler-time refactorings. It’s a compact tool for robust code understanding and transformation.
• Repo: https://github.com/FCO/ASTQuery
• See [REFERENCE.md](https://github.com/FCO/ASTQuery/blob/main/REFERENCE.md) for the complete catalog of groups, built-in functions, and id fields.
Top comments (0)