DEV Community

Cover image for Advanced mq Techniques: From Simple Queries to Complex Workflows
Takahiro Sato
Takahiro Sato

Posted on

Advanced mq Techniques: From Simple Queries to Complex Workflows

#markdown #cli #ai #llm

Unleash the full power of mq for professional Markdown processing

Beyond Basic Queries

In my previous article, we covered the basics of mq. Now let's dive into advanced techniques that make mq indispensable for professional workflows.

Advanced Selectors and Filtering

Conditional Logic

mq supports sophisticated conditional logic:

# Extract different content based on header level
mq 'let text = to_text(self)
    | if (is_h1()):
      s"# Table of Contents for ${text}"
    elif (is_h2()):
      s"## ${text}"
    else:
      None' documentation.md
Enter fullscreen mode Exit fullscreen mode

Complex Filtering 

# Find code blocks that contain specific patterns
mq '.code | select(or(contains("async"), contains("function")))' api-docs.md

# Extract headers that don't contain "deprecated"
mq '.h | select(not(contains("deprecated")))' changelog.md

# Find lists with more than 3 items
mq '.[] | select(len() > 3)' requirements.md
Enter fullscreen mode Exit fullscreen mode

String Interpolation and Variables

Using Variables 

# Define reusable values
mq -I null 'let project_name = "my-awesome-project" |
    let version = "1.0.0" |
    s"# ${project_name} v${version}\n\nWelcome to ${project_name}!"'
Enter fullscreen mode Exit fullscreen mode

Dynamic Content Generation 

# Generate badges dynamically
mq -I null 'let lang = "rust" |
    let repo = "harehare/mq" |
    s"[![${lang}](https://img.shields.io/badge/language-${lang}-orange.svg)](https://github.com/${repo})"'
Enter fullscreen mode Exit fullscreen mode

Working with Multiple Files

Batch Processing 

# Process all markdown files in a directory
mq '.h1 | to_text()' docs/**/*.md

# Combine multiple files with separators
mq -S 's"\n---\n# ${__FILE__}\n---\n"' 'identity()' docs/*.md

# Extract code examples from all files
mq '.code("javascript") | to_text()' tutorials/**/*.md
Enter fullscreen mode Exit fullscreen mode

File-Specific Processing 

# Add file context to extracted content
mq 'let filename = __FILE__
    | let text = to_text(self)
    | .code
    | s"From ${filename}:\n${text}\n"' examples/*.md
Enter fullscreen mode Exit fullscreen mode

Content Transformation Workflows

Documentation Processing 

# Generate API documentation index
mq 'select(.h2) |
    let title = to_text(self) |
    let anchor = to_link(s"#${title}", title, "") |
    to_md_list(anchor, 1)' api/*.md > api-index.md
Enter fullscreen mode Exit fullscreen mode

Content Migration 

# Convert HTML to Markdown and extract specific sections
mq 'select(or(.h1, .h2))
  | let text = to_text(self)
  | if (is_h1()):
      s"# ${text}"
    else:
      s"## ${text}"' legacy-docs.html
Enter fullscreen mode Exit fullscreen mode

LLM and AI Workflows

Prompt Engineering 

# Extract code examples for training data
mq '.code |
    let code_content = to_text(self) |
    s"Code:\n${code_content}\n---"' tutorials/*.md
Enter fullscreen mode Exit fullscreen mode

Content Preparation 

# Prepare content for embedding
mq 'select(or(.h, .text)) |
    let type = to_md_name() |
    let content = to_text(self) |
    s"${type}: ${content}"' knowledge-base/*.md
Enter fullscreen mode Exit fullscreen mode

Advanced Table Processing

CSV to Markdown 

# Convert CSV data to markdown tables
mq 'nodes | csv2table_with_header()' data.csv
Enter fullscreen mode Exit fullscreen mode

Custom Functions and Modules

Creating Reusable Logic

Create a file custom.mq:

def format_api_endpoint(method, path, description):
  s"**${method}** `${path}`\n\n${description}\n";

def extract_endpoints():
  select(or(select(starts_with("GET"), starts_with("POST"))));
Enter fullscreen mode Exit fullscreen mode

Use it in your queries:

mq 'include "custom" | .code | extract_endpoints() | format_api_endpoint("GET", self, "API endpoint")' api-docs.md
Enter fullscreen mode Exit fullscreen mode

Performance Optimization

Parallel Processing 

# Process large numbers of files in parallel
mq -P 5 '.h1 | to_text()' docs/**/*.md
Enter fullscreen mode Exit fullscreen mode

Streaming Processing 

# Process large files without loading everything into memory
mq --unbuffered '.code | to_text()' large-documentation.md
Enter fullscreen mode Exit fullscreen mode

Integration Patterns

CI/CD Integration 

# GitHub Actions example
- name: Extract API Changes
  run: |
    mq 'select(and(.h2, contains("API"))) | to_text()' CHANGELOG.md > api-changes.txt
Enter fullscreen mode Exit fullscreen mode

Build Tool Integration 

# Extract version from changelog
VERSION=$(mq 'select(.h2) | to_text() | match("v?([0-9.]+)")' CHANGELOG.md)
echo "Building version: $VERSION"
Enter fullscreen mode Exit fullscreen mode

Real-World Use Cases

Documentation Maintenance 

# Find outdated documentation sections
mq 'select(or(contains("TODO"), contains("FIXME")))
    | s"File: ${__FILE__}\nIssue: ${to_text(self)}"' docs/**/*.md
Enter fullscreen mode Exit fullscreen mode

Content Audit 

# Check for broken internal links
mq '.link | select(starts_with("#")) |
    let target = slice(1) |
    s"Link target: ${target}"' *.md
Enter fullscreen mode Exit fullscreen mode

Conclusion

mq excels at complex Markdown processing tasks that would be difficult or impossible with traditional text processing tools. By mastering these advanced techniques, you can build sophisticated documentation workflows, automate content processing, and integrate Markdown manipulation into your development pipeline.

The combination of structural understanding, powerful filtering, and transformation capabilities makes mq an useful tool for any developer working with Markdown at scale.

Support

Resources

Top comments (0)