DEV Community: nchika

A Minimal Go Toolkit for Cleaning, Validating, and Querying CSV/TSV/Excel/Parquet Files

nchika — Wed, 10 Dec 2025 08:20:27 +0000

In many software systems, not all data lives inside a database.
Sometimes it’s stored in structured files such as CSVs, TSVs, or spreadsheets. And in practice, these files are not always clean. Data may be entered manually by someone who isn’t familiar with the format, values may be missing, and rows may contain inconsistencies that break downstream processing.

When the file is large, trying to “fix it in Excel” often leads to crashes — or simply makes it difficult to identify where the invalid records are.

I faced these issues repeatedly in real projects, and eventually decided to build a set of libraries to address them. Since I’m a Go developer, adopting the Python ETL ecosystem was never my first choice.

Instead, I created three small Go libraries that focus on preprocessing, validation, and lightweight analysis of structured file data:

fileprep — preprocessing + field-level validation using struct tags
fileframe — a tiny, immutable DataFrame for filtering and inspection
filesql — run SQL directly on CSV/TSV/LTSV/Excel/Parquet via embedded SQLite

These libraries are independent from one another.
What connects them is the shared use of the io.Reader interface.
By passing data as a familiar io.Reader, you don’t have to think about any custom data structures inside the libraries — you can work with the standard Go primitives you already know.

Now, let me explain the characteristics of each library.

fileprep

Features

Preprocessing: trim, replace, Unicode normalization, type coercion, etc.
Clear error reporting: identifies which row and which column caused the failure
Supports composite validation (cross-column rules)
Works with CSV/TSV/LTSV/Parquet/Excel — anything fed through io.Reader

Example

type User struct {
    Name  string `prep:"trim" validate:"required"`
    Email string `prep:"trim,lowercase"`
    Age   string
}

func main() {
    csvData := `name,email,age
  John Doe  ,JOHN@EXAMPLE.COM,30
Jane Smith,jane@example.com,25
`

    processor := fileprep.NewProcessor(fileprep.FileTypeCSV)
    var users []User

    reader, result, err := processor.Process(strings.NewReader(csvData), &users)
    if err != nil {
        fmt.Printf("Error: %v\n", err)
        return
    }

    fmt.Printf("Processed %d rows, %d valid\n", result.RowCount, result.ValidRowCount)

    for _, user := range users {
        fmt.Printf("Name: %q, Email: %q\n", user.Name, user.Email)
    }

    // reader can be passed directly to filesql
    _ = reader
}

Output:

Processed 2 rows, 2 valid
Name: "John Doe", Email: "john@example.com"
Name: "Jane Smith", Email: "jane@example.com"

fileframe

Features

Immutable DataFrame
Filtering, mapping, grouping
Ideal for “one-time transformations” on small/medium CSV/TSV datasets

Example

// Sample sales data
csvData := `product,amount,category
Apple,100,Fruit
Banana,150,Fruit
Carrot,80,Vegetable
Orange,120,Fruit
Broccoli,90,Vegetable`

df, err := fileframe.NewDataFrame(strings.NewReader(csvData), fileframe.CSV)
if err != nil {
    fmt.Println("Error:", err)
    return
}

fmt.Printf("Total rows: %d\n", df.Len())
fmt.Printf("Columns: %v\n", df.Columns())

// Filter
filtered := df.Filter(func(row map[string]any) bool {
    amount, ok := row["amount"].(int64)
    return ok && amount > 100
})
fmt.Printf("Rows with amount > 100: %d\n", filtered.Len())

// GroupBy + Sum
groupedDf, err := df.GroupBy("category")
if err != nil {
    fmt.Println("Error:", err)
    return
}
grouped, err := groupedDf.Sum("amount")
if err != nil {
    fmt.Println("Error:", err)
    return
}

for _, row := range grouped.ToRecords() {
    fmt.Printf("  %s: %.0f\n", row["category"], row["sum_amount"])
}

Output:

Total rows: 5
Columns: [product amount category]
Rows with amount > 100: 2
Fruit: 370
Vegetable: 170

filesql

filesql is not “SQL on files” in the literal sense.
Internally, it loads your data into a temporary SQLite database, giving you SQL capabilities without requiring you to manage the database yourself.

Example

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    db, err := filesql.OpenContext(ctx, "data.csv")
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    rows, err := db.QueryContext(ctx, "SELECT * FROM data WHERE age > 25")
    if err != nil {
        log.Fatal(err)
    }
    defer rows.Close()

    for rows.Next() {
        var name string
        var age int
        if err := rows.Scan(&name, &age); err != nil {
            log.Fatal(err)
        }
        fmt.Printf("Name: %s, Age: %d\n", name, age)
    }
}

If you're a Go developer who doesn’t need the full Apache or Python ETL ecosystems, these three lightweight libraries may fit your workflow well.

The user base is still small, so there may be undiscovered bugs — feedback and issue reports are very welcome.

Building Tornago: A Go Library for Tor Integration Born from Fraud Prevention Needs

nchika — Sun, 23 Nov 2025 06:39:05 +0000

Introduction

Have you ever needed to integrate Tor into your Go application for privacy-focused features or security research? I recently built Tornago, a lightweight Go wrapper for the Tor command-line tool, and I'd like to share the story behind it and how it works.

This project came from a real-world need: monitoring dark web marketplaces for stolen credit card data as part of fraud prevention work. While Python's torproject/stem is commonly used for Tor integration, I wanted a Go alternative that would provide the production stability and simplicity I was looking for.

Why I Built Tornago

I work in fraud prevention, and there are times when accessing the dark web becomes necessary for legitimate security work. This isn't science fiction—it's a daily reality in cybersecurity.

When exploring tools for this purpose, the natural choice was Python's torproject/stem, the official Tor controller library. It's mature, well-documented, and widely used. However, I encountered a few challenges:

Maintenance concerns: The official documentation states "Stem is mostly unmaintained"
Language preference: I prefer Go's stability and compatibility guarantees for production services
Ecosystem diversity: Having multiple language options strengthens the overall ecosystem

So I built Tornago—not to replace stem, but to offer a Go alternative for those who share similar needs.

Design Philosophy

Tornago follows three core principles:

1. Simple Wrapper, Not a Reimplementation

Tornago doesn't reimplement the Tor protocol. Instead, it's a thin wrapper around the existing tor command-line tool. This approach:

Leverages the security-audited Tor implementation
Reduces the attack surface
Makes maintenance simpler
Follows the same philosophy as Python's stem

2. Minimal Interface

The library provides exactly three core functionalities:

Tor Client: Route HTTP/TCP traffic through Tor's SOCKS5 proxy
Tor Daemon Management: Launch and manage Tor processes programmatically
Tor Server: Create and manage Hidden Services (.onion sites) via ControlPort

Nothing more, nothing less. This minimalism serves two purposes:

Easier to understand and use
Discourages potential misuse by not providing unnecessary convenience features

3. Production-Ready

Zero external dependencies: Standard library only
Cross-platform tested: CI runs on Linux, macOS, Windows, FreeBSD, OpenBSD, NetBSD, DragonFly BSD
Robust error handling: Structured errors with errors.Is/errors.As support
Automatic retries: Built-in exponential backoff for transient failures

How Tor Works: A Quick Primer

Before diving into Tornago's implementation, let's understand how Tor actually works. This knowledge is crucial for using the library effectively.

Onion Routing: Multi-Layer Encryption

Tor (The Onion Router) provides anonymity by routing traffic through multiple encrypted layers. Think of it like sending a letter inside multiple envelopes, where each postal worker can only see the next destination, not the final recipient. (Of course, this analogy isn't perfect—unlike envelopes that can be torn open, Tor uses cryptographic encryption that can't be easily broken.)

Key Security Properties

Layered Encryption (Onion Layers):

Each relay only knows its immediate predecessor and successor
Entry node (Guard) knows your IP but not your destination
Exit node knows your destination but not your IP
Middle node knows neither your IP nor destination

Privacy Guarantees:

Your ISP sees: You connect to a Tor entry node (but not what you're accessing)
Entry node sees: Your IP address (but not your destination)
Middle node sees: Only relay traffic (no source or destination)
Exit node sees: Your destination (but not your real IP)
Target server sees: Exit node's IP (not your real IP)

Important Limitations:

Exit nodes can see unencrypted traffic (always use HTTPS!)
Exit node operators could monitor traffic (but can't trace back to you)
Timing analysis might correlate traffic patterns
Slower than direct connections due to 3-hop routing

What Tornago Does

Tornago simplifies Tor integration by handling the complex parts for you:

1. SOCKS5 Proxy Communication

Automatically routes your HTTP/TCP traffic through Tor's SOCKS5 proxy with:

Automatic connection retries with exponential backoff
Configurable timeouts
Support for both HTTP and raw TCP connections

2. Tor Daemon Lifecycle Management

Handles the complex bootstrapping process:

Launches tor process with proper configuration
Waits for SOCKS5 and ControlPort to become available (polls every 200ms)
Manages graceful shutdown and cleanup
Handles temporary or persistent data directories

3. Hidden Service Creation

Simplifies .onion site hosting via ControlPort:

Creates ED25519-V3 onion addresses
Manages port mappings (e.g., onion port 80 → local port 8080)
Supports persistent private keys
Handles client authorization

Getting Started

Prerequisites

Install Tor:

# Ubuntu/Debian
sudo apt update && sudo apt install tor

# Fedora/RHEL
sudo dnf install tor

# Arch Linux
sudo pacman -S tor

# macOS (Homebrew)
brew install tor

Verify installation:

tor --version

Install Tornago:

go get github.com/nao1215/tornago

Real-World Examples

Example 1: Accessing a Website Through Tor

This is the simplest use case—routing HTTP requests through Tor for privacy:

package main

import (
    "context"
    "fmt"
    "io"
    "log"
    "net/http"
    "time"

    "github.com/nao1215/tornago"
)

func main() {
    // Step 1: Launch Tor daemon
    fmt.Println("Starting Tor daemon...")
    launchCfg, err := tornago.NewTorLaunchConfig(
        tornago.WithTorSocksAddr(":0"),     // Random available port
        tornago.WithTorControlAddr(":0"),   // Random available port
        tornago.WithTorStartupTimeout(60*time.Second),
    )
    if err != nil {
        log.Fatalf("Failed to create launch config: %v", err)
    }

    torProcess, err := tornago.StartTorDaemon(launchCfg)
    if err != nil {
        log.Fatalf("Failed to start Tor daemon: %v", err)
    }
    defer torProcess.Stop()

    fmt.Printf("Tor daemon started successfully!\n")
    fmt.Printf("  SOCKS address: %s\n", torProcess.SocksAddr())
    fmt.Printf("  Control address: %s\n", torProcess.ControlAddr())

    // Step 2: Create Tor client
    clientCfg, err := tornago.NewClientConfig(
        tornago.WithClientSocksAddr(torProcess.SocksAddr()),
        tornago.WithClientRequestTimeout(60*time.Second),
    )
    if err != nil {
        log.Fatalf("Failed to create client config: %v", err)
    }

    client, err := tornago.NewClient(clientCfg)
    if err != nil {
        log.Fatalf("Failed to create client: %v", err)
    }
    defer client.Close()

    // Step 3: Make HTTP request through Tor
    fmt.Println("\nFetching https://example.com through Tor...")
    req, err := http.NewRequestWithContext(
        context.Background(),
        http.MethodGet,
        "https://example.com",
        http.NoBody,
    )
    if err != nil {
        log.Fatalf("Failed to create request: %v", err)
    }

    resp, err := client.Do(req)
    if err != nil {
        log.Fatalf("Request failed: %v", err)
    }
    defer resp.Body.Close()

    fmt.Printf("Status: %s\n", resp.Status)

    body, err := io.ReadAll(io.LimitReader(resp.Body, 500))
    if err != nil {
        log.Fatalf("Failed to read response: %v", err)
    }

    fmt.Printf("\nResponse preview (first 500 bytes):\n%s\n", string(body))
}

Output:

Starting Tor daemon...
Tor daemon started successfully!
  SOCKS address: 127.0.0.1:42715
  Control address: 127.0.0.1:35199

Fetching https://example.com through Tor...
Status: 200 OK

Response preview (first 500 bytes):
<!doctype html><html lang="en"><head><title>Example Domain</title>...

Example 2: Creating a Hidden Service (.onion site)

Here's how to host your own .onion site:

package main

import (
    "context"
    "fmt"
    "log"
    "net"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"

    "github.com/nao1215/tornago"
)

func main() {
    // Step 1: Launch Tor daemon
    launchCfg, _ := tornago.NewTorLaunchConfig(
        tornago.WithTorSocksAddr(":0"),
        tornago.WithTorControlAddr(":0"),
        tornago.WithTorStartupTimeout(60*time.Second),
    )

    torProcess, err := tornago.StartTorDaemon(launchCfg)
    if err != nil {
        log.Fatalf("Failed to start Tor daemon: %v", err)
    }
    defer torProcess.Stop()

    fmt.Printf("Tor daemon started!\n")
    fmt.Printf("  SOCKS address: %s\n", torProcess.SocksAddr())
    fmt.Printf("  Control address: %s\n", torProcess.ControlAddr())

    // Step 2: Start local HTTP server
    localAddr := "127.0.0.1:8080"
    mux := http.NewServeMux()
    mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        html := `<!DOCTYPE html>
<html>
<head>
    <title>Tornago Hidden Service</title>
</head>
<body>
    <h1>🧅 Welcome to Tornago Hidden Service!</h1>
    <p>This is a .onion site hosted with the tornago library.</p>
    <p>Your IP: ` + r.RemoteAddr + `</p>
</body>
</html>`
        w.Header().Set("Content-Type", "text/html; charset=utf-8")
        fmt.Fprint(w, html)
    })

    server := &http.Server{
        Addr:              localAddr,
        Handler:           mux,
        ReadHeaderTimeout: 5 * time.Second,
    }

    lc := net.ListenConfig{}
    listener, err := lc.Listen(context.Background(), "tcp", localAddr)
    if err != nil {
        log.Fatalf("Failed to start HTTP server: %v", err)
    }

    go func() {
        if err := server.Serve(listener); err != nil && err != http.ErrServerClosed {
            log.Fatalf("HTTP server error: %v", err)
        }
    }()

    fmt.Printf("\nLocal HTTP server started on http://%s\n", localAddr)

    // Step 3: Get control authentication
    auth, _, err := tornago.ControlAuthFromTor(torProcess.ControlAddr(), 30*time.Second)
    if err != nil {
        log.Fatalf("Failed to get control auth: %v", err)
    }

    // Step 4: Create ControlClient
    controlClient, err := tornago.NewControlClient(
        torProcess.ControlAddr(),
        auth,
        30*time.Second,
    )
    if err != nil {
        log.Fatalf("Failed to create control client: %v", err)
    }
    defer controlClient.Close()

    if err := controlClient.Authenticate(); err != nil {
        log.Fatalf("Failed to authenticate with Tor: %v", err)
    }

    // Step 5: Create Hidden Service
    hsCfg, err := tornago.NewHiddenServiceConfig(
        tornago.WithHiddenServicePort(80, 8080), // Map onion port 80 to local 8080
    )
    if err != nil {
        log.Fatalf("Failed to create hidden service config: %v", err)
    }

    fmt.Println("\nCreating Hidden Service...")
    hs, err := controlClient.CreateHiddenService(context.Background(), hsCfg)
    if err != nil {
        log.Fatalf("Failed to create hidden service: %v", err)
    }
    defer func() {
        if err := hs.Remove(context.Background()); err != nil {
            log.Printf("Failed to delete hidden service: %v", err)
        }
    }()

    fmt.Printf("\n✅ Hidden Service created successfully!\n")
    fmt.Printf("   Onion Address: http://%s\n", hs.OnionAddress())
    fmt.Printf("   Local Address: http://%s\n", localAddr)
    fmt.Println("\nAccess this through Tor Browser using the onion address above.")
    fmt.Println("Press Ctrl+C to stop...")

    // Wait for interrupt signal
    sigChan := make(chan os.Signal, 1)
    signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM)
    <-sigChan

    fmt.Println("\n\nShutting down...")
}

Output:

Tor daemon started!
  SOCKS address: 127.0.0.1:36065
  Control address: 127.0.0.1:37285

Local HTTP server started on http://127.0.0.1:8080

Creating Hidden Service...

✅ Hidden Service created successfully!
   Onion Address: http://f64ekih3d23wxhdb547wfj7nornjw5nb3ehuu4do45tw2wwmuzhad3yd.onion
   Local Address: http://127.0.0.1:8080

Access this through Tor Browser using the onion address above.
Press Ctrl+C to stop...

Understanding Tor Startup Time

One thing to note: StartTorDaemon() takes a few minutes to complete. This is because Tor needs to:

Connect to the Tor network: Download directory information (relay node lists, public keys)
Build circuits: Select and establish encrypted connections through Entry Guard → Middle → Exit nodes
Complete bootstrap: Wait until SOCKS5 proxy and ControlPort are ready

Tornago polls the ports every 200ms until they become reachable. On the first run, expect longer startup times since there's no cached directory information.

Here's the sequence diagram showing the initialization process:

Cross-Platform Testing

One of Tornago's strengths is comprehensive cross-platform support. Our GitHub Actions CI pipeline runs tests on:

Linux (Ubuntu)
macOS
Windows
FreeBSD
OpenBSD
NetBSD
DragonFly BSD

This ensures that whether you're deploying on AWS Linux, developing on macOS, or running on a BSD-based firewall, Tornago will work consistently.

Ethical Considerations

Let me be very clear: Tor can be misused, and I'm aware of that risk.

That's why Tornago is intentionally designed as a thin wrapper with minimal convenience features. I'm not trying to make it easier to do harmful things—I'm trying to provide a legitimate tool for:

Privacy protection: Journalists, activists, researchers who need anonymity
Security research: Authorized penetration testing, vulnerability research
Fraud prevention: Monitoring dark web marketplaces for stolen data (my use case)

Conclusion

While Tornago has comprehensive test coverage and CI testing across multiple platforms, it hasn't been battle-tested in production environments yet. I've written extensive tests to ensure reliability, but real-world usage always reveals edge cases that tests can't predict.

If you try Tornago in your projects, please share your experience—whether it's bugs, performance issues, or feature requests. Your feedback is invaluable for making this library production-ready.

Feel free to open an issue or start a discussion on GitHub!
Thank you for reading, and I look forward to your feedback!

horcrux: Split files into encrypted fragments for Node & React Native (TypeScript library)

nchika — Tue, 07 Oct 2025 06:22:48 +0000

Horcrux is a TypeScript library for splitting files into encrypted fragments — “horcruxes” — inspired by Harry Potter. Think of it as digital horcruxes for your sensitive files. You can split your files into multiple pieces where only a subset is needed to restore the original — no password required.

The original idea comes from jesseduffield/horcrux — the CLI tool from the lazygit author that slices a file into N shares and restores it when any M shares are present.

I wanted a version that plugs into both Node and React Native projects, so “Yet Another Horcrux” became a library.

Repo: https://github.com/nao1215/horcrux

npm: https://www.npmjs.com/package/@nao1215/horcrux

How to use

The API stays simple: use split and bind for files, splitBuffer and bindHorcruxes if you’re juggling data in memory. Both sides run on Node and React Native.

Use files

import { split, bind, nodeAdapter } from '@nao1215/horcrux';
import { saveHorcruxes } from '@nao1215/horcrux/core/split';

// Split a file into 5 pieces, need 3 to restore
const result = await split('secret.pdf', 5, 3);

// Save horcruxes (creates secret_1_of_5.horcrux, etc.)
const files = await saveHorcruxes(result.horcruxes, './output', nodeAdapter);

// Later: Restore from any 3 horcruxes
await bind(files.slice(0, 3), 'restored_secret.pdf');

Use buffer

import { splitBuffer } from '@nao1215/horcrux';
import { bindHorcruxes } from '@nao1215/horcrux/core/bind';

const data = Buffer.from('Secret message');
const result = await splitBuffer(data, 'message.txt', {
  total: 4,
  threshold: 2
});

const restored = await bindHorcruxes(result.horcruxes.slice(0, 2));
console.log(restored.data.toString()); // 'Secret message'

How it works

Under the hood it’s Shamir’s Secret Sharing (SSS).

First the file is encrypted, then the encryption key is encoded as coefficients of a polynomial. Each share gets the ciphertext plus a polynomial evaluation. Set a threshold t and total n; as soon as t shares show up, Lagrange interpolation reconstructs the polynomial and the key drops out.

Sit at t-1 and you’re stuck with infinite candidates — guessing isn’t happening. Each share isn’t just “n equal slices”; it carries redundant structure to make the threshold trick work while staying useless alone.

Why did I make this?

In Japan, there’s an internet meme: “If I die, destroy my HDD (or SSD).”
My counterpoint: “I’ve split my files with horcrux, so unless you know the restoration procedure, you can’t see what’s inside — no need to destroy anything, right?”

Status & feedback

I’ve tested on both Node and React Native and it works, but the React Native side is less battle-tested — bug reports and feedback are welcome.

If this looks useful, a ⭐️ on GitHub would mean a lot!

filesql: SQL Driver for CSV, TSV, LTSV, Parquet, and Excel Files in Go

nchika — Mon, 01 Sep 2025 13:24:46 +0000

Why I Created FileSQL

I previously built sqly and sqluv - both CLI tools for running SQL on CSV/TSV files. After maintaining these projects, I realized I was duplicating the same file-handling logic across both tools. So I thought: why not extract this functionality into a reusable library?

The key insight was using Go's standard sql.DB interface. Every Go developer knows how to use database/sql - it's the de facto standard for database operations. By implementing this familiar interface, FileSQL becomes instantly usable for anyone who's written database code in Go.

Back when I was building sqly and sqluv, I was constantly dealing with massive CSV files that needed complex transformations. Importing them into a real database was overkill, but processing them with basic tools was painful. I know many developers face this same struggle, so I wanted to create a library that makes this task trivial.

How to Use

Here's a real-world example that shows FileSQL's power:

// Analyze sales data across multiple file formats
db, err := filesql.Open(
    "sales_2024.csv",        // Current year sales
    "products.xlsx",         // Product master in Excel
    "customers.parquet",     // Customer data in Parquet
    "regions.tsv",          // Region mapping
)

// Complex analytical query with multiple JOINs and aggregations
rows, err := db.Query(`
    WITH monthly_sales AS (
        SELECT 
            strftime('%Y-%m', s.date) as month,
            s.product_id,
            s.customer_id,
            s.amount,
            r.region_name
        FROM sales_2024 s
        JOIN regions r ON s.region_code = r.code
        WHERE s.amount > 0
    )
    SELECT 
        ms.month,
        p.category,
        ms.region_name,
        c.customer_segment,
        COUNT(DISTINCT ms.customer_id) as unique_customers,
        SUM(ms.amount) as total_revenue,
        AVG(ms.amount) as avg_transaction
    FROM monthly_sales ms
    JOIN products p ON ms.product_id = p.id
    JOIN customers c ON ms.customer_id = c.id
    GROUP BY ms.month, p.category, ms.region_name, c.customer_segment
    HAVING SUM(ms.amount) > 10000
    ORDER BY ms.month DESC, total_revenue DESC
`)

Key Features

Multiple file formats: CSV, TSV, LTSV, Parquet, Excel (XLSX) - all queryable with SQL
Compression support: Automatically handles .gz, .bz2, .xz, .zst compressed files
Stream processing: Handles gigabyte-sized files efficiently through configurable chunk processing
Flexible input sources: Files, directories, io.Reader, embed.FS - whatever you have
Auto-save: Changes can be persisted back to the original files

Current Phase

Right now I'm deep in the unglamorous work: squashing bugs, improving robustness, and refactoring code for maintainability. Once this foundation is solid, I'll move on to feature enhancements and push toward the v1.0.0 release.

The roadmap has all the details, but the focus is on stability first, features second.

I'll be honest - I originally built this just to peek at a handful of local files. But then I realized people wanted to use it for ETL pipelines and more serious data processing, so I'm evolving FileSQL to meet those needs too.

Join the Journey

I want to build FileSQL with the community, not in isolation. There are many ways to contribute:

Share your data challenges: What weird CSV formats are you dealing with? Send me sample files (anonymized of course)
Suggest use cases: How would you use FileSQL in your projects?
Report bugs: Found something broken? Let me know!
Improve docs: Is something confusing? Help me clarify it
Spread the word: Share on social media, write a blog post, tell your team
Code contributions: Yes, using LLMs for coding is totally fine - I embrace all modern development approaches

Even a simple star on GitHub helps more than you know.

My Life Goals

This might sound dramatic, but FileSQL is part of my bucket list:

Reach 1,000 GitHub stars on a project I created
Get my first GitHub Sponsor

These aren't just vanity metrics for me - they represent validation that I've built something genuinely useful for the community. FileSQL is my best shot at achieving these dreams.

By the way, most of my other bucket list items have nothing to do with development, like 'put an arcade cabinet in my room' and 'display Toraya's "96-piece mini yokan set" on my desk and feel smugly satisfied about it'.

Thanks

Special thanks to everyone who's already starred the repo, reported issues, or sent PRs. Your support makes it worthwhile.

Let's make file processing in Go stupidly simple together.

GitHub: https://github.com/nao1215/filesql

sqluv - simple terminal UI for DBMS & local CSV/TSV/LTSV

nchika — Sat, 08 Mar 2025 14:16:05 +0000

I previously released the sqly command. sqly is a tool that allows you to execute SQL on CSV, TSV, LTSV, JSON, and Excel files, and it provided a custom shell to enhance user-friendliness. I have compiled articles about sqly here.

When expanding sqly's features, I reconsidered the interface and realized that a simpler Text User Interface (TUI) would make it more intuitive. As I envisioned a TUI, I came up with the idea that "it would be even more useful if the same UI could connect to a DBMS."

This thought process led me to develop the sqluv command. The name comes from SQL + LOVE. I chose this name with the hope that sqluv would help people become more familiar with SQL.

sqlv demo

Support OS and DBMS

Windows
macOS
Linux
MySQL
PostgreSQL
SQLite3
SQL Server

Execution Example: Running SQL on CSV/TSV/LTSV

When sqluv receives file paths (multiple files supported) as startup arguments, it automatically loads the files and stores their data in an in-memory SQLite3 database. The table names are derived from the filenames by removing their extensions (e.g., if the file is sample.csv, the table name will be sample).

Due to the nature of writing data into SQLite3, executing an UPDATE statement in SQL does not modify the original local files. In the original sqly, there was a feature to reflect UPDATE results back into the local files, so I plan to port this feature to sqluv at a later time.

Database Connection Feature

If no file paths are specified at startup, sqluv enters DBMS mode.

New: Enter new database connection details and connect.
List: Select from previously saved connection details and connect.

The New Database Connection screen, which appears after pressing the New button, allows users to input connection details. If the connection is successful, the information is saved in ~/.config/sqluv/dbms.yml.

The Saved Connections screen, which appears after pressing the List button, displays a list of previously entered connection names. Users can navigate the list using the arrow keys and press Enter to connect.

Currently, there is no feature to manually delete connection entries. However, if a connection attempt fails, a modal will appear asking whether to delete the failed connection entry.

HOME Screen

The HOME screen consists of:

Sidebar: Displays the list of tables.
Top-right area: SQL input field.
Bottom-right area: SQL execution result display.

Press Ctrl-D or ESC to exit the application, and use TAB or Shift-TAB to switch focus between elements.

After entering an SQL query and pressing the Execute Query button, the table data is displayed. Users can navigate and scroll through the table using the arrow keys or mouse.

Initially, I planned to execute SQL queries with Ctrl-Enter, but due to technical limitations where Ctrl-Enter was mistakenly recognized as a regular Enter, I had to reluctantly introduce the Execute Query button instead.

Conclusion

sqluv is a newly developed command, and I expect there may still be some bugs. Moving forward, my goals are to fix bugs and provide a more user-friendly TUI.

To keep my development motivation high, I’d really appreciate it if you could press the GitHub Star button! Even better, if you’re willing to sponsor me, that would be amazing (I was recently surprised to receive my very first sponsor!). Bug reports are also more than welcome.

Thank you all for reading until the end!

vogen - Value Object Generator in golang

nchika — Sat, 28 Dec 2024 13:04:26 +0000

Introduction

Do you understand Domain-Driven Design (DDD)? I still haven’t been able to understand it completely.

Recently, I’ve been diving into books on Domain-Driven Design (DDD). In DDD, the concept of a Value Object appears. A Value Object primarily has the following characteristics (elements unrelated to the main topic of this article are intentionally omitted):

Immutable
Objects are considered equal when their values are equivalent

To satisfy the above specifications in Golang, the implementation would need to look something like this:

type Person struct {
    name string
}

func NewPerson(name string) Person {
    return Person{name: name}
}

func (o Person) Name() string {
    return o.name
}

func (o Person) Equal(other Person) bool {
    return o.Name() == other.Name() 
}

Frankly speaking, implementing this kind of functionality is a hassle. Additionally, writing unit tests for Getter() or Equal() feels pointless. I often found myself wishing, "If only Golang had something like Kotlin’s value class or data class."

Generates value object code

The nao1215/vogen package is a library that automatically generates Value Object code with New(), Getter, and Equal() methods. The name stands for "Value Object Generator."

With this library, you write metadata for Value Objects in Golang, and based on that metadata, the code is automatically generated. The inspiration for this specification comes from shogo82148/myddlmaker (a library that generates DB DDL from metadata).

Intended Usage

The typical usage involves defining metadata in value_object/gen/main.go and running go generate ./... to generate the value_object/value_object.go file. It is also possible to distribute the output across multiple files.

Below is an example implementation for value_object/gen/main.go.

package main

import (
    "fmt"
    "path/filepath"

    "github.com/nao1215/vogen"
)

//go:generate go run main.go

func main() {
    // Step 1: Create a Vogen instance with custom file path and package name.
    // By default, the file path is "value_objects.go" and the package name is "vo".
    gen, err := vogen.New(
        vogen.WithFilePath(filepath.Join("testdata", "example_output.go")),
        vogen.WithPackageName("vo_example"),
    )
    if err != nil {
        fmt.Printf("Failed to create Vogen instance: %v\n", err)
        return
    }

    // Step 2: Append the ValueObject definition
    if err := gen.AppendValueObjects(
        vogen.ValueObject{
            StructName: "Person",
            Fields: []vogen.Field{
                {Name: "Name", Type: "string", Comments: []string{"Name is the name of the person."}},
                {Name: "Age", Type: "int", Comments: []string{"Age is the age of the person."}},
            },
            Comments: []string{
                "Person is a Value Object to describe the feature of vogen.",
                "This is sample comment.",
            },
        },
        // Use auto generated comments.
        vogen.ValueObject{
            StructName: "Address",
            Fields: []vogen.Field{
                {Name: "City", Type: "string"},
            },
        },
    ); err != nil {
        fmt.Printf("Failed to append ValueObject: %v\n", err)
        return
    }

    // Step 3: Generate the code
    if err := gen.Generate(); err != nil {
        fmt.Printf("Failed to generate code: %v\n", err)
        return
    }
}

In vogen.New(), you can specify the file path and package name for the generated code, but these are optional. If omitted, a value_objects.go file will be generated under the vo package by default.

vogen.ValueObject() corresponds to the metadata. Comments for the struct and its fields are optional. If omitted, the output will include soulless English comments. For types, you can specify Defined Types (user-defined types), but in such cases, you must also provide the module path. Since Defined Types have not been tested yet, I’ve deliberately omitted them from the example code (tests are planned for later).

Auto-Generated Code

Below is the code automatically generated using the sample above:

// Code generated by vogen. DO NOT EDIT.
package vo_example

import (
    "fmt"
)

// Person is a Value Object to describe the feature of vogen.
// This is sample comment.
type Person struct {
    // Name is the name of the person.
    name string
    // Age is the age of the person.
    age int
}

// NewPerson creates a new instance of Person.
func NewPerson(name string, age int) Person {
    return Person{name: name, age: age}
}

// Name returns the name field.
func (o Person) Name() string {
    return o.name
}

// Age returns the age field.
func (o Person) Age() int {
    return o.age
}

// Equal checks if two Person objects are equal.
func (o Person) Equal(other Person) bool {
    return o.Name() == other.Name() && o.Age() == other.Age()
}

// Address represents a value object.
type Address struct {
    city string
}

// NewAddress creates a new instance of Address.
func NewAddress(city string) Address {
    return Address{city: city}
}

// City returns the city field.
func (o Address) City() string {
    return o.city
}

// Equal checks if two Address objects are equal.
func (o Address) Equal(other Address) bool {
    return o.City() == other.City()
}

Why I Wrote This Article

I wanted to know whether functionality like the vogen package would be of interest to Golang users.

nao1215/csv - Read csv with validation in golang

nchika — Sun, 12 May 2024 02:11:14 +0000

What is nao1215/csv package

The nao1215/csv package is a library that performs validation when loading CSV files. Validation is executed based on the rules specified in the struct tags.

The nao1215/csv package was developed with inspiration from go-playground/validator and shogo82148/go-header-csv. I would like to express my gratitude here.

How to use

Please attach the "validate:" tag to your structure and write the validation rules after it. It's crucial that the "order of columns" matches the "order of field definitions" in the structure. The csv package does not automatically adjust the order.

When using csv.Decode, please pass a pointer to a slice of structures tagged with struct tags. The csv package will perform validation based on the struct tags and save the read results to the slice of structures if there are no errors. If there are errors, it will return them as []error.



package main

import (

    "bytes"

    "fmt"

<span class="s">"github.com/nao1215/csv"</span>



)

func main() {

    input := id,name,age

1,Gina,23

a,Yulia,25

3,Den1s,30



    buf := bytes.NewBufferString(input)

    c, err := csv.NewCSV(buf)

    if err != nil {

        panic(err)

    }

<span class="k">type</span> <span class="n">person</span> <span class="k">struct</span> <span class="p">{</span>
    <span class="n">ID</span>   <span class="kt">int</span>    <span class="s">`validate:"numeric"`</span>
    <span class="n">Name</span> <span class="kt">string</span> <span class="s">`validate:"alpha"`</span>
    <span class="n">Age</span>  <span class="kt">int</span>    <span class="s">`validate:"gt=24"`</span>
<span class="p">}</span>
<span class="n">people</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="n">person</span><span class="p">,</span> <span class="m">0</span><span class="p">)</span>

<span class="n">errs</span> <span class="o">:=</span> <span class="n">c</span><span class="o">.</span><span class="n">Decode</span><span class="p">(</span><span class="o">&amp;</span><span class="n">people</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">errs</span><span class="p">)</span> <span class="o">!=</span> <span class="m">0</span> <span class="p">{</span>
    <span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="k">range</span> <span class="n">errs</span> <span class="p">{</span>
        <span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="n">err</span><span class="o">.</span><span class="n">Error</span><span class="p">())</span>
    <span class="p">}</span>
<span class="p">}</span>

<span class="c">// Output:</span>
<span class="c">// line:2 column age: target is not greater than the threshold value: threshold=24.000000, value=23.000000</span>
<span class="c">// line:3 column id: target is not a numeric character: value=a</span>
<span class="c">// line:4 column name: target is not an alphabetic character: value=Den1s</span>



}

Struct tags

You set the validation rules following the "validate:" tag according to the rules in the table below. If you need to set multiple rules, please enumerate them separated by commas.

Validation rule without arguments

Tag Name	Description
boolean	Check whether value is boolean or not.
alpha	Check whether value is alphabetic or not
numeric	Check whether value is numeric or not
alphanumeric	Check whether value is alphanumeric or not
required	Check whether value is empty or not

Validation rule with numeric argument

Tag Name	Description
eq	Check whether value is equal to the specified value. e.g. `validate:"eq=1"`
ne	Check whether value is not equal to the specified value e.g. `validate:"ne=1"`
gt	Check whether value is greater than the specified value e.g. `validate:"gt=1"`
gte	Check whether value is greater than or equal to the specified value e.g. `validate:"gte=1"`
lt	Check whether value is less than the specified value e.g. `validate:"lt=1"`
lte	Check whether value is less than or equal to the specified value e.g. `validate:"lte=1"`
min	Check whether value is greater than or equal to the specified value e.g. `validate:"min=1"`
max	Check whether value is less than or equal to the specified value e.g. `validate:"max=100"`
len	Check whether the length of the value is equal to the specified value e.g. `validate:"len=10"`
oneof	Check whether value is included in the specified values e.g. `validate:"oneof=male female prefer_not_to"`

Why I wrote nao1215/csv

It's the beginning of a long tale of memories.

Have you ever had the task of finding 10 errors within a CSV containing tens of thousands of rows and over 300 columns? I've experienced that several times.

I developed nao1215/sqly(article is here), which allows executing SQL on CSV files, because I found correcting CSVs to be challenging. However, sqly alone was insufficient.

In Japan, it's common to manage master data using Excel. However, it's not practical for apps to directly read data from Excel. Therefore, it's sometimes necessary to convert Excel to CSV and create apps to import the data into multiple database tables. However, there are cases where incorrect data is present in the CSV (it's strange to have incorrect data in master data, isn't it?), which prevents it from being imported into the database.

I despised this task.

I just wanted to make it even a bit easier. I had someone else implement a validation process to check the CSV columns and identify where errors were (I didn't do it myself). However, I believed I could write the validation process more generically.

Over a year has passed since I started contemplating this. To bid farewell to this challenge, I secluded myself in my room for a day and created nao1215/csv. It took over a year to accomplish what could have been done in just one day.

Next work

I plan to increase the variety of validations and implement faster validation processes. As a side note, I am not currently working with CSVs. I spent a day to resolve lingering regrets from past work. Therefore, the development pace of nao1215/csv may seem slow.

If everyone could gift GitHub Stars, I believe the development pace would be faster.

Thank you for reading this article.

simple mermaid builder in golang: support ER/sequence diagram, pie chart

nchika — Sun, 05 May 2024 14:17:02 +0000

Markdown with mermaid

I previously introduced the nao1215/markdown package as a markdown builder for the Go language (previously it was go-spectest/markdown, but the Owner has changed). What syntax do you want when writing markdown? I thought it was important to support mermaid. When we write design documents, we should be able to write not only text but also sequence diagrams and ER diagrams.

I decided to support some mermaid syntax in the markdown package. This article introduces that.

Mermaid sequence diagram syntax

Sample code

package main

import (
    "os"

    "github.com/nao1215/markdown"
    "github.com/nao1215/mermaid/sequence"
)

//go:generate go run main.go

func main() {
    diagram := sequence.NewDiagram(io.Discard).
        Participant("Sophia").
        Participant("David").
        Participant("Subaru").
        LF().
        SyncRequest("Sophia", "David", "Please wake up Subaru").
        SyncResponse("David", "Sophia", "OK").
        LF().
        LoopStart("until Subaru wake up").
        SyncRequest("David", "Subaru", "Wake up!").
        SyncResponse("Subaru", "David", "zzz").
        SyncRequest("David", "Subaru", "Hey!!!").
        BreakStart("if Subaru wake up").
        SyncResponse("Subaru", "David", "......").
        BreakEnd().
        LoopEnd().
        LF().
        SyncResponse("David", "Sophia", "wake up, wake up").
        String()

    markdown.NewMarkdown(os.Stdout).
        H2("Sequence Diagram").
        CodeBlocks(markdown.SyntaxHighlightMermaid, diagram).
        Build()
}

Generated markdown with sequence diagram

## Sequence Diagram


```mermaid
sequenceDiagram
    participant Sophia
    participant David
    participant Subaru

    Sophia->>David: Please wake up Subaru
    David-->>Sophia: OK

    loop until Subaru wake up
    David->>Subaru: Wake up!
    Subaru-->>David: zzz
    David->>Subaru: Hey!!!
    break if Subaru wake up
    Subaru-->>David: ......
    end
    end

    David-->>Sophia: wake up, wake up
```

Mermaid entity relationsip diagram syntax

Sample code

package main

import (
    "os"

    "github.com/nao1215/markdown"
    "github.com/nao1215/markdown/mermaid/er"
)

//go:generate go run main.go

func main() {
    f, err := os.Create("generated.md")
    if err != nil {
        panic(err)
    }

    teachers := er.NewEntity(
        "teachers",
        []*er.Attribute{
            {
                Type:         "int",
                Name:         "id",
                IsPrimaryKey: true,
                IsForeignKey: false,
                IsUniqueKey:  true,
                Comment:      "Teacher ID",
            },
            {
                Type:         "string",
                Name:         "name",
                IsPrimaryKey: false,
                IsForeignKey: false,
                IsUniqueKey:  false,
                Comment:      "Teacher Name",
            },
        },
    )
    students := er.NewEntity(
        "students",
        []*er.Attribute{
            {
                Type:         "int",
                Name:         "id",
                IsPrimaryKey: true,
                IsForeignKey: false,
                IsUniqueKey:  true,
                Comment:      "Student ID",
            },
            {
                Type:         "string",
                Name:         "name",
                IsPrimaryKey: false,
                IsForeignKey: false,
                IsUniqueKey:  false,
                Comment:      "Student Name",
            },
            {
                Type:         "int",
                Name:         "teacher_id",
                IsPrimaryKey: false,
                IsForeignKey: true,
                IsUniqueKey:  true,
                Comment:      "Teacher ID",
            },
        },
    )
    schools := er.NewEntity(
        "schools",
        []*er.Attribute{
            {
                Type:         "int",
                Name:         "id",
                IsPrimaryKey: true,
                IsForeignKey: false,
                IsUniqueKey:  true,
                Comment:      "School ID",
            },
            {
                Type:         "string",
                Name:         "name",
                IsPrimaryKey: false,
                IsForeignKey: false,
                IsUniqueKey:  false,
                Comment:      "School Name",
            },
            {
                Type:         "int",
                Name:         "teacher_id",
                IsPrimaryKey: false,
                IsForeignKey: true,
                IsUniqueKey:  true,
                Comment:      "Teacher ID",
            },
        },
    )

    erString := er.NewDiagram(f).
        Relationship(
            teachers,
            students,
            er.ExactlyOneRelationship, // "||"
            er.ZeroToMoreRelationship, // "}o"
            er.Identifying,            // "--"
            "Teacher has many students",
        ).
        Relationship(
            teachers,
            schools,
            er.OneToMoreRelationship,  // "|}"
            er.ExactlyOneRelationship, // "||"
            er.NonIdentifying,         // ".."
            "School has many teachers",
        ).
        String()

    err = markdown.NewMarkdown(f).
        H2("Entity Relationship Diagram").
        CodeBlocks(markdown.SyntaxHighlightMermaid, erString).
        Build()

    if err != nil {
        panic(err)
    }
}

Generated markdown with entity relationship diagram

## Entity Relationship Diagram


```mermaid
erDiagram
    teachers ||--o{ students : "Teacher has many students"
    teachers }|..|| schools : "School has many teachers"
    schools {
        int id PK,UK "School ID"
        string name  "School Name"
        int teacher_id FK,UK "Teacher ID"
    }
    students {
        int id PK,UK "Student ID"
        string name  "Student Name"
        int teacher_id FK,UK "Teacher ID"
    }
    teachers {
        int id PK,UK "Teacher ID"
        string name  "Teacher Name"
    }
```

Mermaid pie chart syntax

Sample code

package main

import (
    "io"
    "os"

    "github.com/nao1215/markdown"
    "github.com/nao1215/markdown/mermaid/piechart"
)

//go:generate go run main.go

func main() {
    f, err := os.Create("generated.md")
    if err != nil {
        panic(err)
    }

    chart := piechart.NewPieChart(
        io.Discard,
        piechart.WithTitle("mermaid pie chart builder"),
        piechart.WithShowData(true),
    ).
        LabelAndIntValue("A", 10).
        LabelAndFloatValue("B", 20.1).
        LabelAndIntValue("C", 30).
        String()

    err = markdown.NewMarkdown(f).
        H2("Pie Chart").
        CodeBlocks(markdown.SyntaxHighlightMermaid, chart).
        Build()

    if err != nil {
        panic(err)
    }
}

Generated markdown with pie chart diagram

## Pie Chart


```mermaid
%%{init: {"pie": {"textPosition": 0.75}, "themeVariables": {"pieOuterStrokeWidth": "5px"}} }%%
pie showData
    title mermaid pie chart builder
    "A" : 10
    "B" : 20.100000
    "C" : 30
```

Next work

This new markdown package, which incorporates Mermaid support, enables a more diverse range of expressions when writing design documents, including sequence diagrams, ER diagrams, and more, alongside traditional text.

In the future, I will continue to develop to support more Mermaid syntax. If you're interested, please try using the nao1215/markdown package.

Thank you for reading this article.

s3hub - user-friendly AWS S3 management tool with interactive mode

nchika — Sun, 04 Feb 2024 16:08:20 +0000

Why I Created s3hub

When building AWS resources with CloudFormation, I often find myself needing to check the contents of S3 buckets. How do you typically check the contents of S3 buckets? Personally, I'm not fond of the AWS Management Console (GUI), and I find the AWS CLI somewhat cumbersome.

It's a bad habit of mine, but I tend to create tools to solve problems. The tool I've created this time is named s3hub, and its main feature is its interactive mode.

Features

The s3hub command provides following features:

Create a S3 bucket
List S3 buckets
List S3 objects in the S3 bucket
Copy files to S3 bucket
Delete contents from the S3 bucket
Delete the S3 bucket
Interactive mode

How to install

go install github.com/nao1215/rainbow/cmd/s3hub@latest

How to use

The s3hub command allows you to specify a profile as an option, but it is more user-friendly to use the AWS_PROFILE environment variable. S3hub operates without requiring the 's3://' protocol to be added to the bucket name.

Create a bucket(s)

s3hub mb ${YOUR_BUCKET_NAME}

List buckets

s3hub ls

List contents of a bucket

s3hub ls ${YOUR_BUCKET_NAME}

Copy files to a bucket

From local to S3:

s3hub cp ${YOUR_FILE_PATH} s3://${YOUR_BUCKET_NAME}

From S3 to local:

s3hub cp s3://${YOUR_BUCKET_NAME} ${YOUR_FILE_PATH}

Delete a object from a bucket

If you want to delete a specific object, use the following command:

s3hub rm ${CONTENT_PATH_IN_BUCKET}/${S3_KEY}

If you want to delete all contents in a bucket, use the wildcard:

s3hub rm ${YOUR_BUCKET_NAME}/*

Delete a bucket with objects

When the number of S3 objects is large, we parallelize the deletion process to enhance speed.

s3hub rm --recursive ${YOUR_BUCKET_NAME}

s3hub is Part of the Rainbow Project

I use AWS for work, and I plan to develop numerous tools to make development with AWS easier. I thought "rainbow" would be a fitting name for the repository that manages multiple tools. I borrowed it from my favorite band, Ritchie Blackmore's Rainbow (I'm 33 years old, but I love this band).

I plan to continue developing many tools in the future, so if you're interested, please support me with GitHub Star and such. GitHub Stars are a great way to boost developers' motivation for free.

The Rainbow Project incurs costs because it utilizes AWS. For those who are willing to provide financial support, please consider supporting us through GitHub Sponsors. We would be deeply grateful and appreciative.

actions-hottest🚀: GitHub Actions for Commenting on Golang Unit Test Results in Pull Requests

nchika — Sat, 02 Dec 2023 07:46:32 +0000

Make the test results more user-friendly

I previously introduced a feature in the go-spectest/spectest, which extracts error messages from Golang unit test results.

I have now separated that functionality into the nao1215/hottest command.

The 'hottest' command can use the same options as 'go test':

hottest on GitHub Actions

Many people run unit tests using GitHub Actions. Therefore, I created nao1215/actions-hottest to make it easy to execute 'hottest' in GitHub Actions.

Here is an example workflow for running unit tests using 'hottest':

name: SampleTest

on:
  push:

jobs:
  sample_test:
    name: sample test

    strategy:
      matrix:
        platform: [ubuntu-latest]

    runs-on: ${{ matrix.platform }}

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-go@v4
        with:
          go-version: "1"
          check-latest: true

      - name: Download dependencies
        run: go mod download

      - uses: nao1215/actions-hottest@v1
        with:
          # This argument is same as `go test` command.
          args: '-cover -coverpkg=./... -coverprofile=coverage.out ./...'

Here are the logs for when the unit tests succeed and when they fail:

when all test pass
when some test fails

Comments Test Results in Pull Request

I find it inconvenient to check the workflow logs when tests fail. Therefore, I have decided to have 'hottest' leave the test results as comments in the Pull Request.

Here are the comments for when the unit tests succeed and when they fail:

when all test pass
when some test fails

Conclusion

'hottest' is a command that extracts error messages from failed unit tests. You can easily adopt GitHub Actions. Its usage is similar to 'go test,' and there is no need to rewrite the test code.

lease do give 'hottest' a try!! Additionally, your support in the form of a GitHub star would be encouraging for further development.

spectest: 'go test' output only FAIL results and statistics

nchika — Sun, 19 Nov 2023 06:10:20 +0000

Improving Readability of Test Results

Developers in the Golang community execute unit tests using 'go test'. As the number of tests grows, it becomes increasingly challenging to locate information about test cases that have failed within the results of unit tests.

The tool called rakyll/gotest has addressed this issue by adding color to the output of 'go test'.

However, as an alternative solution to this problem, I have created the spectest command that "extracts information about failed unit tests" and outputs statistical information about the tests.

I write down the same content as in the GIF image below.

$ spectest ./...
....................................................................................................................................................................................................................................................................................................................................................................................................
[Error Messages]
 --- FAIL: Test_goldenFile_write (0.00s)
     --- FAIL: Test_goldenFile_write/should_write_data_to_the_golden_file (0.00s)
         file_system_test.go:51: write() file does not exist
 --- FAIL: TestIntervalDuration (0.00s)
     --- FAIL: TestIntervalDuration/get_1[ns]_duration (0.00s)
         time_test.go:55: duration should be 1 second, but 8760h0m0.000000001s

[Test Results]
 - Execution Time: 4.918126564s
 - Total         : 388
 - Passed        : 384
 - Failed        : 4
 - Skipped       : 0

How It Works

spectest can be used with the same options as 'go test'. This is because spectest directly passes the command-line arguments it receives to 'go test'.

spectest reads the output (stdout) of 'go test' in real-time. If a test is successful, it displays a green dot; if it fails, it shows a red dot. After all tests are completed, it searches the results of 'go test' for the "failed test method name" and the "reason for failure" if there are any failed test cases.

（In the parsing process, I'm writing some clumsy code :) Perhaps, a month later, it might be hard to understand.）

Promotion

The go-spectest/spectest project provides commands and libraries related to testing. While the library is specialized for API testing, the commands are designed to be usable by all Golang users (meaning you don't need to use the spectest library).

Ref. spectest - API testing library for Go that generate E2E test result document in markdown
Ref. spectest roadmap

I have a keen interest in testing and aim to create an environment that makes testing easy for developers. Even if the entire project doesn't persist, I continue to develop some of the open-source software (OSS) I've created, hoping it remains useful to everyone.

If you support my efforts, receiving GitHub Stars would be encouraging. I would be delighted if you use spectest and provide feedback.

spectest - API testing library for Go that generate E2E test result document in markdown

nchika — Mon, 06 Nov 2023 14:45:53 +0000

Generate document from end-to-end tests

In recent years, Domain-Driven Design (DDD) has become mainstream, making it easier to implement unit tests for methods in each layer.

However, implementing end-to-end (E2E) tests can be quite labor-intensive. To achieve a return on the effort invested, I am developing go-spectest/spectest.

The spectest libarary/cli (command line interface) generate E2E test results as documents in Markdown format.

Simple usecase for generating E2E test results

You implement unit tests for your API endpoints with spectest.
You run the E2E tests.
The spectest generate Markdown documents with the test results.
When you execute $ spectest index, spectest generates an index markdown for a directory full of markdown file.

Test code example

For example, let's consider an API that returns the health check of a server (GET /v1/health). The code to test the status code and body of this API is as follows:

func TestHealthCheck(t *testing.T) {
    spectest.New().
        Report(spectest.SequenceReport(spectest.ReportFormatterConfig{
            Path: filepath.Join("docs", "health"),
            Kind: spectest.ReportKindMarkdown,
        })).
        CustomReportName("health_success").
        Handler(api).
        Get("/v1/health").
        Expect(t).
        Body(`{"name": "naraku", "revision": "revision-0.0.1", "version":"v0.0.1"}`).
        Status(http.StatusOK).
        End()
}

In spectest.SequenceReport, you specify the "output destination for E2E test results" and indicate that the results should be output in Markdown format. With CustomReportName, you specify the filename for the Markdown file. The above code will generate a Markdown file named "docs/health/health_success.md".

E2E test result document example

The spectest also supports cases where API responses are not in JSON format. "Additionally, it can generate documentation in HTML format, not just markdown. For example, if the response body includes an image, it generates documentation in HTML format like the following:

Document index markdown file example

When you write a lot of E2E tests, the spectest generate multiple Markdown files.

It can become challenging for you to reference all these Markdown files individually. Therefore, you can use the spectest CLI to generate an index file with links to the Markdown files.

You execute the following command to generate an document index markdown file:

$ spectest index docs --title "naraku api result"

Roadmap

Roadmap is here.

Goal

Generate Markdown documents from E2E test results to reduce the time developers spend on document creation.
Support BDD-style testing methods.
Become a versatile library that allows easy writing of unit tests for entities other than APIs.
Provide CLI commands to enhance the developer experience during unit test execution.

Related project

The spectest dynamically generates markdown without using text/template, however instead utilizes its own Markdown Builder (go-spectest/markdown) during the markdown generation.

This approach makes the code more readable and intuitive compared to using templates.

DEV Community: nchika

A Minimal Go Toolkit for Cleaning, Validating, and Querying CSV/TSV/Excel/Parquet Files

fileprep

Features

Example

fileframe

Features

Example

filesql

Example

Building Tornago: A Go Library for Tor Integration Born from Fraud Prevention Needs

Introduction

Why I Built Tornago

Design Philosophy

1. Simple Wrapper, Not a Reimplementation

2. Minimal Interface

3. Production-Ready

How Tor Works: A Quick Primer

Onion Routing: Multi-Layer Encryption

Key Security Properties

What Tornago Does

1. SOCKS5 Proxy Communication

2. Tor Daemon Lifecycle Management

3. Hidden Service Creation

Getting Started

Prerequisites

Real-World Examples

Example 1: Accessing a Website Through Tor

Example 2: Creating a Hidden Service (.onion site)

Understanding Tor Startup Time

Cross-Platform Testing

Ethical Considerations

Conclusion

Links

horcrux: Split files into encrypted fragments for Node & React Native (TypeScript library)

How to use

How it works

Why did I make this?

Status & feedback

filesql: SQL Driver for CSV, TSV, LTSV, Parquet, and Excel Files in Go

Why I Created FileSQL

How to Use

Key Features

Current Phase

Join the Journey

My Life Goals

Thanks

sqluv - simple terminal UI for DBMS & local CSV/TSV/LTSV

sqlv demo

Support OS and DBMS

Execution Example: Running SQL on CSV/TSV/LTSV

Database Connection Feature

HOME Screen

Conclusion

vogen - Value Object Generator in golang

Introduction

Generates value object code

Intended Usage

Auto-Generated Code

Why I Wrote This Article

nao1215/csv - Read csv with validation in golang

What is nao1215/csv package

How to use

Struct tags

Validation rule without arguments

Validation rule with numeric argument

Why I wrote nao1215/csv

Next work

simple mermaid builder in golang: support ER/sequence diagram, pie chart

Markdown with mermaid

Mermaid sequence diagram syntax

Sample code

Generated markdown with sequence diagram

Mermaid entity relationsip diagram syntax

Sample code

Generated markdown with entity relationship diagram

Mermaid pie chart syntax

Sample code

Generated markdown with pie chart diagram

Next work