DEV Community: Murahashi [Matt] Kenichi

nbwipers: Setup and Troubleshooting

Murahashi [Matt] Kenichi — Sat, 30 May 2026 00:43:13 +0000

What is nbwipers?

nbwipers is a CLI tool that strips outputs and metadata from Jupyter notebooks before git commit.

Written in Rust - faster than nbstripout
Supports git clean filter
Works with .ipynb files

Why use it?

Jupyter notebooks store cell outputs inside the .ipynb file (JSON). This causes problems:

Noisy diffs - output changes pollute every commit
Repo size - images and large outputs bloat the repo
Security - sensitive data can leak in outputs (API keys, query results)

The solution: strip outputs automatically on git add via a clean filter.

Why not nbstripout?

nbstripout is written in Python. It is slow - git status, git diff, and git add all became noticeably slow on this repo because nbstripout was invoked for every .ipynb file.

The main cause is Python startup time. With 100+ notebooks, nbstripout can take 40+ seconds where a Rust-based tool takes ~1 second.

Faster alternatives:

Tool	Language	Notes
nbstripout-fast	Rust	Up to 200x faster; no git filter install support
nbwipers	Rust	Inspired by nbstripout-fast; adds git filter + pyproject.toml config

nbwipers is essentially nbstripout-fast with better git integration. Switching to nbwipers fixed the slowness.

Setup

1. Install

felixgwilliams/nbwipers is now in the aqua registry as of v4.517.0.

Using aqua, add to aqua.yaml:

packages:
  - name: felixgwilliams/nbwipers@v0.6.2

Then run:

aqua install

2. Configure git filter

Run once per repo (writes to .git/config):

git config filter.nbwipers.clean "nbwipers clean -"
git config filter.nbwipers.smudge cat
git config filter.nbwipers.required true

Or edit .git/config directly:

[filter "nbwipers"]
    clean = nbwipers clean -
    smudge = cat
    required = true

required = true makes the commit fail if nbwipers is not installed. This prevents accidentally committing outputs.

3. Add .gitattributes

In the repo root, add .gitattributes:

*.ipynb filter=nbwipers
**/.ipynb_checkpoints/*.ipynb !filter
**/.virtual_documents/*.ipynb !filter

The !filter lines exclude checkpoint and virtual document files from filtering.

4. Verify

git check-attr filter path/to/notebook.ipynb
# Expected: filter: nbwipers

Today's issue: nbwipers was not working

Symptom

Committed a notebook in example2/. Outputs were not stripped.

Debug

git check-attr filter example2/pipeline.ipynb
# Output: example2/pipeline.ipynb: filter: nbstripout  ← wrong!

The filter was nbstripout, not nbwipers, even though .gitattributes says nbwipers.

Root cause

Found the culprit in .git/info/attributes:

*.ipynb filter=nbstripout
*.zpln filter=nbstripout
*.ipynb diff=ipynb

This file was written by nbstripout --install in the past.

Git attribute priority (highest to lowest):

.git/info/attributes ← highest priority
.gitattributes in the same directory as the file
.gitattributes in parent directories (up to repo root)
Global core.attributesfile

Because .git/info/attributes has the highest priority, nbstripout was winning over nbwipers in .gitattributes.

Fix

Remove the nbstripout lines from .git/info/attributes:

Before:

*.ipynb filter=nbstripout
*.zpln filter=nbstripout
*.ipynb diff=ipynb

After:

*.ipynb diff=ipynb

Verify fix

git check-attr filter example2/pipeline.ipynb
# Output: example2/pipeline.ipynb: filter: nbwipers  ← correct

Key takeaway

When switching from nbstripout to nbwipers, check .git/info/attributes.
nbstripout --install writes there, not to .gitattributes.
If both tools have entries, .git/info/attributes always wins.

json output from k6 on docker

Murahashi [Matt] Kenichi — Tue, 13 Apr 2021 16:03:44 +0000

You need to add --user $UID with docker run.

docker run \
  -i \
  -v $(pwd):/ci/ \
  --user $UID \
  loadimpact/k6:latest \
  run \
  --out json=/ci/performance/full.json \
  --summary-export=/ci/performance/summary.json \
  /ci/packages/faas-penguin-local/performance-test.js

And this is actual example:
faas-penguin/config.yml at c31ab0c47feb9c26ed8b3297468568dfbf6f25cd · sanemat/faas-penguin

refs:
Permission denied on k6 docker json output - k6 support - k6 community forum

Apply rubocop one by one

Murahashi [Matt] Kenichi — Mon, 12 Apr 2021 07:39:07 +0000

Use rubocop_auto_corrector

rubocop_auto_corrector v0.4.0をリリースした - くりにっき
 https://rubygems.org/gems/rubocop_auto_corrector/

Show npm/yarn current package version in repository on GitHub organization

Murahashi [Matt] Kenichi — Mon, 12 Apr 2021 02:19:42 +0000

Show Ruby/Gem version in repository on GitHub organization - DEV Community 👩‍💻👨‍💻

yarn

$ for dir in */; do (cd $dir; git grep -n -A1 -E '^jquery@' yarn.lock 2>/dev/null|perl -ne 'print if s/  version "(.*?)"/$1/'| tr -s '\n' '\0' | xargs -0 -I {} printf "$(basename $(pwd))/{}\n") done

npm

$ for dir in */; do (cd $dir; git grep -n -A1 -E '"jquery"' package-lock.json 2>/dev/null|perl -ne 'print if s/"version": "(.*?)"/$1/'| tr -s '\n' '\0' | xargs -0 -I {} printf "$(basename $(pwd))/{}\n") done

8cc.wasi: Pure WASI C Compiler

Murahashi [Matt] Kenichi — Sun, 07 Jun 2020 16:46:57 +0000

8cc.wasi is a Pure WebAssembly System Interface (WASI) C Compiler.

Compile C Code

Prepare your C code. Following is a 'Hello world' example.

int putchar(int x);

int main() {
  const char* p = "Hello, world!\n";
  for (; *p; p++)
    putchar(*p);
  return 0;
}

8cc.wasi compiles C code into ELVM IR (EIR).

$ cat hello.c | wasmtime 8cc.c.eir.wasi > hello.eir

$ cat hello.eir
    .text
main:
    #{push:main}
    mov D, SP
    add D, -1
    store BP, D
    mov SP, D
    mov BP, SP
    sub SP, 1
(snip)

elc.wasi compiles EIR into WASI.

$ (echo "wasi" && cat hello.eir) | wasmtime elc.c.eir.wasi > hello.wasi

$ cat hello.wasi
(module
 (import "wasi_unstable" "fd_write" (func $__wasi_fd_write (param i32 i32 i32 i32) (result i32)))
 (import "wasi_unstable" "fd_read" (func $__wasi_fd_read (param i32 i32 i32 i32) (result i32)))
 (import "wasi_unstable" "proc_exit" (func $__wasi_proc_exit (param i32)))
 (memory (export "memory") 1025)
 (global $a (mut i32) (i32.const 0))
 (global $b (mut i32) (i32.const 0))
(snip)

Run hello.wasi and get a result!

$ wasmtime hello.wasi 
Hello, world!

Refs

backend code for wasi

mp4 to gif

Murahashi [Matt] Kenichi — Tue, 21 Apr 2020 04:43:50 +0000

I converted mp4 file to animation gif.
ffmpeg is better than imagemagick on my purpose.

At first I tried convert command (imagemagick). It took 5 minutes and more, 100x file size, and it didn't finish!
convert from.mp4 to.gif

mp4 size is 5MB, 30 seconds.

Next I try ffmpeg. It took 10 seconds. It's enough for me.
ffmpeg -i from.mp4 to.gif

https://askubuntu.com/questions/648603/how-to-create-an-animated-gif-from-mp4-video-via-command-line

Local HTTP Server

Murahashi [Matt] Kenichi — Fri, 07 Feb 2020 06:10:58 +0000

npx http-server -a localhost

Simple One-Line Local HTTP Servers - sweetmeat - Medium

Show Ruby/Gem version in repository on GitHub organization

Murahashi [Matt] Kenichi — Tue, 10 Dec 2019 09:04:41 +0000

TL;DR

Clone repositories and check them on your local machine.

shallow clone into ~/tmp/packsaddle20191210

mkdir -p ~/tmp/packsaddle20191210
cd ~/tmp/packsaddle20191210
github-repos -org packsaddle -z | xargs -0 -P4 -I {} git clone {} --depth 1

https://github.com/sanemat/go-githubrepos
Git clone repositories in a GitHub organization

check Ruby version

for dir in */; do (cd $dir; git grep -n -E "^[ ]{3}ruby 2" -- Gemfile.lock| tr -s '\n' '\0' | xargs -0 -I {} printf "$(basename $(pwd))/{}\n") done
// or
for dir in */; do (cd $dir; grep -nH -E "^[ ]{3}ruby 2.*$" -- Gemfile.lock| tr -s '\n' '\0' | xargs -0 -I {} printf "$(basename $(pwd))/{}\n") done

check gem version, e.g. rails

for dir in */; do (cd $dir; git grep -n -E "^[ ]{4}rails " -- Gemfile.lock| tr -s '\n' '\0' | xargs -0 -I {} printf "$(basename $(pwd))/{}\n") done
// or
for dir in */; do (cd $dir; grep -nH -E "^[ ]{4}rails .*$" -- Gemfile.lock| tr -s '\n' '\0' | xargs -0 -I {} printf "$(basename $(pwd))/{}\n") done

For example

ruby-cron_for_github-app/Gemfile.lock:14:    rake (0.9.2.2)
ruby-saddler/Gemfile.lock:24:    rake (13.0.1)

go install from tools.go on cross-platform

Murahashi [Matt] Kenichi — Mon, 09 Dec 2019 15:35:04 +0000

TL;DR

go install \
  github.com/sanemat/go-importlist/cmd/import-list \
  github.com/sanemat/go-xgoinstall/cmd/x-go-install
import-list -z tools.go | x-go-install -0

Sometimes I want to track and use tool dependencies.

// +build tools

package tools

import (
    _ "github.com/sanemat/go-importlist/cmd/import-list"
    _ "github.com/sanemat/go-xgoinstall/cmd/x-go-install"
    _ "golang.org/x/lint/golint"
    _ "golang.org/x/tools/cmd/goimports"
)

How can I track tool dependencies for a module?
Manage Go tools via Go modules | Marco Franssen

Recommended way:

$ cat tools.go | grep _ | awk -F'"' '{print $2}' | xargs -tI % go install %

But I want to use it on cross-platform, ubuntu, mac, and windows.

$ import-list tools.go
github.com/sanemat/go-importlist/cmd/import-list
github.com/sanemat/go-xgoinstall/cmd/x-go-install
golang.org/x/lint/golint
golang.org/x/tools/cmd/goimports

Then go install all dependencies.

$ import-list tools.go | x-go-install

Split by NULL is more secure 😉

$ import-list -z tools.go | x-go-install -0

https://github.com/sanemat/go-importlist
https://github.com/sanemat/go-xgoinstall

git grep in subdirectories

Murahashi [Matt] Kenichi — Sat, 07 Dec 2019 18:33:15 +0000

git grep in subdirectories, then show the result of git grep + base directory name.

for mac:

for dir in */; do (cd $dir; git grep -n Gemfile| tr -s '\n' '\0'|xargs -0 -I {} printf "$(basename $(pwd))/{}\n") done

(snip)
ruby-saddler-reporter-text/.gitignore:3:/Gemfile.lock
ruby-saddler-reporter-text/README.md:59:Add this line to your application's Gemfile:
ruby-saddler-reporter-text/bin/console:9:# (If you use this, don't forget to add pry to your Gemfile!)
ruby-saddler-reporter-text/example/Gemfile:1:# A sample Gemfile
ruby-text_to_checkstyle/.gitignore:3:/Gemfile.lock
ruby-text_to_checkstyle/README.md:47:Add this line to your application's Gemfile:
ruby-text_to_checkstyle/bin/console:9:# (If you use this, don't forget to add pry to your Gemfile!)
ruby-uri-ssh_git/.gitignore:3:/Gemfile.lock
ruby-uri-ssh_git/README.md:65:Add this line to your application's Gemfile:
ruby-uri-ssh_git/bin/console:9:# (If you use this, don't forget to add pry to your Gemfile!)

Git clone repositories in a GitHub organization

Murahashi [Matt] Kenichi — Tue, 03 Dec 2019 17:15:10 +0000

github-repos

TL;DR

Show git clone url in GitHub organization.

Usage

github-repos -search "org:packsaddle archived:false" -z | xargs -0 -P4 -I {} git clone {} --depth 1

required: GITHUB_TOKEN=__YOUR_PERSONAL_TOKEN__

Motivation

There are several ways for cloning GitHub repositories in an organization.
How to clone all repositories in a Github Organization
Cloning All Repositories in a GitHub Organization - Scott's Weblog - The weblog of an IT pro focusing on cloud computing, Kubernetes, Linux, containers, and networking
Clone all repos from a GitHub organization

But I don't want to see my personal token for GitHub. And I want to handle pagination because my organization has over 400 repos 😅

I tried caarlos0/clone-org, but sometimes I got errors.

Repo

https://github.com/sanemat/go-githubrepos

Install

ghg

ghg get sanemat/go-githubrepos

go get

go get https://github.com/sanemat/go-githubrepos/cmd/github-repos

Updated: 2020-10-31

Feelings about GopherCon VietNam 2019

Murahashi [Matt] Kenichi — Mon, 04 Nov 2019 04:15:39 +0000

GopherCon VietNam 2019

2019-11-02
Capella Gallery Hall

The most interesting topic for me is "CloudRetro." Thanks speakers!

I'm very surprised. In Gopher con VietNam, most of the speakers have presentations in English. Most of the questions and answers are also in English. I can understand easily because the speakers talk in English. Awesome! But the reaction of attendees is better the speaker in Vietnamese 😅

People in the booth ask something to me, but Vietnamese is difficult 😓
A few attendees use their laptops. There are desks only the front row.

Tanuki

Tanuki is one solution for dependency hell. https://github.com/sausheong/tanuki

In my understanding, the speaker wants to talk about "the Unix philosophy."
But this approach seems each handler has dependencies. This means each handler has its own dependency hell. This is still thought experiment.

SRE

The fundamental introduction of SRE: Google - Site Reliability Engineering

We need to understand "error budgets" more.
5 key principles for SRE

Metrics & Monitoring
Capacity planning
change management
Emergency response
Culture

Start with Service Level Objectives
SLO and/or error budget

CloudRetro

This topic is crazy 😍
https://github.com/giongto35/cloud-game/wiki/FAQ
We can play a video game on the cloud server via a web browser, and the server encodes video and audio, then we can see this video. We can share the game state by permanent link.

Open-source Cloud Gaming Service For Retro Games
Video demo: https://www.youtube.com/watch?v=GUBrJGAxZZg
giongto35/cloud-game Web-based Cloud Gaming service with Retro Game