DEV Community: Nyctef

Automatically resolve formatting conflicts with jj fix

Nyctef — Fri, 04 Apr 2025 15:26:54 +0000

Jujutsu (jj) is a git-compatible versioning tool. It makes working with git repositories much nicer, and also supports more advanced workflows that would be difficult or infeasible in git.

As an example, here's a situation I ran into earlier this week. Let's say we have a class called Answer which has diverged on two branches. On main, the class has been reformatted so that we're not indenting everything by the namespace:

namespace MyApp;

internal class Answer
{
    public int GetAnswer()
    {
        return 54;
    }
}

while on a branch, the contents of the method have been changed:

namespace MyApp
{
    internal class Answer
    {
        public int GetAnswer()
        {
            return 42;
        }
    }
}

If we run jj log in this situation, we might get output like the following:

$ jj log
@  tuwwzvyz nyctef 2 seconds ago b2c07c4d
│  (empty) (no description set)
○  kwuuwuwx nyctef 1 minute ago git_head() 7e2d3b10
│  correct answer
○  lwowvoox nyctef 2 minutes ago 975fb9af
│  another change on this branch
○  svwxvtuo nyctef 3 minutes ago 665dcf1b
│  some change on this branch
│ ◆ xnrtvtyn nyctef 6 minutes ago main 9fd679a9
├─╯  reformat files
◆  kqxstvxn nyctef 7 minutes ago 679659f2
│  previous commit history on main
~  (elided revisions)

A quick primer on the log output: @ shows the change that we're currently editing, which is empty at the moment. After that the log shows the graph of previous changes. Changes marked with ◆ are considered to be immutable.

From left to right, the log shows:

○ is a node in the change graph
kwuuwuwx is the change ID. When editing changes, the change ID stays consistent, but the underlying git commit ID will be updated
nyctef is the author of the change
1 minute ago is the timestamp of the change
git_head() or main show any references pointing to that change
7e2d3b10 shows the current git commit ID for the change.

In the real terminal output, the first few characters of the change and commit IDs will be bolded, showing the minimum unique prefix. This is handy when typing out these IDs in commands.

Let's say we want to try fixing the conflict. The choice between merging and rebasing is somewhat arbitrary, but jj has a really nice rebase command so let's show that off. All we have to do is

jj rebase --destination main

and the current branch relative to main will be put on top of it. Jujutsu also has a nice set of arguments for specifying the set of source revisions to move. Compared to git, arguments are much more consistent between different commands, so they're much easier to remember as well.

In git, we might expect this rebase to stop halfway through once it encounters the conflict. That's not what happens in Jujutsu, though:

$ jj rebase -d main
Rebased 4 commits onto destination
Working copy now at: tuwwzvyz 9031b581 (conflict) (no description set)
Parent commit      : kwuuwuwx bd4ca349 (conflict) correct answer
Added 0 files, modified 3 files, removed 0 files
There are unresolved conflicts at these paths:
Answer.cs    2-sided conflict
New conflicts appeared in these commits:
  tuwwzvyz 9031b581 (conflict) (no description set)
  kwuuwuwx bd4ca349 (conflict) correct answer
To resolve the conflicts, start by updating to the first one:
  jj new kwuuwuwx
Then use `jj resolve`, or edit the conflict markers in the file directly.
Once the conflicts are resolved, you may want to inspect the result with `jj diff`.
Then run `jj squash` to move the resolution into the conflicted commit.

If we re-run jj log, we can see that the rebase has actually happened:

$ jj log
@  tuwwzvyz nyctef 9 minutes ago 8d02a9f0 conflict
│  (no description set)
×  kwuuwuwx nyctef 9 minutes ago git_head() 5062af7d conflict
│  correct answer
○  svwxvtuo nyctef 9 minutes ago ec16ab36
│  another change on this branch
○  lwowvoox nyctef 9 minutes ago fab09cd9
│  some change on this branch
◆  xnrtvtyn nyctef 28 minutes ago main 9fd679a9
│  reformat files
~  (elided revisions)

However, our change (and its descendants) are now marked as conflicts in the log. Jujutsu treats conflicts as a first-class object, and allows them to exist in the history. This means we aren't forced to fix conflicts as soon as they appear, unlike for git merge or git rebase. We can take time to fix each problem when necessary and when we have the right information to do so. However, Jujutsu won't let us push commits that contain conflicts to github or another git repo, so we do have to fix them eventually.

We can also inspect the conflict itself by editing the file locally or using jj show kw:

$ jj show k --config ui.conflict-marker-style=git
    ...
   4    4: {
   5    5:     public int GetAnswer()
   6    6:     {
        7: <<<<<<< Side #1 (Conflict 1 of 1)
   7    8:         return 54;
        9: ||||||| Base
       10:         public int GetAnswer()
       11:         {
       12:             return 54;
       13:         }
       14: =======
       15:         public int GetAnswer()
       16:         {
       17:             return 42;
       18:         }
       19: >>>>>>> Side #2 (Conflict 1 of 1 ends)
   8   20:     }
   9   21: }

Here we can see that the conflict is particularly annoying, because parts of the GetAnswer function boilerplate are copied both inside and outside the conflict area. If we just try to solve this by picking one side or the other in some merge tool, we're going to end up with broken code. It's not too bad in this toy example, but this problem gets much worse in realistic code.

Having changes containing conflicts is a bit of a scary state, though, especially if you're new to Jujutsu. It's worth noting another of Jujutsu's killer features: you can undo any operation using jj op undo and get the repo back to the previous state. If you want to go back further in the history, then you can use jj op log and jj op restore <revision> to get there. This is like a supercharged version of git's reflog command, and it works way more reliably.

If we ran jj op undo after the above rebase, and then ran jj log again, we would see that we're back in the original state with two branches and no conflicts.

The root of our troubles are the formatting changes on main: we'd be able to easily resolve the conflicts if those formatting changes were consistently applied to all the changes on our branch. We could add a new change to the branch with some formatting applied, but that wouldn't help with earlier changes. We could go through and edit the changes one by one (this is actually comparatively easy with Jujutsu!) but it would still be a pain for longer branches. Instead, I ended up using jj fix to solve this problem.

jj fix (documented here) is a command that takes a set of changes to apply to, and then runs various configured tools against each of the files that has a diff in those changes.

For C# formatting we happen to use the Jetbrains CLI, but this can be configured to work with any formatting tool. jj fix happens to require a tool which accepts file contents on stdin, and writes the updated file contents to stdout¹, so we need a wrapper script to make that work with the jetbrains cli:

#!/usr/bin/env bash

set -e # abort on error
set -x # print out commands to help debugging

RAND=$(tr -dc A-Za-z0-9 < /dev/random | head -c 10)
FILE="/tmp/file-to-format-$RAND.cs"

cat - > $FILE
# realpath is required since the file isn't part of the solution,
# so we need an absolute path to force jb to consider it
# https://youtrack.jetbrains.com/issue/RSRP-489150/
dotnet tool run jb -- cleanupcode  --no-build --include=$(realpath $FILE) 1>&2
cat $FILE
rm $FILE

This script creates a temporary file, formats it using the JetBrains CLI, and outputs the result. The realpath command ensures the file is recognized by the formatter, even though it's not part of the solution.

With that, we can run a command like the following:

jj fix -s k --config fix.tools.jb.command='["./scratch/cleanup-stdin.sh"]' --config fix.tools.jb.patterns='["glob:**/*.cs"]'

to automatically reformat files changed in k (and its descendants). jj fix is guaranteed to never introduce new conflicts, but it can fix conflicts in existing changes.

If we want to run this command more often, we can save the fix.tools settings to jj's toml config, and they'll automatically get picked up the next time we run jj fix. We can also use fix.tools.X.enabled to turn tools on or off as needed.

Because both branches now have consistent formatting, our merge conflict goes away and the fix can be integrated without issue. Jujutsu saves the day again! 🎉

If you're curious about trying out Jujutsu, I'd highly recommend installing it and giving it a try. It's compatible with existing git repos, so you don't have to worry about forcing other people to use it as well :)

Footnotes

In the future the proposed jj run command may be a better fit for jb cleanupcode, since it's designed to work with tools that expect a whole working copy to run against, rather than individual files

Shapes, slots, and traces: optimizing the property pattern

Nyctef — Fri, 21 Mar 2025 17:05:06 +0000

A common aspect of dynamic programming languages is the ability to create objects with arbitrary types, and to add or remove properties of these objects at runtime.

These objects are usually implemented as a hashmap (AKA a hashtable, dictionary or associative array). A hashmap provides amortized constant-time inserts, updates and deletes for properties in an object. However, despite a ton of research that has gone into making hashmaps as fast as possible, they still have substantial overhead due to the requirements to hash property names and search for the matching entries.

Even in statically typed programming languages, it is often useful to work with unstructured data. For example, in the schema comparison tool we're currently working on, we store the schema objects themselves as nested property bags. This means the core comparison code can be almost entirely generic—just comparing lists of named properties to each other—regardless of which database the schema objects came from.

Despite the massive expressive power of dynamic objects, it turns out that most objects in most programs end up looking very similar to each other. This insight means there's a bunch of possibilities for optimizing property access on objects—one of the most common operations in any language that has objects—and overcoming the overhead of the naive hashmap implementation.

I'm only scratching the surface of these optimizations here (and I'll probably get a bunch of the details wrong) but I've linked to some references + further reading that I found useful if you want to dive in more.

Javascript: shapes

There are a lot of different Javascript runtimes, but most of them have a similar concept of tracking object shapes (inspired by "Hidden Classes" from the SELF language). Given some code like the following:

const point = {};
point.x = 3;
point.y = 4;

which produces a point object with two properties, a naive implementation might store the object in memory something like this:

object point:
  - number of properties: 2
  - property 0: x
  - property 0 value: 3
  - property 1: y
  - property 1 value: 4

The first idea behind shapes is that the structure of an object can be stored separately from its contents, which is much more efficient when we have many copies of similar objects:

shape shapeA:
  - number of properties: 2
  - property 0: x
  - property 1: y

object point:
  - shape: shapeA
  - value 0: 3
  - value 1: 4

[... plus a lot more point values]

There's a lot more nuance about how shapes are discovered and tracked, which is explained in depth in the links below. For now, though, the other interesting aspect is "Inline Caches". For example, assuming we have a function like the following:

function getX(point) {
  return point.x;
}

If we want to resolve the .x property on the incoming object, we now have to follow the shape pointer on the object, and then look up the property descriptor in the list of properties on the shape. But this is still the same kind of slow hashmap lookup we've been trying to avoid! The next trick is to reserve a bit of space in the bytecode to cache shape of the last object that was passed into the function. Then the function gets compiled into something like the following:

function getX(point) {
  if (point.shape === shapeA) { // a fast pointer comparison
    return point.values[0];     // directly index into the object's values
  } else {
    // ... slow path: look up `x` property in `point.shape`
    //     (and then update the cache)
  }
}

The inline cache starts out empty, and gets updated when an object with a different shape passes through the function. (For example, the resulting code might compare against shapeC and access object index .3 instead).

Since the majority of objects passed into this getX function probably have the same shape, this turns out to be a useful optimization.

references

Python: `slots`

Shapes in Javascript relies on the runtime automatically detecting "hidden classes" during the program's execution, but Python is more explicit.

By default, every python object has a hashmap attached (named __dict__) where the object's properties are stored. Arbitrary properties can be added and removed from objects later:

class Foo():
    pass

foo = Foo()
foo.bar = 3
print(foo.__dict__) # prints {'bar': 3}

To optimize memory usage and potentially improve performance, Python provides the __slots__ declaration. By defining __slots__ in a class, you can explicitly declare data members (like properties) and prevent the creation of the default __dict__ for each instance. This can save memory and make attribute access faster because the properties are stored in a more compact structure.

Here's an example:

class Foo():
  __slots__ = ['bar']

foo = Foo()
foo.bar = 3
print(foo.bar)  # prints 3

foo.baz = 4 # AttributeError: 'Foo' object has no attribute 'baz'
print(foo.__dict__) # AttributeError: 'Foo' object has no attribute '__dict__'.

It's hard to tell exactly how much slots speeds up attribute access, and reports vary. In particular the cpython bytecode for slots vs non-slots is the same:

def get_bar(obj):
    return obj.bar

print(get_bar(foo)) # prints 3

import dis
dis.dis(get_bar) # disassemble the bytecode of `get_bar`
# prints the following:
"""
  2           0 LOAD_FAST                0 (obj)
              2 LOAD_ATTR                0 (bar)
              4 RETURN_VALUE
"""

here LOAD_FAST puts the first parameter on the stack, LOAD_ATTR calls getattr(top_of_stack, "bar") and RETURN_VALUE returns the top of the stack out of the function.

The implementation for getattr ultimately ends up going down different paths based on whether it needs to look up an attribute in __dict__ or in slots.

references

LuaJIT: traces

LuaJIT is a Just-In-Time (JIT) compiler for Lua that uses a technique called tracing to optimize code execution. Unlike many JIT compilers which compile a method at a time, tracing JIT compilers work by recording the operations performed during the execution of "hot" loops or frequently executed code paths. These traces can span across many methods and layers of abstraction, making optimizations such as devirtualization much easier than they would be otherwise. Once a hot path is identified, the JIT compiler generates optimized code for that specific path, which can then be executed much faster than the original interpreted code.

In LuaJIT, traces are sequences of instructions that represent the execution flow of a program. When a loop or function is executed multiple times, LuaJIT starts recording a trace of the operations performed. This trace is then compiled into highly optimized asesembly. The trace will largely be linear, with lots of guard clauses on conditionals checking that the current execution of the code is going down the same path as the trace. If any of these guard clauses fail, LuaJIT can fall back to the interpreter (which may cause new traces to be recorded) or continue with a side trace.

Tracing JIT vs more traditional method-based JITs are an interesting tradeoff. Since the traces that LuaJIT records generally don't have branches, and can span across wide areas of code, a lot of optimizations become much easier or trivial. On the other hand, the runtime can spend a long time interpreting bytecode and recording data before it's able to start compiling traces, so warmup time is much worse. Tracing JIT can also perform poorly in situations where branches are hard to predict. Some browsers have moved away from tracing JIT as a result, while others are using "meta-JIT" frameworks that combine per-method and tracing JITs at various levels of optimization.

For our purposes, we're interested in how LuaJIT optimizes constant attribute lookups. Let's take a look at some lua code this time:

point = { x = 3, y = 4 } -- this defines a table with two named attributes

function getX(point)
  return point.x
end

Since x is a constant attribute in this case, LuaJIT can emit a HREFK ("constant hash reference") opcode into its IR while compiling. The IR is what then gets transformed into machine-specific assembly code. The trick is that the HREFK instruction also includes the number of the slot in the table's hashmap where the entry is expected. If the key stored in that slot matches the expectation, then the program can skip the entire hash lookup algorithm and the trace can continue. Otherwise, the program will fall back to the slow path.

LuaJIT includes tools (particularly -jdump) that we can use to inspect the recorded trace, along with the IR and machine code that gets generated for it:

luajit -jdump -e "
point = { x = 3, y = 4}

function getX(point) return point.x; end

for i=1,1000000 do getX(point) end -- a big loop to force a trace to happen
"

references

What should you use?

Given all of the above, it's unlikely that you're optimizing your own programming language (and if you are, you could probably do a better job writing this blog post! :)

In cases like a schema comparison tool, though (where we're storing objects as property bags but have some additional knowledge about their structure) there's a lot of potential improvements that can be done by implementing something similar to __slots__ in Python.

It's not a critical performance improvement right now (hashmaps are already pretty fast, after all) but it's definitely something I want to keep an eye on.

Other references:

On the property pattern in general:

https://steve-yegge.blogspot.com/2008/10/universal-design-pattern.html

Building a cli parser from scratch

Nyctef — Sat, 11 Jul 2020 17:25:46 +0000

There are plenty of CLI parser libraries out there for pretty much every programming language under the sun. However, sometimes you might not find one with the right combination of features you want, or maybe you just want to understand what makes them tick. Building a CLI parser is probably much easier than you think!
To begin with, we're going to assume we're passed some arguments as a string array, and we don't have to worry about any complicated shell escaping rules! This makes our life much easier already.

Quick note: I’ll try to stick to the convention of talking about “parameters” for the various options we define, and “arguments” for the actual values passed to those parameters.

Our first CLI parser

Let’s assume that we want to handle a decent number of possible parameters, and some (or all) of those parameters might be optional. If we only want to handle a small, fixed number of positional parameters, you can usually get away with something pretty simple:

using System;
public static class SimpleArgs
{
    public static int Main(string[] args)
    {
        if (args.Length < 2)
        {
            Console.WriteLine("Usage: foo.exe [api key] [target]");
            return 1;
        }
        var apiKey = args[0];
        var target = args[1];

        // do something with apiKey and target...

        return 0;
    }
}

Instead, let's imagine we want to switch to named parameters like --apiKey and --target to make code using the CLI more readable.
The main complication is we can now specify the arguments in either order:

> foo.exe --apiKey abcd1234 --target production
> foo.exe --target production --apiKey abcd1234

The key thing to notice here is that we always alternate between a parameter name and an argument value. Instead of addressing specific args elements directly, we’ll iterate through the list and match up each argument with the right parameter:

using System;
public static class ArgsLoop
{
    public static int Main(string[] args)
    {
        string currentParameter = null;

        string apiKey = null;
        string target = null;

        foreach (var arg in args)
        {
            if (currentParameter == null)
            {
                currentParameter = arg;
                continue;
            }

            if (currentParameter == "--apiKey")
            {
                apiKey = arg;
                currentParameter = null;
                continue;
            }

            if (currentParameter == "--target")
            {
                target = arg;
                currentParameter = null;
                continue;
            }

            throw new Exception("Unexpected arg " + arg);
        }

        Console.WriteLine($"Got {apiKey} and {target}");
        return 0;
    }
}

In this loop currentParameter acts as a tiny state machine: either it’s set to some value and we’re expecting an argument value next, or it’s unset and we’re expecting a parameter name next.

We’re pretty close to making this into a generic parser! The next key insight comes from the design of CLI parser libraries like NDesk.Options or Mono.Options: we want to provide a list of parameter names and a callback for each one:

using System;
using Parameters = System.Collections.Generic.Dictionary<
                       string, System.Action<string>>;
public static class GenericLoop
{
    public static int Main(string[] args)
    {
        string apiKey = null;
        string target = null;

        var parameters = new Parameters {
            {"--apiKey", x => apiKey = x},
            {"--target", x => target = x},
        };

        Parse(parameters, args);

        Console.WriteLine($"Got {apiKey} and {target}");
        return 0;
    }

    private static void Parse(Parameters parameters, string[] args)
    {
        string currentParameter = null;
        foreach (var arg in args)
        {
            if (currentParameter == null)
            {
                currentParameter = arg;
            }
            else if (parameters.TryGetValue(currentParameter,
                                            out var callback))
            {
                callback(arg);
                currentParameter = null;
            }
            else
            {
                throw new Exception("Unexpected value: " + arg);
            }
        }
    }
}

And that’s it! We could totally stop here if we wanted. But there’s plenty of improvements we can make from this point, too.

With a slight tweak, we can allow for parameters which don’t take a value. This is useful for boolean flags like --quiet and similar. First we add the new parameter to our list:

        string apiKey = null;
        string target = null;
        bool silent = false;

        var parameters = new Parameters {
            {"--apiKey", x => apiKey = x},
            {"--target", x => target = x},
            {"--silent", _ => silent = true},
        };

(Note how the new callback ignores values given to it and just sets the flag.)

Secondly, we change the loop to prioritize matching parameter names over satisfying parameters:

    private static void Parse(Parameters parameters, string[] args)
    {
        Action<string> currentCallback = null;
        foreach (var arg in args)
        {
            if (parameters.TryGetValue(arg, out var callback))
            {
                // if the previous arg was a parameter name, then it
                // has no value
                currentCallback?.Invoke(null);
                currentCallback = callback;
            }
            else if (currentCallback != null)
            {
                // This arg doesn't match any parameter names, so it
                // must be an argument value.
                currentCallback(arg);
                currentCallback = null;
            }
            else
            {
                throw new Exception("Unexpected value: " + arg);
            }
        }

        if (currentCallback != null)
        {
            // we've run out of arguments to pass, but there's still
            // a pending parameter
            currentCallback(null);
        }
    }

Previously, if we saw arguments like foo.exe --target --apiKey, then we would have assumed that "--apiKey" was the value passed to --target. The updated code will invoke both callbacks for --target and --apiKey with null instead. The advantage is we can now call a CLI like this:

> foo.exe --silent --apiKey abcd1234 --target staging

and all the values will get set correctly, even though --silent didn’t get passed a value.

Personalizing your parser

It might not look like much, but this little loop can be the foundation for a lot of parser features:

Now that we have a programmatic list of all the arguments, we can generate help text when something goes wrong, or if the user specifies a --help argument. We could also add description text for each of the parameters and print that out too.
Error handling can be a bit tricky: we need to either throw and catch a custom ShowHelpException, or stop the process ourselves with Environment.Exit(1). Both methods work but have their own problems, so play with them and see which is the best fit for your case.
Currently if we want to have multiple aliases for each parameter, we need to repeat the whole parameter definition including the callback. A nice quality of life feature is supporting multiple aliases within the parameter spec, so eg "-q|--quiet" provides two alternatives for a quiet parameter.
One extra feature we needed was environment variable support for passing more sensitive arguments. Extending the above idea, we assumed environment variables followed a SHOUTY_SNAKE_CASE convention, so used parameter specs like "-p|--password|DATABASE_PASSWORD". Environment variables get checked first, so they can be overridden by cli arguments later if required.
The loop currently only supports named parameters, but we can add support for positional parameters too. The trick is to keep a separate queue of positional parameters, then check whether there are any remaining before carrying on with the error case:

    // track how many positional parameters have been consumed    
    var position = 0;

    // ... same loop as before ...
    }    
    else if (position < positionalParams.Length)
    {                  
        positionalParams[position].Handler(arg);
        position++;    
    }  
    else      
    {
        // and raise error as before

We only support string parameters, but you can easily add support for other parameter types. Just wrap the callbacks in code which implements parsing for integers, enums, FileInfos or any other custom type you need. You may find it helpful to replace the dictionary with a list of Parameter classes, then subclass for BoolParameter, IntParameter and so on.

That’s just a brief and incomplete list — once you have your own parser code that you’re fully in control of, you can implement as many or as few features as you need! I’d recommend starting off as minimal as possible. Don’t be afraid to implement one-off features outside the parser at first. Aim to convert these features into generic parser code once you have a few examples to guide you.

Reflection-based parsing

So our callback-based parser, while nice and simple to implement, isn’t very ergonomic. We have to set up a bunch of mutable variables, call the parser to populate them, and then manually verify that required parameters have a value. We’ll probably also want to also bundle the variables up into an options object to pass to the rest of the program.

What modern CLI parsing libraries tend to implement is something a bit more “magic.” They let us decorate a class representing the various parameters with attributes for how each parameter is specified, and then automatically create an instance of that class. Let’s dig in to how we might actually implement this using reflection in C#: other languages will differ, but the main concepts should be the same.

This process is going to consist of four steps:

Use reflection to inspect the passed-in class, and find the attributes with parameter specifications.
Build a callback-based parser with the list of parameters we’ve discovered, and implement callbacks that store the argument values in some intermediate bucket.
Run the callback-based parser against the args list.
Use reflection again to call the class’s constructor with the list of parameters‍ from the parser.

Once we’re done we’ll be able to define a class like this:

public class Example
{
    [Parameter("-a|--apiKey|FOO_API_KEY")]
    public string ApiKey { get; }

    [Parameter("-t|--target")]
    public Target Target { get; }

    [Parameter("-s|--silent")]
    public bool Silent { get; }

    public Example(string apiKey, Target target, bool? silent)
        => (ApiKey, Target, Silent)
         = (apiKey, target, silent ?? false);
}

and have our parser output nicely immutable, ready-to-go instances from the CLI arguments.

First we need to define a new attribute type:

public class ParameterAttribute : Attribute
{
    public string Spec { get; }
    public ParameterAttribute(string spec) => Spec = spec;
}

It’s pretty minimal — if you want you can add extra properties like some help text to make it more friendly. We’ll just be passing the values down into our callback parser, anyway.

Let’s get started with the actual Parse method:

    public static T Parse<T>(string[] args) where T : class
    {
        var type = typeof(T);
        var props = type.GetProperties()
            .Select(p => (
                  name: p.Name,
                  attr: p.GetCustomAttribute<ParameterAttribute>()))
            .Where(x => x.attr != null)
            .ToList();

The first thing to notice is that the method is static and generic. The T type parameter gives us an implicit reference to the type of parameter object to create. The typeof operator turns the implicit reference into an explicit value that we can inspect. Next, we search the list of properties on the type and find those with a Parameter attribute.

Once we have a set of parameters with their matching C# properties, we need to find a constructor which has an equivalent list of parameters:

        var propNames = props.Select(x => x.name).ToArray();
        var constructor = type.GetConstructors()
            .Where(c => ConstructorMatchesProps(c, propNames))
            .Single();

This means that once we’ve parsed all the parameters, we should have a set of values to call the constructor with. We assume that the constructor will do the obvious thing and pass its arguments to the equivalent properties, but the constructor is also allowed to implement extra behavior like default values. This will come in handy later when we want to implement optional parameters.

Some other libraries assume a different convention, and require the constructor parameters to match the order of properties on the class, but this feels a bit more fragile to me. Of course, given you control this code you can make whatever implementation decisions are best for you :)

The next step is to build up the underlying parser:

        var argsBucket = new Dictionary<string, object?>(
            props.Count,
            StringComparer.OrdinalIgnoreCase);
        var parameters = new List<Parameter>();
        foreach (var prop in props)
        {
            parameters.Add(
                new Parameter(
                    prop.attr.Spec,
                    x => argsBucket[prop.name] = x));
        }

Previously we had explicit variables to hold the results of each callback, but now we collect everything into a big Dictionary. The store here needs to be case-insensitive, since in C# constructor parameters and property names will have different case by convention.

Once that’s all set up, we just run the parser we built earlier:

        CallbackParser.Parse(parameters, args);

and argsBucket should be filled with all the values from the commandline.

At this point we can do something new: check for required parameters. We’re still in the parser code and we have a reference to all of the argument values, which we didn’t have before with the callback-based parser.

Usually we’d use a ParameterAttribute property to make parameters required or optional, but with C#’s new nullable reference types we can go one better and use a language feature. The nullability of each constructor parameter tells us whether the relevant option is required. This is a bit unintuitive (why don’t we use the nullability of the properties themselves?) but it means we can easily put default values in the constructor, like how --silent defaults to false in the example above.

      var requiredParams = constructorParams
          .Where(p => !Utils.IsNullable(constructor, p.param))
          .ToArray();
      var errors = new List<string>();
      foreach (var (attr, parameterInfo) in requiredParams)
      {
          if (!argsBucket.ContainsKey(parameterInfo.Name!))
          {
              errors.Add("Missing argument: " + attr?.Spec);
          }
          else if (argsBucket.GetValueOrDefault(parameterInfo.Name)
                   == null)
          {
              errors.Add("Missing argument value: " + attr?.Spec);
          }
      }
      if (errors.Any())
      {
          throw new Exception(string.Join(",", errors));
      }

Here’s an implementation of that Utils.IsNullable function — it’s a bit too big to include inline!

After all that, the actual magic part is a bit underwhelming. We build up the array of constructor arguments, invoke the selected constructor, and cast the result to the output type:

        var constructorArgs = constructor.GetParameters()
            .Select(p => argsBucket.GetValueOrDefault(p.Name!))
            .ToArray();
        return (T)constructor.Invoke(constructorArgs);

And that’s it! To use this, we just call Parse<Example>(args), and get a constructed Example object with all the properties filled in. I think you’ll agree this is a much nicer interface to work with.

To sum up, hopefully you’ve seen a few things:

We can build a simple CLI parser in a few lines of code.
Once we have our own CLI parser, we can add features much more easily compared to a third-party library.
We can use reflection to provide a much nicer “magic” parsing style, while building on top of a simpler parser. If you want more, you might also like “Building a virtualized list from scratch”, or Gary Bernhardt’s excellent FROM SCRATCH screencasts which inspired these posts!

This article was originally posted on Ingeniously Simple.

Better living through git aliases

Nyctef — Thu, 04 Jun 2020 08:35:05 +0000

There are many varied and wonderful git clients available, but by far the fastest, most flexible and most extensible is the git commandline itself. The problem is that it's ... a bit of a mess. There have been some improvements over the years, like changing push.default's default and splitting git checkout in two. But there are still fundamental problems, like the same concept being referred to as the "index", "staging area" or "cache" depending on the command.

Fortunately, we can mitigate a lot of the problems with git using aliases. A personalized set of aliases can help you be much more productive. I’m going to take you through some of the aliases I’ve created so far, but I recommend starting with the default git commands for a while and gradually introducing aliases as you see fit. One of the great things about aliases is they can become completely personal to the way you want to work.

There are a couple of ways to create aliases. You can create them on the commandline using something like git config --global alias.co checkout, or you can edit aliases in a text editor by running git config --global --edit. This will open an editor for your user-wide git config, where aliases will be in the [alias] section.

If you're ever unsure about what an alias does, then git help will quickly expand the alias for you:

> git help di
'di' is aliased to 'diff --color-words'

The simplest kind of aliases reduce the typing needed for some other command:

st = status
ci = commit --verbose
fa = fetch --all
co = checkout
di = diff --word-diff
amend = commit --amend
ama = commit --amend -a --no-edit
aa = add -A
mt = mergetool
rhard = reset --hard

This next one is pretty incomprehensible! It produces a useful git log that shows branches with one line per commit, much like you'd get in a UI client:

lg = log --graph --abbrev-commit --decorate --date=relative --format=format:'%h%C(reset) - %C(bold green)(%ar)%C(reset) %s%C(reset) %C(bold green)- %an%C(reset)%C(bold yellow)%d%C(reset)'

By default the log will expand to fill the terminal and let you page through the history. If you're looking for a recent commit you can limit the output with an extra option: git lg -20.

The next few aliases I use get a bit more complicated. You can combine aliases with other options, or make variants of the aliases like so:

cia = commit --verbose -a
cip = commit --verbose -p
dc = diff --cached --word-diff
fap = fetch --all --prune
cob = checkout -b

The next alias is pretty important. Git pull is broken, so I've aliased git p to a safe version. Since I'm lazy and it's fewer characters to type, I'll generally do the right thing :)

p = pull --ff-only

What happens if the above command fails to do a fast-forward? Well you might need to rebase, or merge, or you might just decide to discard your current branch and update to the version of the branch from the remote:

rhu = reset --hard @{upstream}

Here, @{upstream} is a useful alias for whatever the remote tracking branch is (usually origin/foo for some branch foo). It's a bit fiddly to type, though, so having the alias here is especially handy.

This next one lets you do a fast-forward on master while a different branch is currently checked out. Useful when other people have pushed changes to master that you want in your branch, since you don't need to switch branches before rebasing or merging.

fomm = fetch origin master:master

I used to have an alias like dlast = diff HEAD^ HEAD --word-diff for checking the contents of a commit that just happened, but later I discovered git show without any arguments will do the same thing :)

hg outgoing and hg incoming are a couple of commands I used to really miss from mercurial, but haven't used much in the last few years. These days it's rare that we'll have multiple people committing to the same branch, but they might still be useful to other people.

These aliases are a little more complicated than the previous ones - the ! at the beginning means "pass this to the shell" rather than just being a direct alias for another command. In this case we immediately run git again, but we could potentially use other commands as well.

outgoing = !git lg FETCH_HEAD..
incoming = !git fetch && git lg ..FETCH_HEAD
outin = !git fa && git lg --left-right ...FETCH_HEAD

Here's another one I haven't used in a few years, but has been especially useful in the past:

addw = !git diff -b | git apply --ignore-whitespace --cached

Short for “add-whitespace-insensitive”: stage changes while ignoring whitespace. Really useful when certain editors don't like whatever line endings you're using and decide to flip them over. Note: this breaks in powershell, since powershell implicitly changes strings in its pipeline to all have CRLF line endings! 🤦

Has this ever happened to you?

$ git push
fatal: The current branch foo has no upstream branch.
To push the current branch and set the remote as upstream, use

    git push --set-upstream origin foo

It's super annoying, and while there are more amusing solutions I like the following alias, which figures out the current branch name and automatically inserts it into the command:

puom = !git push --set-upstream origin `git rev-parse --abbrev-ref HEAD`

Now you just have to remember to use git puom instead of git push for the first push of a new branch.

And one final one:

fix-master = update-ref refs/heads/master origin/master HEAD

fix-master is useful when you start making commits, but realize you forgot to make a branch! You can checkout a new branch to point at the commit you're currently on, then use fix-master to repoint the master branch back where it should be.

A side note: If you’re working on windows, there’s a couple of things you’ll want in order to make the commandline actually pleasant to use. The first is a decent shell. This can be powershell, git bash or something similar - you'll probably want to avoid the default cmd unless you're already used to it. The second is a decent console to run the shell in. I've used ConEmu in the last few years and grown to love it, but Microsoft are finally investing in a new terminal which looks very promising.

Extracting game text from Nier:Automata

Nyctef — Sat, 30 May 2020 20:01:47 +0000

[Originally published in 2018]

Recently I’ve been playing through Nier:Automata again, and trying to stick to Japanese for more of the playthrough. This is a bit of a challenge since my level of Japanese comprehension is still roughly about that of a two-year-old baby. I ended up taking a lot of screenshots like the one above and then figuring out how to translate them after the fact.

This started me wondering, though - surely all these subtitles were tucked away in the game files and could be extracted if we just had the right tools. And it turns out there’s a pretty dedicated mod community that does stuff just like this. After some investigation, I found two useful repos - CriPakTools and att - which handled pulling apart the games archive format and then the individual data files respectively. We can chain them together with a quick powershell script:

function att ($inDir, $outDir) {
    new-item -force -ItemType directory $outDir
    C:\git\micktu-att\x64\Debug\att.exe export $inDir $outDir
}

function cripakexport ($inFile, $outDir) {
    new-item -force -ItemType directory $outDir
    C:\git\wmltogether-CriPakTools\CriPakTools\bin\Debug\CriPakTools.exe -x -i $inFile -d $outDir
}

gci G:\SteamLibrary\steamapps\common\NieRAutomata\data\*.cpk | foreach {
    cripakexport $_ F:\nier_unpacked_2
}

att F:\nier_unpacked_2 F:\nier_unpacked_2_extracted

and get a nested folder structure full of files like this:

...
ID: M5920_S0100_G0040_001_op60
JP: いえいえ、そうではなくて。天気がいいと気分が良いのかなー、なんて。
EN: Not really! I just figured it might feel nice to have some good weather.
RU: 

ID: M5920_S0100_G0050_001_a2b
JP: 気分が良くても良くなくても、作戦には関係ない。
EN: Feeling nice has no bearing on completing missions.
RU: 

ID: M5920_S0100_G0060_001_op60
JP: ははっ……２Ｂさんらしいですね。
EN: Hee hee! That is so like you, 2B.
RU: 
...

with the matching subtitle lines for English and Japanese, along with a RU: line (I believe the original author was working on a Russian translation)

This is already useful, but now we have a folder full of plain text files we can do some fun analysis, like this:

$folder = "F:\nier_unpacked_2_extracted"
$files = gci -recurse $folder | where { ! $_.PSIsContainer }
$fileContents = $files | foreach { gc -encoding utf8 $_.fullname }
$lines = $fileContents | foreach { if ($_ -match "^JP: (.*)$") { $matches[1] } }
$chars = $lines | foreach { $_.ToCharArray() }
$groups = $chars | group-object
$totals = $groups | sort-object -desc -property count

which finds the most common characters on all the lines in all files which begin with JP::

Count Name                      Group                           
----- ----                      -----                               
11496 。                         {。, 。, 。, 。...}
11445 …                          {…, …, …, …...}
 9108 の                         {の, の, の, の...}                      
 8533 い                         {い, い, い, い...}
 6542 、                         {、, 、, 、, 、...}
 6529 て                         {て, て, て, て...}
 6401 に                         {に, に, に, に...}

 ...

  190 兵                         {兵, 兵, 兵, 兵...}                    
  185 話                         {話, 話, 話, 話...}                     
  185 奨                         {奨, 奨, 奨, 奨...}
  184 的                         {的, 的, 的, 的...}
  184 墟                         {墟, 墟, 墟, 墟...}

 ...

which is pretty neat. Obviously we get basic kana all over the top of the chart, but further down we start getting kanji like 体 (body), 機 (machine/mechanism/chance), 生 (life) and 命 (life/fate). A lot of these kanji end up in 機械生命体 (lit. machine-lifeform), the name of the enemies in this game, which is probably not a coincidence. As you’d expect, the counts of character frequencies definitely look like they form some sort of power law distribution.

Anyway, this ended up being a pretty fun programming diversion - hopefully this’ll turn out to be a useful resource for learning more sentences.

DEV Community: Nyctef

Automatically resolve formatting conflicts with jj fix

Footnotes

Shapes, slots, and traces: optimizing the property pattern

Javascript: shapes

references

Python: __slots__

references

LuaJIT: traces

references

What should you use?

Other references:

Building a cli parser from scratch

Our first CLI parser

Personalizing your parser

Reflection-based parsing

Better living through git aliases

Extracting game text from Nier:Automata

Python: `slots`