DEV Community: Matthew Watkins

Yes, commit messages can be fun!

Matthew Watkins — Wed, 01 Apr 2020 19:18:38 +0000

Today is April 1st. A perfect day for pranks and jokes and a little bit of fun. In that spirit, I want to share a reminder that commit messages doe not have to be dry and boring. It's OK to spice it up a bit and have some fun. In that light, I want to share a commit message I worked with recently that I think embodies the best practices of a commit message (clear, understandable, narrative form), but also is just fun to read. Some irrelevant technical details have been removed.

The problem

When I looked at the output JSON file, I noticed that the value was always the string null. You would think that would break parsing, but of course, the Gson library interprets the word null as a null reference to the object. And yet this worked for other objects and even for this same object in other sections of code!

After some digging, it turns out that there are two culprits to this crime:

First culprit: We are setting up the object to persist in this section using Java's double-brace initialization. That kind of inline property initialization works miracles in sane languages like C#, but apparently in Java where the type system is wonky, initializing your properties inline results in an anonymous type and (according to this blogger) dead kittens. Fair enough.
Second culprit: But when you pass said anonymous type to Gson.toJson(), it will not break. It will not throw. It will not complain. It just happily returns a string saying null. Workaround? If you pass in the destination type, everything serilializes fine.

Those are the two co-conspirators in the crime (certainly none of the blame falls at the feet of my own bad code, right?) But enough talk. Demo anyone?

MyObject myObject1 = new MyObject();
myObject1.setMessage("Hello world");
System.out.println(new Gson().toJson(myObject1));
// Prints "Hello world"

MyObject myObject2 = new MyObject() {{ setMessage("Hello world"); }};
System.out.println(new Gson().toJson(myObject2));
// A kitten is dead. Prints "null."

MyObject myObject3 = new MyObject() {{ setMessage("Hello world"); }};
System.out.println(new Gson().toJson(myObject3, MyObject.class));
// A kitten is dead. But at least is still prints "Hello world"

The solution

Look, no one likes dead kittens. But lots of people like me come from a background where the language doesn't create an anonymous type just so you can set properties (why Java why?). So we have to assume that some of the objects passed in to our serialization code may be full of dead kittens.

A dead kitten is bad enough. But it's even worse when it goes undetected for several days (like it was in my case). You see problems in your application and you know there's something wrong but you can't tell exactly where the smell is coming from and you start removing your drywall. By returning the word null, that's exactly what we're doing. Instead, we should let the caller know right away that they've got a problem and save them the unpleasant surprise later.

So I pulled the Gson dependency and abstrcted it into a class creatively named JsonConverter. That class interacts with Gson and when Gson serializes the @NonNull input object to null, we know that someone fell in the felinicidal trap so we can throw a proper exception to let them know they either need to get orthodontal on their code and start removing braces (ba-doom sh), or else they need to pass in the destination type because we can't have nice things in Java.

If you areoffended by jokes about dead kittens, please direct your ire to https://explodingkittens.com/contact.

Falling in love again

Matthew Watkins — Tue, 27 Nov 2018 13:46:37 +0000

Once upon a time, back in school, I discovered Cloud 9. A website that gives you an IDE in your web browser? And root terminal access to your very own Linux VM? I was in heaven. I had a beefy machine at work, but for my school work, I just had an old refurbished Chromebook. But I loved my old Chromebook, and didn't want to spend a bunch of money getting a proper laptop just to haul in my backpack to school every day so that I could compile code for my C or Java class. Plus, I needed Python for this and my other blog running on Jekyll. And Node for the Echo and Google Home actions I was working on on the side. Cloud 9 fit that need perfectly. I could spin up a new project for each stack and edit right from my Chromebook.

But really, editing on Cloud 9 is not the same as editing on a proper IDE. Auto-complete is extremely limited, and while the nature of the app forced me to get a lot better at the command line, when it's 2 AM and you're still working on that C assignment, you don't really care that you're getting really good at traversing directories via the command line or parsing out obscure GCC compilation errors from typos a "real" IDE would pick up right away.

I tried dual-booting my Chromebook using Crouton, but the overhead was so large I started running out of drive space before I even installed an editor (like I said, this is an old Chromebook). I almost gave up and bought a proper laptop, but in the end, I stuck with it. Partially to save money, and partially just to say that I managed to get through my 4-year CS degree with basically nothing but a Chromebook. :)

Now, I'm out of school and I've still been using Cloud 9. Why? Well, partially because that old Chromebook still is my laptop of choice around the house, and partially because I just feel weird installing RVM and Python just to write on my blogs. Or installing and configuring Node just so I can keep my Google Home action up to date. The list goes on. I don't want 50 versions of 30 frameworks in 5 languages stacking up on my personal laptop.

But at the same time, I've yearned for the nice things a native environment/IDE can offer. Even for my blog, which, leaving the underlying Jekyll guts aside is nothing but Markdown, YAML, and web frontend. It would just be nice to have real IntelliSense. And extensions. And breakpoint debugging. And all the other niceties that come with decent editors (like my prize child, VS Code). Even just the markdown preview would be nice. At one point, I wondered: since VS Code is an Electron app running on Node, how hard would it be to create a web-served instance of VS Code on some Linux container that could give me something like that?

Then yesterday I heard the guys over at CodingBlocks mention this new thing called Coder. It's exactly what I dreamed about: a web-accessible container that runs full instances of VS Code directly in the browser. I couldn't believe my ears. I was giddy. I ran home that day, threw open my laptop, created a free account. Yes, it's free. It's awesome. And it's everything I hoped for.

I crated a free account, cloned my GitHub-hosted blogs, installed RVM, Python, and Jekyll, read the docs on the ports to bind and... there it was! To celebrate, I'm writing this article on that same now 7-year old Chromebook in Coder.

I am in love. A proper IDE. In my browser. With all my extensions, task running capabilities, etc. In a container I can blow away and re-hydrate at will. It's beautiful.

I'm going to keep playing with Coder as they work through their public alpha. Next up on my "play" list is .NET Core and Node.js. I've noticed some bugs in the UI that defintely make themselves known, but so far, nothing that really gets in the way of a normal workflow. So far I am thoroughly hooked.

Casting stinks. Generic classes are worse.

Matthew Watkins — Thu, 12 Apr 2018 21:39:57 +0000

Let's go on an imaginary journey in class design for a minute.

Suppose that you have the following classes representing vehicles. You go the polymorphic approach and either write them to inherit from a common base class or implement a common interface:

public abstract class Vehicle
{
  public void Move() { }
  public abstract int PassengerCapacity { get; }
  // ...
}

public class Sedan : Vehicle
{
  public override int PassengerCapacity => 4;
  public bool IsConvertible { get; set; }
  public int TireSize { get; set; }
  // ...
}

// Other classes for motorcycle, truck, boat, etc.

So far so good, but these are complicated objects and require a factory to build them. And they vary enough that each one requires its own typed custom factory, assembly line, assembly line configuration, and engineers. So you create base classes or interfaces for those entities:

public abstract class VehicleAssemblyLineInstructions
{
  public int MaxNumberOfUnitsToBuildInParallel { get; set; }
  // ...
}

public abstract class VehicleEngineer
{
  public string Name { get; set; }
  // ...
}

public abstract class VehicleAssemblyLine
{
  public VehicleEngineer LeadEngineer { get; set; }
  public abstract Vehicle Build(VehicleAssemblyLineInstructions instructions);
}

public abstract class VehicleFactory
{
  public string FactoryName { get; set; }
  public VehicleAssemblyLine AssemblyLine { get; set; }
  // ...
}

Now that the base classes or interfaces exist, you move forward and create concrete implementations:

public class SedanAssemblyLineInstructions : VehicleAssemblyLineInstructions
{
  public int AirbagSafetyRating { get; set; }
  // ...
}

public class SedanEngineer : VehicleEngineer
{
  public bool IsCertifiedByFord { get; set; }
  // ...
}

public class SedanAssemblyLine : VehicleAssemblyLine
{
  public string SedanMakersUnionType { get; set; }
  public abstract Vehicle Build(VehicleAssemblyLineInstructions instructions)
  {
    // TODO: Implementaton here
    throw new NotImplementedException();
  }
}

public class SedanFactory : VehicleFactory
{
  public string SedanManufacturerName { get; set; }
}

OK so here's where we run into our first problem. In your concrete implementation's build method, in order to access sedan-specific members of the assembly instructions or the engineer, you need to cast them to the specific type. And who knows if they will cast or not. The compiler is not enforcing that they pass what you want:

public class SedanAssemblyLine : VehicleAssemblyLine
{
  public abstract Vehicle Build(VehicleAssemblyLineInstructions instructions)
  {
    // The compiler enforces that I have a VehicleAssemblyLineInstructions object. No idea whether the caller knew my
    // inner workings enough to guess that I really want a SedanAssemblyLineInstructions. Guess we'll find out now!
    SedanAssemblyLineInstructions sedanAssemblyInstructions = (SedanAssemblyLineInstructions)instructions;
    SedanEngineer sedanEngineer = (SedanEngineer)this.LeadEngineer;
    if (sedanAssemblyInstructions.AirbagSafetyRating < MIN_SAFETY && sedanEngineer.IsCertifiedByFord)
    {
      // ...
    }
  }
}

Yuck. And then after all that work, we return back the abstract type representation. So even if the caller magically knew to pass in the concrete input types, they have to magically know to cast the output types (and pray that we returned the right one):

SedanFactory sedanFactory = new SedanFactory
{
  AssemblyLine = new SedanAssemblyLine
  {
    SedanMakersUnionType = "Honda Sedan Workers of America",
    LeadEngineer = new SedanEngineer
    {
      Name = "Bob Jones",
      IsCertifiedByFord = true
    }
  }
};

SedanAssemblyLineIstructions sedanAssemblyLineIstructions = new SedanAssemblyLineIstructions
{
  MaxNumberOfUnitsToBuildInParallel = 2,
  AirbagSafetyRating = 4
};

Vehicle vehicle = sedanFactory.AssemblyLine.Build(sedanAssemblyLineIstructions);
Sedan sedan = (Sedan)vehicle;

Console.WriteLine($"Is convertible? {sedan.IsConvertible}. Tire size: {sedan.TireSize}");

So. much. Casting. Can we get rid of it? Well, yes, but it's not pretty. Here are the base classes:

public abstract class Vehicle
{
  public void Move() { }
  public abstract int PassengerCapacity { get; }
  // ...
}

public abstract class VehicleEngineer
{
  public string Name { get; set; }
  // ...
}

public abstract class VehicleAssemblyLineInstructions
{
  public int MaxNumberOfUnitsToBuildInParallel { get; set; }
  // ...
}

public abstract class VehicleAssemblyLine<E, I, V>
 where E : Engineer
 where I : VehicleAssemblyLineInstructions
 where V : Vehicle
{
  public E LeadEngineer { get; set; }
  public abstract V Build(I instructions);
}

public abstract class VehicleFactory<L, E, I, V>
  where L : VehicleAssemblyLine<E, I, V>
  where E : Engineer
  where I : VehicleAssemblyLineInstructions
  where V : Vehicle
{
  public string FactoryName { get; set; }
  public L AssemblyLine { get; set; }
  // ...
}

And here are the new concrete classes:

public class Sedan : Vehicle
{
  public override int PassengerCapacity => 4;
  public bool IsConvertible { get; set; }
  public int TireSize { get; set; }
  // ...
}

public class SedanEngineer : VehicleEngineer
{
  public bool IsCertifiedByFord { get; set; }
  // ...
}

public class SedanAssemblyLineInstructions : VehicleAssemblyLineInstructions
{
  public int AirbagSafetyRating { get; set; }
  // ...
}

public class SedanAssemblyLine : VehicleAssemblyLine<SedanEngineer, SedanAssemblyLineInstructions, Sedan>
{
  public string SedanMakersUnionType { get; set; }
  public abstract Vehicle Build(SedanAssemblyLineInstructions instructions)
  {
    // Look ma, no casting!
    if (instructions.AirbagSafetyRating < MIN_SAFETY && LeadEngineer.IsCertifiedByFord)
    {
      // ...
    }
  }
}

public class SedanFactory : VehicleFactory<SedanAssemblyLine, SedanEngineer, SedanAssemblyLineInstructions, Sedan>
{
  public string SedanManufacturerName { get; set; }
}

There are a ton of benefits to this on our end. All the method arguments, properties, etc are strongly-typed to the generic constraints by the compiler. The caller can't pass anything to us that is not the exact type we want, and we don't have to cast a thing. Same goes for the caller code:

SedanFactory sedanFactory = new SedanFactory
{
  AssemblyLine = new SedanAssemblyLine
  {
    SedanMakersUnionType = "Honda Sedan Workers of America",
    LeadEngineer = new SedanEngineer
    {
      Name = "Bob Jones",
      IsCertifiedByFord = true
    }
  }
};

SedanAssemblyLineIstructions sedanAssemblyLineIstructions = new SedanAssemblyLineIstructions
{
  MaxNumberOfUnitsToBuildInParallel = 2,
  AirbagSafetyRating = 4
};

// The sedan factory's assembly line returns a sedan. Just like I want!
Sedan sedan = sedanFactory.AssemblyLine.Build(sedanAssemblyLineIstructions);
Console.WriteLine($"Is convertible? {sedan.IsConvertible}. Tire size: {sedan.TireSize}");

Much nicer on the consumer end. But wow, that VehicleAssemblyLine class with 4 generic parameters makes me sick. Also, the generic arguments make it really hard to handle in an abstract way. For example, say you previously had a method that took in an arbitrary factory and printed its name:

public void Print(VehicleFactory factory)
{
  Console.WriteLine($"Welcome to {factory.FactoryName}");
}

Not possible anymore. Now that one method gets really ugly:

public void Print<L, E, I V>(VehicleFactory<L, E, I, V> factory)
  where L : VehicleAssemblyLine<E, I, V>
  where E : Engineer
  where I : VehicleAssemblyLineInstructions
  where V : Vehicle
{
  Console.WriteLine($"Welcome to {factory.FactoryName}");
}

One more thing: let's say you were serializing your objects somewhere in XML, JSON, or YML. And let's say you want to read a collection of them, but you don't know or care about the specific types in this case-- just that you want to read the common properties. You remove the abstract keyword from those classes so the parser can construct them. Without the generics, you could just do this:

VehicleFactory factory = JsonConvert.DeserializeObject<Factory>(payload);
Console.WriteLine($"Factory {factory.FactoryName}'s assembly line is run by {factory.AssemblyLine.LeadEngineer.Name}");

But no more. Now, it gets really, really hairy:

VehicleFactory<VehicleAssemblyLine<Engineer, VehicleAssemblyLineInstructions, Vehicle>, Engineer, VehicleAssemblyLineInstructions, Vehicle> factory = JsonConvert.DeserializeObject<VehicleFactory<VehicleAssemblyLine<Engineer, VehicleAssemblyLineInstructions, Vehicle>, Engineer, VehicleAssemblyLineInstructions, Vehicle>>(payload);
Console.WriteLine($"Factory {factory.FactoryName}'s assembly line is run by {factory.AssemblyLine.LeadEngineer.Name}");

Even if you're one of the "cool kids" who use the var keyword for all your variable declarations, you still end up wrapping lines just to declare a type.

There are other issues with the generic approach, like how there is compiler weirdness that doesn't seem to handle generic inheritance quite right, but in my opinion, the biggest issue is the fact that the number of generics in your class declaration will grow linearly with the number of concrete properties you want to expose through the generic type declarations. And while it will always start out with only one or two properties, it will always grow.

But on the other hand, the alternative to growing your generic type arguments is growing the number of annoying and dangerous casts you and the caller both have to perform.

I just don't see a nice way to do this. Is there some other design pattern or practice that I am forgetting about that will solve this? Comments are really appreciated.

Flattening JSON in JSON.NET

Matthew Watkins — Wed, 04 Apr 2018 01:10:05 +0000

Originally published on AnotherDevBlog.

The use case

I've got an interesting problem at work where I need to take any arbitrary JSON blob (object or array) and represent the leaf nodes in memory as a collection of key/value pairs. For example, given this JSON:

[
  {
    "Name": "Fish",
    "Color": "Silver",
    "Attributes": [
      {
        "Name": "Environment",
        "Value": "Aquatic"
      },
      {
        "Name": "Parts",
        "Value": [
          {
            "Type": "fin",
            "Length": 3
          }
        ]
      }
    ]
  }
]

I want to see something like this output:

[0].Name = Fish
[0].Color = Silver
[0].Attributes[0].Name = Environment
[0].Attributes[0].Value = Aquatic
[0].Attributes[1].Name = Parts
[0].Attributes[1].Value[0].Type = fin
[0].Attributes[1].Value[0].Length = 3

Here's what I did, and the lessons I learned

Lesson 1: Not everything is on StackOverflow

"This sounds like a pretty common use case," I said to myself, "surely there is something on the documentation or StackOverflow."

Nope. I searched StackOverflow for quite a while, and while I found a few answers referring to Java libraries, I couldn't find one for the Json.NET library we are using here. The most popular NuGet package on the internet and no one has ever faced this issue before? Seriously?!

It took a few hours and lots of debugging, but I eventually wrote an extension method to allow me to grab the leaf node values of arbitrary JSON:

public static class JExtensions
{
    public static IEnumerable<JValue> GetLeafValues(this JToken jToken)
    {
        if (jToken is JValue jValue)
        {
            yield return jValue;
        }
        else if (jToken is JArray jArray)
        {
            foreach (var result in GetLeafValuesFromJArray(jArray))
            {
                yield return result;
            }
        }
        else if (jToken is JProperty jProperty)
        {
            foreach (var result in GetLeafValuesFromJProperty(jProperty))
            {
                yield return result;
            }
        }
        else if (jToken is JObject jObject)
        {
            foreach (var result in GetLeafValuesFromJObject(jObject))
            {
                yield return result;
            }
        }
    }

    #region Private helpers

    static IEnumerable<JValue> GetLeafValuesFromJArray(JArray jArray)
    {
        for (var i = 0; i < jArray.Count; i++)
        {
            foreach (var result in GetLeafValues(jArray[i]))
            {
                yield return result;
            }
        }
    }

    static IEnumerable<JValue> GetLeafValuesFromJProperty(JProperty jProperty)
    {
        foreach (var result in GetLeafValues(jProperty.Value))
        {
            yield return result;
        }
    }

    static IEnumerable<JValue> GetLeafValuesFromJObject(JObject jObject)
    {
        foreach (var jToken in jObject.Children())
        {
            foreach (var result in GetLeafValues(jToken))
            {
                yield return result;
            }
        }
    }

    #endregion
}

Then in my calling code, I just extract the Path and Value properties from the JValue objects returned:

var jToken = JToken.parse("blah blah json here");
foreach (var jValue in jToken.GetLeafValues()
{
    Console.WriteLine("{jValue.Path} = {jValue.Value}");
}

Awesome!

Lesson 2: But it's always on StackOverflow

So it turns out there is an answer on StackOverflow for this use case (link). I was searching for terms like "get all leaf nodes" or "get all values with paths," but the magic keyword to make the answer appear is "flatten." Here's the answer code that was posted:

JObject jsonObject=JObject.Parse(theJsonString);
IEnumerable<JToken> jTokens = jsonObject.Descendants().Where(p => p.Count() == 0);
Dictionary<string, string> results = jTokens.Aggregate(new Dictionary<string, string>(), (properties, jToken) =>
{
    properties.Add(jToken.Path, jToken.ToString());
    return properties;
});

Lesson 3: But you can't always just copy what's on StackOverflow

Wow, that code snippet is a lot shorter than my solution, so I tried it out. But ultimately went back to my own. Here's why:

This solution doesn't work, at least not for my case. See, I need it to handle an arbitrary JSON blob. I can't promise it's going to be a JObect-- it could be an array or something else, so this solution, unfortunately, fails for me out the gate with my first test case (an array). And JToken doesn't have a handy little Descendants() method I can call like JObject does, so I'd have to do some type checking anyway. Yuck.
Another problem: this solution builds a dictionary in memory to represent the flattened structure. I'm dealing with some pretty massive objects, and it's already painful enough to load up that initial JToken. I'd really rather not add the memory pressure of the dictionary on top of that.
Speaking of memory, I'd like to (eventually) only return the JValue if it's not null or default for the value type.
That .Count() looks really expensive since it's a method being called on every single descendant, whether you end up using the descendant at all. Probably safer to just select only descendants that you know are JValue objects: .Descendants().OfType<JValue>(). Then you can call .Value. And when you have a JValue object, you can call .Value and get the underlying primitive (or pseudo-primitive string) value without calling the .ToString().

Help, these objects have the same shape but don't share an interface!

Matthew Watkins — Sat, 13 Jan 2018 16:23:51 +0000

I'm in a bit of a conundrum, and I can't seem to find anywhere online where someone has addressed it. Here's what I'm dealing with:

I'm using a third-party library to parse some documents into objects. Unfortunately for me, the documents are not XML or JSON, hence the need for this third-party library. Doubly unfortunate, the schema of this document type comes in three different versions, mostly the same but slightly different. These differences are represented in the objects that this third-party library returns.

Here's an example of what I'm talking about. Three POCO classes, each representing an order version:

// Order V1
public class ThirdPartyOrderV1
{
  public decimal Amount { get; set; }
  public ThirdPartyOrderItemV1[] LineItems { get; set; }
  public ThirdPartyOrderAddressV1 BillingAddress { get; set; }
  public ThirdPartyOrderAddressV1 ShippingAddress { get; set; }
  public string SomeV1ThingIDontCareAbout { get; set; }
  public decimal AnotherV1ThingIDontCareAbout { get; set; }
  // ...
}

public class ThirdPartyOrderItemV1
{
  public int ItemID { get; set; }
  public int Quantity { get; set; }
  // ...
}

public class ThirdPartyOrderAddressV1
{
  public string Name { get; set; }
  public string Street { get; set; }
  // ...
}

// Order V2
public class ThirdPartyOrderV2
{
  public decimal Amount { get; set; }
  public ThirdPartyOrderItemV2[] LineItems { get; set; }
  public ThirdPartyOrderAddressV2 BillingAddress { get; set; }
  public ThirdPartyOrderAddressV2 ShippingAddress { get; set; }
  public object SomeV2ThingIDontCareAbout { get; set; }
  // ...
}

public class ThirdPartyOrderItemV2
{
  public int ItemID { get; set; }
  public int Quantity { get; set; }
  // ...
}


public class ThirdPartyOrderAddressV2
{
  public string Name { get; set; }
  public string Street { get; set; }
  // ...
}

// Order V3
public class ThirdPartyOrderV3
{
  public decimal Amount { get; set; }
  public ThirdPartyOrderItemV3[] LineItems { get; set; }
  public ThirdPartyOrderAddressV3 BillingAddress { get; set; }
  public ThirdPartyOrderAddressV3 ShippingAddress { get; set; }
  // ...
}

public class ThirdPartyOrderItemV3
{
  public int ItemID { get; set; }
  public int Quantity { get; set; }
    public string Metadata { get; set; }
  // ...
}

public class ThirdPartyOrderAddressV3
{
  public string Name { get; set; }
  public string Street { get; set; }
  // ...
}

Notice that they look almost identical. And for my purposes, they are identical in that I don't need to consume the properties that are different between them. When we first wrote this project, we didn't have time to think too deeply about it, so we (horror of horrors) copy and pasted our logic after the parse:

public class Program
{
    public static void Main()
    {
        var parser = new ThirdPartyOrderParser();
        var order = parser.ParseOrder("text");
        if (order is ThirdPartyOrderV1)
        {
            IngestOrderV1(order as ThirdPartyOrderV1);
        }
        else if (order is ThirdPartyOrderV2)
        {
            IngestOrderV2(order as ThirdPartyOrderV2);
        }
        else if (order is ThirdPartyOrderV3)
        {
            IngestOrderV3(order as ThirdPartyOrderV3);
        }
    }

    private static void IngestOrderV1(ThirdPartyOrderV1 order)
    {
        Console.WriteLine("Order total: " + order.Total);
    }

    private static void IngestOrderV2(ThirdPartyOrderV2 order)
    {
        Console.WriteLine("Order total: " + order.Total);
    }

    private static void IngestOrderV3(ThirdPartyOrderV3 order)
    {
        Console.WriteLine("Order total: " + order.Total);
    }
}

-- .NET Fiddle

All three of those IngestOrderV* methods do the same logic on the same properties of the order, but handle each version individually, and have no shared code. Not ideal, and not maintainable. Now, we have some time to go back and see a better way to do it. Obviously, what we'd like to do is something like this:

public void IngestOrder(string text)
{
  var order = thirdPartyLib.ParseOrder(text);
  var orderTotal = order.Amount;
  var totalQuantity = order.LineItems.Sum(i => i.Quantity);
}

At this point, I would love to throw the developers of this third-party library under the bus for not using any polymorphism in their document classes. But I can't. Here's why:

This library parses hundreds of different document types, so I'm sure these classes are auto-generated by their system, which would probably have a dickens of a time trying to come up with an effective base class or interface for some of the documents that have changed more from version to version than this document I care about has changed.
I need to handle different versions in a common way. That's not true for all their users, who probably more often than not are using this library to only parse the version that their company has adopted, so for most of this library's user base, this is a non-issue.
Declaring an IThirdPartyOrder interface sounds fine and dandy, except that the properties are not simple value types. You would need IThirdPartyOrderItem and IThirdPartyOrderAddress, etc. And if you replace all the properties of all the version objects with their non-specific types like that, you lose all binding between a V3Item and a V3Order. The developer would then always have to cast his properties to a specific version every time he or she wants to reference a version-specific aspect of the object. This class has hundreds of custom-typed properties, so that gets really hairy really fast.

OK, so enough whining (and forgiving). What can I do with this situation where these objects are the same "shape" but don't share an interface? Here are some ideas I've thought about and/or played with:

Get creative with `dynamic`

If you treat the object coming back from the parser as dynamic, you can access all the properties without caring about type bindings. So, as long as the properties have the same names, we can get away with something like this:

public class Program
{
    public static void Main()
    {
        var parser = new ThirdPartyOrderParser();
        dynamic order = parser.ParseOrder("text");
        Console.WriteLine("Order total: " + order.Total);
        // ...
    }
}

-- .NET Fiddle

Doing this is awesome. We can now handle any object that comes our way, as long as it has properties with the names we specify. Effectively, we're accessing the object as we would in JavaScript land. Write it once and we're good to go, right?

Well, there are a few huge downsides to this. JavaScript returns undefined for properties that don't exist on our object, but .NET will throw a hard exception if we try to reference a non-existent property. So basically you're deferring the property checking to runtime instead of compile time. You can get around this by casting the dynamic to an object again, then using reflection to check if the property exists prior to calling it, but that's really tedious. Delaying type and property checking until runtime is extremely dangerous. It's removing the safety from your gun.

Not only is it unsafe, you lose all IntelliSense if we go this route. Refactoring becomes a huge pain, and debugging issues in production become a problem.

Finally: you can't invoke extension methods dynamically. So, maybe you know that the LineItems property is always going to be a collection, but the compiler doesn't. It has no idea that LineItems even exists, let alone its type. Heck, it doesn't even know what order is. So, without a bunch of casting and magical trickery, you can't iterate through your properties or call LINQ or other extension methods

Other "dynamic" approaches

A few other ideas that came to my mind: This library states that since orders are one-way POCOs (no backreferences), they can all easily be serialized to JSON, XML, YAML, etc. So maybe I could serialize the object to JSON and traverse through the object using the Json.NET JObject traversal method:

public class Program
{
    public static void Main()
    {
        var parser = new ThirdPartyOrderParser();
        var order = parser.ParseOrder("text");
        var json = JsonConvert.SerializeObject(order);
        var obj = JObject.Parse(json);
        Console.WriteLine("Order total: " + obj["Total"]);
    }
}

-- .NET Fiddle

Like with the dynamic class idea above, we won't get an IntelliSense or compiler type checking. But checking for property existence is a lot easier, and you don't have to get creative with reflection. And you get LINQ back. Plus, if I really wanted to, I could write a class (say, ThirdPartyOrderCommon that has the same common structure as the third-party order classes and do a strongly-typed deserialization to get what I want.

But on the other hand, like I said, this orders object is HUGE and contains hundreds of property types. That's a lot of code for me to write and maintain (and duplicate). Plus, it feels wrong to serialize an object just to translate it from one type to another (maybe AutoMapper would be a better candidate for that?)

So, I'm stumped

Surely, I'm not the only one who has run into something like this. What do you all do?

Dear LINQ, I love you, but I'm going back to my roots

Matthew Watkins — Tue, 19 Dec 2017 18:07:58 +0000

I faced an interesting situation today. I was getting a collection of objects from a database using a stored proc, but I only needed some of the objects that were returned. And I had to return the objects as a List.

Sounds like a good job for a LINQ where clause, right?

public List<object> DoWork() => return GetObjects().Where(ShouldKeep).ToList();
private bool ShouldKeep(object o)
{
  // Do a bit of work here...
}

Ah, nice. Short. Readable. Beautiful. And unfortunately, not a good idea in this case.

When LINQ is not appropriate

Listen, I love LINQ. When Java devs ask me what makes me love C# so much, LINQ is at the very top of my list. That said, I think that one of the biggest sins we .NET developers commit is the overuse of LINQ. My use case today gave me several reasons not to use my favorite feature. Anyone of them individually would probably be enough to disqualify LINQ as a valid tool for this problem. Here are a few:

Two heads are better than one. But two collections aren't.

Because of the way our data repository is implemented, that GetObjects() method doesn't return an IEnumerable<object>. It doesn't yield return the elements one at a time. It uses no delayed execution. It returns a list. A list that is taking up memory on my heap. A list that can get pretty big.

And unfortunately, I need to return a list, too. Just a (potentially) smaller one. And here's the rub: That .Where(ShouldKeep).ToList() creates a second list that is anywhere up to the same size as the potentially large list I fetched from my database call. You will rightly point out that the first list, in this case, becomes just an intermediate list. It will fall out of scope and be marked for garbage collection the moment the method returns the final results list. And that's true. But while that final results list is being built by my beloved LINQ, you can't escape the fact that will have two potentially large lists putting memory pressure on your process.

LINQ is best for atomic operations

OK, I lied. My ShouldKeep() method wouldn't just take the element as its parameter. The decision of whether any given element should be returned in the final list is a lot more complex and nuanced (and very stateful). It depends not only on the nature of the element itself, but also on features of other elements in the list, elements of other lists, and even (gulp) potentially another database call.

private bool ShouldKeep(List<object> list, object o, int index, DataProvider database, CustomConfiguration cfg, ...)
{
  // Do a LOT of work here...
}

All of these factors bring in their own weight of timers and logging and business logic. I will grant that some developer out there can probably break it into a bunch of separate predicates and use some lesser-known LINQ overloads that return the index with the element and perform all sorts of wonderful voodoo, but that's not a game I want to play right now. It makes the code harder to read and even harder to set breakpoints in. LINQ excels at handling atomic operations for its predicates, but if you try to get too fancy, you're probably going to end up sacrificing something big.

IEnumerable is awesome. But only once.

This next point isn't strictly related to my particular use case since I already mentioned that the database method I'm calling returns a list, but since I'm already on my soapbox, I might as well. Don't treat IEnumerable<T> as an actual collection. Here's an example:

public static void Main()
{
  Console.WriteLine("Starting");
  var myNumbers = GetNumbersFromDatabase();
  if (myNumbers.Any())
  {
    Console.WriteLine("Got back " + myNumbers.Count() + " results. Here they are:");
    foreach (var number in myNumbers)
    {
    Console.WriteLine(number);
    }
  }
  else
  {
    Console.WriteLine("Didn't get any results");
  }
  Console.WriteLine("Done");
}

At first glance during an over-the-shoulder code review, this may look fine. Maybe you even know that GetNumbersFromDatabase() hits 3 different tables so it's kind of expensive, but that's OK, you're only hitting it once, so it's fine, right?

Sure it's all fine. Fine until it hits production and your performance slows to a crawl. And your users start complaining that they are seeing, "Got back 0 results. Here they are:" instead of "Didn't get any results." So, after hours of looking for the bug, you finally add logging to the GetNumbersFromDatabase() method:

private static IEnumerable<int> GetNumbersFromDatabase()
{
  Console.WriteLine("In GetNumbersFromDatabase. Let's hit the database!");
  for (var i = 1; i <= 3; i++)
  {
    Console.WriteLine("Querying table " + i);
    yield return i;
  }
}

And you see the following in the console:

Starting
In GetNumbersFromDatabase. Let's hit the database!
Querying table 1
In GetNumbersFromDatabase. Let's hit the database!
Querying table 1
Querying table 2
Querying table 3
Got back 3 results. Here they are:
In GetNumbersFromDatabase. Let's hit the database!
Querying table 1
1
Querying table 2
2
Querying table 3
3
Done

Whenever you call a LINQ extension method on anything that implements IEnumerable<T>, it will re-evaluate the provider of the collection. If the underlying object is an array or a list, that's usually fine since it will just work on the object that already exists in memory. But if the implementation is doing something fancy like yield-returning its elements one at a time, you end up doing whatever work that method does each and every time.

The fix? If you need to know anything about (or do anything to) a bunch of objects other than iterate through it exactly once, put the elements in a collection. Add .ToArray() to the end of the call to GetNumbersFromDatabase() fixes it right away.

Another tip: I like the var keyword in general. But in this case, the var hid the fact that the returned object from the database call was an iterator instead of a collection. So maybe rethink your use of var in that case.

Back to the use case

So what did do I do for my GetObjects() case? Well ultimately I wanted to do something like this:

List<object> listFromDB = GetObjects();
foreach (var element in listFromDB)
{
  if (!ShouldKeep(element, ...))
  {
    listFromDB.Remove(element);
  }
}
return listFromDB;

But alas, if I try to run that I will get the good old "Collection was modified; enumeration operation may not execute" exception when I run it. It's almost like the CLR is begging you to duplicate the collection (which was reason #1 for not using LINQ in the first place).

Ditching LINQ and foreach

So how can we get around this? Well, we already ditched LINQ. Let's ditch our other C# favorite, foreach. If we access the elements by index, the responsibility of keeping track of iteration state falls on us instead of the runtime, so .NET won't complain and yell at us when we try to change something:

List<object> listFromDB = GetObjects();
for (var i = 0; i < listFromDB.Count; i++)
{
  if (!ShouldKeep(element, ...))
  {
    listFromDB.RemoveAt(i);
  }
}
return listFromDB;

Let's try that with a toy example where we filter out numbers between 3 and 7 inclusive:

public static void Main()
{
  var list = Enumerable.Range(1, 10).ToList();
  for (var i = 0; i < list.Count; i++)
  {
    if (list[i] >= 3 && list[i] <= 7)
    {
      list.RemoveAt(i);
    }
  }
  Console.WriteLine(string.Join(",", list));
}

A collection removal gotcha

Let's look at the output:

1,2,4,6,8,9,10

Whoah, what's with that 4,6,8 nonsense? Well, like I said, when we access the elements by index, the responsibility of keeping track of iteration state falls on us instead of the runtime. And we're doing it wrong. When we remove an element at an index, the indices of all elements past this one shift down by one. So we remove the element at i, then the for loop takes us on to element i + 1. But what used to be i + 1 has dropped down to become i, so we're now evaluating what used to be i + 2, skipping i + 1 entirely. Whenever you remove at the current index, make sure to decrement the iterating index so you don't skip anything:

public static void Main()
{
  var list = Enumerable.Range(1, 10).ToList();
  for (var i = 0; i < list.Count; i++)
  {
    if (list[i] >= 3 && list[i] <= 7)
    {
      list.RemoveAt(i);
      i--;
    }
  }
  Console.WriteLine(string.Join(",", list));
}

Tha's better:

1,2,8,9,10

Now we have a method that doesn't create an additional collection, and in fact, will only shrink the existing collection.

So next time you're faced with filtering a large data set in a non-trivial way, take a hard look at your LINQ and foreach and see if you can't easily release some memory pressure on your application by getting back to your roots.

Hacking in everyday life (fun with the Sprint API)

Matthew Watkins — Mon, 24 Jul 2017 16:28:01 +0000

Let me explain the situation I found myself in last week:

If you didn't already know, Sprint is offering a free year of unlimited service through the end of July if you port your non-Sprint number to them. It's a great deal, but if you are not a Verizon customer, they really make you work for it. If you're with a smaller carrier, the free years page will probably say you are ineligible, even if you have an eligible type of device. That's because Sprint maintains its own database of phones with some indicator on each record saying whether the phone is on the Sprint network or not (you heard me right: a database of phones that are not on their network). So, if your carrier (like mine) has not told Sprint that your phone is not on the Sprint network, the Sprint system assumes you are ineligible for the offer.

The solution is to create a support case with Sprint to get your device info (WiFi MAC address, IMEI number, phone number, make/model, and current carrier) added to that database. From the experience of family members who tried this, I know it would take up to three days for the information to get processed into the database and reflected on the free year page, and potentially longer for me to get the email from Sprint saying everything is hunky-dory. But time was of the essence, so I would have to check the free year page every few hours to try and find out when the offer was available for each of the devices I wanted to port. What a hassle!

This is where being a developer comes in handy.

I fired up my trusty Postman Interceptor Chrome extension and fired off a query for my still-ineligible phone. And there in the traffic history, I saw an AJAX HTTP request to a promising-looking sprint.com endpoint that returned some promising-looking JSON:

{
    "errors": [
        {
            "errorCode": "COULD_NOT_VALIDATE_SERIAL_NUMBER",
            "errorMessage": "Server.704:NMS returned status_code=49, status_text=INVALID_PHONE_OWNERSHIP: Phone owner can not be PLBL."
        }
    ]
}

The request was a GET, but it was pretty gnarly: all sorts of message IDs and tokens and timestamps. Luckily, Postman makes modifying and replaying it easy. So I started stripping off headers to see which ones I could sacrifice. Fortunately, most of the headers my browser had sent turned out to be unnecessary, and of the remaining, required headers, none were actually validated for accuracy-- the Sprint API just requires some value in those fields. At the end of the day, I could trim the entire HTTP request down to this (where xxxxxxxxxxxx represent the IMEI of the device I wanted to query):

GET /api/digital/byod/v1/foreign-device/xxxxxxxxxxxx HTTP/1.1
Host: www.sprint.com
messageid: 1000
enterprisemessageid: ECMW593313374
messagedatetimestamp: 2017-07-19T22:56:59
applicationuserid: LVO
applicationid: LVO

Awesome. But it's still tedious to have to do this manually every few hours. Time to automate it. I could have set up a chron job on my machine for this, but this is actually a perfect opportunity to bring in my good friend Google Apps Script. I created a new script and wrote the following, simple function and setup a trigger to have Google run it every 15-minutes:

function run() {
  if (isReady('xxxxxxxxxxxx')) {
    GmailApp.sendEmail('myemail@mydomain.com', 'YOUR PHONE IS IN THE DB NOW!', 'Go sign up for your free year');
    // Delete the 15-minute trigger I manually created earlier so I don't get spammed
    ScriptApp.deleteTrigger(ScriptApp.getProjectTriggers()[0]);
  } else {
    Logger.log('The phone is NOT in the DB'); 
  }
}

function isReady(imei) {
  Logger.log('Calling the Sprint API');
  var responseText = UrlFetchApp.fetch('https://www.sprint.com/api/digital/byod/v1/foreign-device/' + imei, {
    'method': 'get',
    'headers': {
      'messageid': '1000',
      'enterprisemessageid': 'ECMW593313374',
      'messagedatetimestamp': '2017-07-19T22:56:59',
      'applicationuserid': 'LVO',
      'applicationid': 'LVO'
    },
    'muteHttpExceptions': true
  }).getContentText();

  // Log the response from the API
  Logger.log(responseText);

  // Parse it and return whether the object exists and has no errors
  var response = JSON.parse(responseText);
  return response && !response.errors;
}

I received an email the next day from my script, informing me that I could now pounce on the free year plan. I figured I must have had a bug in the program, so I checked the logs. And there was the last logged API response, happily confirming that I was indeed now eligible for 365 days of unlimited LTE:

{
  "byodDetails": {
    "serialNo": "xxxxxxxxxxxx",
    "deviceSKU": "yyyyyyyyyyyyy",
    "netCompatibility": null,
    "iccId": null,
    "simKitInfo": [
      {
        "simSKU": "zzzzzzzzzzzzzzz",
        "simPartNumber": "aaaaaaaaaaaa",
        "simPriority": "1",
        "simSKUPricingInfo": {
          "finalPrice": 2.99,
          "basePrice": 2.99
        }
      }
    ]
  }
}

The point I want to make is that setting up Postman, playing with the request, and writing my Google Apps Script took all of one hour. One hour to automate what would have been a tedious, multi-day process of manual verification for each of my household's devices each day.

It got me wondering: what else can I hack? What other APIs are out there? Where else can I write a bit of code and get a few hours of my life back?

What in your life can you automate?

This post originally appeared on Another Dev Blog

Taming untracked files in a large git repository

Matthew Watkins — Sat, 25 Feb 2017 02:13:58 +0000

Have you ever tried to add a NuGet package to a Visual Studio project but accidentally added it to the whole solution? And now you have 100 untracked .config files showing up in your changed files list? Or have you ever stopped git in the middle of a big rebase and end up with a lot of new files on your machine and you need to rewind and get to a pristine state?

Well, that's why we have

git clean -fd

. But if you work in a very large codebase like I do at work, that command takes forever to run. I'm talking "go to your one-hour meeting and hope it finishes by the time you get back" forever.

Luckily, if the number of files that show as untracked is manageable, there's a faster way to get rid of those pesky untracked files than running

git clean

:

git ls-files --others --exclude-standard | xargs -n 1 rm -fr

I'm not one to post a command or code snippet without some explanation, but hopefully, this is obvious enough.

git ls-files --others --exclude-standard

is the magical incantation that prints the list of untracked file paths to stdout (just as they are printed in the last section you see when you run

git status

). We then pipe that list into the xargs utility, which iterates over them 1 at a time and passes each file path into the delete (


) command, leaving you with no files left in the working directory that git doesn't know about.

I maintain a bash script that I use to auto-create and synchronize my git aliases across my computers [here](https://gist.github.com/watkinsmatthewp/79093d4bf0d55c2e7269f534f1b6a689). I've added these commands as

 ```git list-untracked-files```

 and

 ```git delete-untracked-files```

 since I use them so often.

Know of any better ways to clean untracked files from a large codebase in Git? Leave a comment below!

There are two types of engineers. Be both

Matthew Watkins — Thu, 29 Dec 2016 18:03:08 +0000

A while back I listened to this talk. It's by Michael Lopp-- an engineering head who worked at companies like Netscape, Apple, Pinterest, and now works as Engineering VP for Slack. Pretty bright guy. At any rate, in his talk, he broke engineers into two basic archetypes: the stables and the volatiles.

Stables-- the Vogons

He gave a description of stables as follows:

Happily work with direction
Appreciate that there appears to be a plan
Like schedules
Play nice with others
Value efficiently run no-drama teams
Calmly assess risk
Carefully work to mitigate failure (however distant or improbable it might be)
Tend to generate process because the know process creates predictability and measurability

If I were to give a caricature of this type, it would be the Vogons from Hitchhiker's Guide to the Galaxy.

Volatiles-- the cowboys

And here's how he described volatiles:

Prefer to define strategy rather than follow it
Cannot conceive of failing
Find thrill in risk
Often don't build predictable or stable things, but they sure build a lot
Often are only reliable if it's in their interest
See working with others as time-consuming and onerous
Prefer to work in small, autonomous teams
Could care less how you feel

Think of Mark Zuckerberg's old mantra, "Move fast and break things." The term Ive heard most often in the industry to describe volatiles is "cowboy developers."

The inevitable conflict

With such differences in ways of thinking, there's bound to be conflict. Lopp points out that stables and volatiles really do hate each other. "Volatiles see stables as slow, lazy, and political bureaucrats." I think of Stables on the other hand "see volatiles as holding nothing sacred, do whatever the *^#! they want." Both are invariably chafing under the burden of having to work with the other.

But Lopp points out that "this conflict is a good thing."

Don't be overrun by one side. You need both

Lopp pointed out that startups usually start out with just a handful of volatiles-- quick-thinking innovators who can crank out a working 1.0 product really, really fast. They do the absurd, achieve the impossible, and blow everyone away with new ideas. But once the company starts gaining traction, these volatiles who poured their lives into the 1.0 turn into stables. When a new hotshot comes in and want to start pruning and building the next dangerous, amazing thing, the old stables push back and protect their well-running product from disruption from the new generation of volatiles.

"This," he said, "is how companies die." They transition fully from volatiles into stables. They start defending the well-running, reliable product from disruption from haphazard "innovation" by a new generation of volatiles. Newly hired volatiles join but quickly leave. They feel weighed down by the rigamarole. They smell the death and stagnation. This is what happened with Apple years back. it's why Steve Jobs was brought back on-- because he never became a stable. He always remained a volatile, and because of that, he saved a dying company.

It is critical to make sure volatiles and stables both feel comfortable and at home in your company. Without volatiles nothing exciting or new happens, and without stables, nothing stays working long enough to build a business. "Volatiles," he said, "are there to remind you that nothing lasts. They consider it their mission in life to replace that which is inefficient, boring, uninspiring." You need that. "Stables," he added, "are there to remind you of reality and to define processes. They bring predictability, credibility, and repeatability to your execution." You need that, too, and in equal representation once you reach scale. The challenge for companies, he said, "is to "create a world where [volatiles] can disrupt, and everyone is clear about the value of their disruption," with an equal appreciation for stables and volatiles for "the other side of the fence."

The best company has both. The best developer is both

Lopp presents the archetype of the stable and volatile developers as a spectrum, with most engineers tending towards one side or the other, some few finding a balance in the middle. I would, however, disagree a bit with this. The best developers I have worked with have not been just "in between" the two extremes-- they have been both at the same time. Volatiles don't look at them and see a good dev who's a bit too process-heavy, and stables don't see a restrained cowboy. They fit all the good qualities of both archetypes at once and fully. They are bold, brave, and willing to do something crazy and new. But they also maintain a healthy respect for the product and the cash cow that got them this far. They innovate, but they innovate smart.

I'm still trying to figure out which archetype I fit best in this "developer personality test." Early on, my coworkers would have probably, nay, definitely say I was out leading the camp of volatile cowboys. But over time, I feel I have definitely started gravitating away from that a bit. And when there's a rushed project, a changing deadline, or no time to put due diligence into unit testing, design, and refactoring, I find myself pulling back on the reins and balking at the breakneck speed that ecommerce development requires. Maybe I'm moving towards center. But better if I was becoming both.

So, where do you stand?

This post first appeared on Another Dev Blog

About Matthew

Matthew Watkins — Wed, 28 Dec 2016 17:58:43 +0000

My name is Matthew Watkins. I'm a software engineer, a husband, a father, a blogger, and a Mormon.

What I do

I currently work at a Morrisville, NC ecommerce company called ChannelAdvisor. We are a SAAS business, working to make selling online easier for businesses. Seriously great place, seriously good people.

Why I do it

When I was young, I always wanted to be an inventor. I always dreamed I would be some Wayne Szalinski, creating some new and innovative products that would make peoples' lives easier or less tedious. Software development allows me to design, build, and invent like that every single day. It's awesome.

What I do while code is compiling

Achieve Inbox Zero
Check out other dev blogs
Wish my code compiled faster

This post first appeared on Another Dev Blog

Every developer should write a personal automation API

Matthew Watkins — Sun, 20 Nov 2016 04:40:37 +0000

IFTTT, the building block of personal automation

A few years ago I fell in love with If This Then That (IFTTT). It's a remarkable, free service that lets you build simple rules for hundreds of apps, web services, IOT devices, and more. It's a great concept, and so amazing that I could not do it justice in this post. If you're not familiar with IFTTT (or the new Microsoft-hosted competitor, Flow), drop what you are doing and go check them out right now. Seriously, I can wait.

IFTTT filled a gap in my digital life I hadn't thought to fill before. But after using it for a while, I found myself itching for an even higher level of customization. I wanted to transform outputs, to chain rules together, and to add condition statements. Basically, I wanted IFTTT to be a free, hosted cron job executor with a nice UI, mobile apps, and hundreds of libraries I could use. I loved the simplicity of the rudimentary rules engines they offered, but I wanted more. I suspect many developers felt the same way.

Luckily, IFTTT heard the voice of the devs and added the Maker Channel, opening up endless possibilities. Think of it as a very simple, standardized API endpoint for IFTTT itself-- an HTTP API shell around all of the apps/services that IFTTT supports. You know all those times you wish there was some simple API around some service you use that doesn't require tons of setup and OAuth and SDKs and documentation? Well, you're welcome.

Between being able to receive requests and being able to call out to arbitrary HTTP endpoints, too, the Maker channel gives you all you need to build any automation you can dream of. Need some inspiration? Here are some of the things I'm doing with my API.

Ideas for your own API

Location reporting

I installed Tasker on the my phone and my wife's phone and setup a rule so that each time our phones connect to or disconnect from a Wifi network, and every few minutes when we're not connected to Wifi, our phones call out to my API and report its location and connected network.

On the API end, I group each person into home or away. From there you can track history, see the most recent location, etc. This by itself is a neat, bare-bones alternative to a more full-featured, third-party app like Life360 or Google+ location widgets. Helpful for emergencies (or alibis), but mainly crucial for the next piece.

Smart home automation

We have a smart alarm system and thermostat both compatible with IFTTT. When my API detects that all phones have reported back as being "away," it automatically arms our alarm system and turns the thermostat to away mode. The minute any one of us pulls into the driveway, our phones connect to the Wifi and report back to the API that we are home. The API disarms the system and sets the thermostat to home mode before we even make it into the house. In addition to location-based home/away automation, I wrote a cron job to toggle the alarm and thermostat between night and day mode as well.

I've found that the glass break sensor in our alarm system is super sensitive. I've had it go off when it hears the ice dispenser. Or plates clinking together. Or when I sneeze. So I also installed a simple HTTP request app on my Pebble watch to allow to me disarm the alarm with the press of a button on my wrist.

In the event that the alarm goes off and is not silenced within a minute, a text message is sent to a tech-savvy relative living nearby letting them know that if they don't receive a call from us, there may be a problem. I set up an SOS feature on the watch that does the same thing.

Build monitoring

Most build monitoring systems have some sort of post-build hook that you can use to attach to your API nicely. But even if not, almost all will send out emails with every build. That's where I'm at with my current job. I filter all build messages to a specific folder of my work Gmail and have a script check that folder every minute for any new build messages. It looks at the message and parses it to determine the current state of the build and if I contributed to it. If it's broken and I have contributed, it hits out to my API to send me a threatening message to my phone and watch via Pushbullet.

Site uptime monitoring

Recently a homework-critical web resource at school started going through a nasty bout of server issues. A lot. So I added an API cron job to query UptimeRobot for the status of that resource and post to the class Slack channel when that site went down or came back up again. That way we could all go start doing other things while we wait for the site to come back up instead of hitting refresh over and over again.

Automated stock management

Yes, IFTTT has a very basic integration for monitoring individual stocks. But the depth of the data you get is quite sad. So, I added functionality to my API to maintain a list of stock symbols I'm watching and query the the richer data for those symbols every few minutes from the practically undocumented Yahoo Finance API. When the stocks move into certain high/low thresholds of the 52-week highs/lows, I get notified in case I want to buy or sell.

Bonus tip: For the truly daring, I hear you can supposedly automate the whole thing with your Robinhood account and its secret API. The free trades makes the deal almost perfect if you can get it automated right. But if you're going to go through that hassle, just sign up with Quantopian and do it the right way.

General data querying

Really, anything becomes an API if you want it to. Let me give you an example. Before my wife and I gave in and got a second car, I would take the bus from school to work every day. But depending on where I was and what time it was, I had various options of which routes came to which stops at which times. And trying to piece together 4 different route schedules every day was a nightmare.

So I entered all the schedule information into a Google spreadsheet for the routes and stops I cared about. Then I added functionality to the API to query the next arriving buses/locations and added a trigger for that action to my smart watch. Get out of class, press a button, and I receive a notification letting me know exactly which buses would be arriving at which stops next and when they would get me to my workplace. Simple yet super helpful.

Remote shutdown/restart your computer

If my work machine has been unused for longer than a few hours, it stops responding to RDP connections. No friggin clue why. I've gone down the "check your power settings" route and the BIOS route and… nothing. Even looked at a possible network issue with the docking firmware.

Fortunately, the machine still receives and responds to all other network traffic in this state. But TeamViewer, LogMeIn, and Chrome Remote Desktop are big No-nos. So I wrote a quick Windows task that-- you guessed it-- queries my API to ask every 5 minutes to ask whether it needs to restart. This is a very temporary workaround (I sincerely hope), but so far it's been useful (and frequently used) enough that this particular action has earned a place as a DO button widget on my phone's home screen.

Writing your own API

So, how do you write your API? What stack will you use? What about hosting? If you already own a personal website running the stack of your choice, mazel tov. But if not, I have a good, powerful, free alternative to suggest:

If you don't want to go through the hassle of setting up your own server and are comfortable with JavaScript, check out Google Apps Script as an option for your API. Scripts you write can be configured to execute functions you select on a schedule, and and can act like a server in processing HTTP requests, calling other APIs, and largely running whatever code you want. Additionally, they come with built-in libraries to let you interact with your account's Google Calendar, Gmail, Drive, Docs, and basically anything else.

On the other hand, if you want an actual, proper server environment but still don't want to pay anything, check out the many providers offering tiny but free Linux VMs. Cloud 9 is probably my favorite in that arena. Extra points if you load .NET Core on your Linux VM and write your API in beautiful C## just because you can :) Azure, AWS, and others are fine options, too.

I ended up going down the Google Apps Script route. I had used Google Apps Script before when writing GmailSnooze, so I was already fairly comfortable with it, and even on my personal account, I haven't even come close to hitting their rate limits and haven't had any issues with reliability. So far no regrets. If you want to try your hand at Google Apps Script, I'll be writing about how I set up my personal API in my next post. Stay tuned.

Is it worth it?

As I've worked on this personal API project, I've often thought of Randall Munroe's famous automation effort graph:

I often ask myself, "How much time have I really saved compared to the original task? Am I just going down an automation rabbit hole?" I know that for some aspects, the automation has been convenient, but not life-changing. And some definitely took way too many hours debugging and rewriting to get just right. But as I add each new feature to my API, the next one becomes even easier to implement.

Most importantly though, my very way of thinking has changed. Every time a thought pops into my heads that starts with "Man, I wish I didn't have to…" it usually ends with "... could I automate this somehow?" And really, that's what software development is all about. We spend all day at work tinkering and automating and fine-tuning and improving various aspects of our business. How often do we take a few minutes to see what we could do to apply that to our lives? Who knows, you might find a way to automate just the right thing, and it ends up being the next big thing.

Uh-oh, my watch just buzzed. Got to go fix the build :)

This post first appeared on Another Dev Blog

DEV Community: Matthew Watkins

Yes, commit messages can be fun!

The problem

The solution

Falling in love again

Casting stinks. Generic classes are worse.

Flattening JSON in JSON.NET

The use case

Lesson 1: Not everything is on StackOverflow

Lesson 2: But it's always on StackOverflow

Lesson 3: But you can't always just copy what's on StackOverflow

Help, these objects have the same shape but don't share an interface!

Get creative with dynamic

Other "dynamic" approaches

So, I'm stumped

Dear LINQ, I love you, but I'm going back to my roots

When LINQ is not appropriate

Two heads are better than one. But two collections aren't.

LINQ is best for atomic operations

IEnumerable is awesome. But only once.

Back to the use case

Ditching LINQ and foreach

A collection removal gotcha

Hacking in everyday life (fun with the Sprint API)

Taming untracked files in a large git repository

There are two types of engineers. Be both

Stables-- the Vogons

Volatiles-- the cowboys

The inevitable conflict

Don't be overrun by one side. You need both

The best company has both. The best developer is both

About Matthew

What I do

Why I do it

What I do while code is compiling

Every developer should write a personal automation API

IFTTT, the building block of personal automation

Ideas for your own API

Location reporting

Smart home automation

Build monitoring

Site uptime monitoring

Automated stock management

General data querying

Remote shutdown/restart your computer

Writing your own API

Is it worth it?

Get creative with `dynamic`