DEV Community: Donald Merand

ExAirtable - Airtable in your Elixir

Donald Merand — Mon, 09 Nov 2020 23:42:18 +0000

tl;dr: If you use Elixir and like Airtable, check out a new library I wrote called ExAirtable.

At EXPLO, we often reach for Airtable when we're trying out new ideas. We love it because it really reduces friction for putting together simple relational systems that are the backbone of knowledge work. We are also very happy with Elixir and Phoenix when it comes time to build apps for the web, which seems to happen more and more these days!

I've often wondered whether we could use Airtable as a content management back-end for some of these sites. It solves a few problems that we consistently have:

A simple and consistent management front-end that includes relational data along with tabular views, forms, calendars, etc.
File-hosting and image auto-resizing! You just make a "file" field type and you can upload files right into your 'tables. Images get automatically scaled to a variety of sizes.
API! Every project has the same API structure and the API is sane to use.

We've used Ruby with Sinatra or Rails and the excellent AirRecord library to do Airtable-based sites before. But for Elixir, the current Airtable API libraries didn't seem to be in active development, and didn't have features like rate-limiting or caching. But it would be so cool if we had an Elixir library that would allow us to handle Airtable rate-limiting and caching as a single Elixir-native dependency!

Meanwhile, Ricardo Garcia Vega wrote this amazing article demonstrating the use of Elixir and Phoenix as a caching layer for Airtable for a content management system. This "native" Elixir solution was just the push I needed to fix Elixir's Airtable library problem.

So here's the buried lede: I wrote an Airtable library (actually two! read onward...) for Elixir called ExAirtable. This library aims to provide a solid foundation to use Elixir (with or without Phoenix) with Airtable interaction. It handles rate-limiting against the Airtable API, and (optionally) caches all requests in local memory to drastically speed up reads.

A Solid Foundation

I designed ExAirtable to work in two different ways:

Just hit the Airtable API directly. Don't worry about rate limits too much and give me all of the pages of data when I want a list. You'd use this method if you have a simple case where you don't expect things to get crazy in terms of number of requests or size of data.
Run as a supervised, cached, and rate-limited server tree. All client requests will go into a per-base queue which will never exceed 5 API requests per second. All desired data will be stored in an in-memory cache that is periodically refreshed via the API. Client request retrieved from the cache whenever possible to drastically speed up read times. This mode is useful when you're dealing with larger bases or apps that may have a lot of clients performing read and write operations.

Here's how the system comes together:

If you decide to run the service in cached and rate-limited mode, here's how the supervision tree comes together:

If folks are interested, I can write more articles with code examples. In the meantime the documentation has tons of examples and information. I've tried hard to make it comprehensive and useful all by itself, which is a truly game-changing trick that I've learned from the Elixir community.

Use with Phoenix and/or Ecto

Realistically, most folks who want to try this with Elixir are going to use Phoenix as their application framework. Since Phoenix plays so well with Ecto, I found it beneficial to create a set of conventions around creating ExAirtable Table models in an Ecto context with embedded schemas.

That library (called ExAirtable.Phoenix) is here: https://github.com/exploration/ex_airtable_phoenix

Once again, I can write more follow-up if folks are interested. In the meantime, give it a try and let me know what you think!

DNS is a Thing that People Hack Now, I Made an Open-Source App to Help

Donald Merand — Thu, 30 Jan 2020 21:24:56 +0000

This article about recent DNS hijacking attacks is a wake-up call. I recommend you read it, but here's a bit of a summary in case you're not a big link-clicker (emphasis mine):

In the case of e-hawk.net, however, the scammers managed to trick an OpenProvider customer service rep into transferring the domain to another registrar with a ... social engineering ruse — and without triggering any verification to the real owners of the domain.

These folks were able to hijack DNS using old-fashioned social engineering techniques. They got a domain registrar to transfer a domain over the phone! Then they stole all of the data. Okay, there's a lot more to it, you should maybe go read the article...

The big take-home actions for me (they're listed at the end of that article) were:

Set up multi-factor authentication at your domain registrar. If they don't do multi-factor, consider changing to a registrar that does, because you are now at risk due to their lax security.
Configure DNSSEC signing zones + validating responses. See above if your registrar doesn't have this.
Lock your domains with a domain lock. This is different from a "transfer lock" because it requires a laborious personal contact process to undo. This is what you want once your domain is configured correctly. Don't do it until you've got things configured correctly!
Monitor your DNS

On that last point, I have some news for you:

DNS-Monitor, a New Ruby-Based CLI Tool

I found myself wishing that I had a little robot to monitor all of my hostnames and warn me if there were any changes to them. So, I wrote that app yesterday and open-sourced it as the DNS-Monitor Gem. The big thing you get is a command-line utility called dns-monitor. Here's how it works:

You feed it a text file with all of your host names (just one line per host).
You figure out the RDAP (JSON WHOIS) endpoint of your domain registrar. I've done zero research on who uses RDAP, but it seems like a thing many of them use? At EXPLO we use Pair Networks, so I put their server in as a default, but you can change it with the -u or --rdap_url flag.
- Note that in the current configuration, I'm assuming only one registrar (because that's what we have). Send a pull request if you'd like to add the ability for multiple RDAP servers! :)
You run the app like this: dns-monitor --check or dns-monitor -c. It will run through your list of hosts, and compare the current RDAP (WHOIS) value for that host with a (SQLite3) database of previous values.
The output of the program is a JSON array of all of your domain RDAP (WHOIS) information, with a diff showing changes for any domains that have them.
At EXPLO we use Google Hangouts Chat for 'bot notifications, and Mandrill for transactional email, so I put in some extra functionality for those two things. If you send a Mandrill API key + email you can get email notifications if any domain has changed. But even if you use neither of those things, you can just pipe the program output into whichever utility you do use.

We at EXPLO are using it currently, and it works pretty well for our use case. If you'd like to give it a try, just gem install dns-monitor to get it for yourself. Here's an example cron script for how we use it in practice. It does some extra work to specify where the hosts.txt and database files live:

In summary, DNS is scary now, I hope this helps. Send me a note if you end up using it!

Any FileMaker Devs Here?

Donald Merand — Fri, 06 Apr 2018 13:12:11 +0000

I spend a lot of my days in FileMaker, and I wonder: are there any FileMaker devs lurking around here? If so, what are your favorite parts? What would you like to learn more about? If not, would you be interested in learning about it?

Is Touch-Typing an Important Skill for Software Developers?

Donald Merand — Sun, 01 Apr 2018 12:38:55 +0000

When I watch my expert colleagues navigate their work, I'm impressed by how their tools look like an extension of their body. A videographer with a custom Final Cut jogwheel + keypad setup, a designer who knows every Illustrator shortcut - they go about their work with a fluidity that looks like a dance. They never look at their hands while they're working.

In college, a professor told me that "Once you learn vim, you can move mountains." This sounded impressive, but didn't really click with me until 5 years after I graduated when I finally went through :vimtutor and dedicated myself to using Vim for all of my code development. I can move mountains (of text)! But vim is all but useless if you can't touch-type.

I've always told my colleagues at work to memorize keyboard shortcuts. "Imagine how many hours of your life you've spent just moooooving your hand over to the mouse so that you can click some icon!" I say. But then I realized: keyboard shortcuts will only save you time if you can touch-type. If you can't find home row, a keyboard shortcut is not going to make you faster.

So that's my question - given that our range of professions involve fending our way through gigantic piles of text, is true mastery only attainable to those who know how to touch type? Or is it merely a nice tool to have in one's proverbial belt? Or am I missing the point entirely?

Do You Prefer Multiple Monitors or Just One? Why?

Donald Merand — Tue, 27 Mar 2018 01:14:34 +0000

Using JavaScript Promises for Non-Async Control Flow?

Donald Merand — Sun, 11 Mar 2018 01:58:59 +0000

I love functional languages like Elixir + Elm that give you a pipe syntax for control flow:

data
|> first_transformation
|> second_transformation
|> etc

It's easy to understand what this program is doing! I've been learning about how promises work in JavaScript, and they seem like a good way to mimic this kind of data-chaining, even if your functions aren't really doing anything that needs to be asynchronous:

first_transformation(data)
.then(second_transformation)
.then(third_transformation)
.catch(handle_errors_from_the_whole_chain)

The MDN docs on Control flow and error handling suggest that promises are mostly for async/deferred operations. Is it normal to use promises merely as a mechanism for control flow and organization? Is it a horrible idea for some reason?

Explain It Like I’m #{[5, 10, 15, 20]}

Donald Merand — Tue, 06 Mar 2018 22:10:28 +0000

Cross-posted at donald.merand.org.

According to Richard Feynmann, the true measure of understanding a concept is to be able to relate it to a toddler. This is what I love about the #explainlikeimfive channel here on Dev.to: explaining tech concepts in a couple of pithy paragraphs, without too much technical jargon, is a fun challenge to see if I myself truly understand the concept. I may have a problem with compulsively commenting on these discussions...

But there is one thing that bothers me: I'm not sure that the asker always wants an answer suitable for a five-year-old! Five-year-olds aren't typically concerned with specific terminology, but responses to #explainlikeimfive are often very jargon-heavy. The requester may have wanted that, but that's not an answer you'd want to give a 5-year-old! Not that it matters whether the requester truly wanted a 5-year-old response versus a 7-year-old response, or what-have-you. What truly matters is that when your response doesn't match the question, you run the risk of being ignored or misunderstood.

I think understanding these nuances is fun! I'm going to try explaining the same concept to a theoretical 5, 10, 15, and 20-year old. In doing this, I hope to learn something about my own process of teaching, and hopefully find some varying approaches that work well. I'll attempt to Explain Git + GitHub Like I'm Five (and Ten, and Fifteen, and Twenty).

#EXPLAINLIKEIMFIVE

One good way to #explainlikeimfive is to tell a short story which is an analogy. Five-year-olds aren't typically concerned with specific terminology, but they are very good at understanding metaphor. Your goal here would be to get them to understand how this idea solves a problem that's very similar to one that they may have had.

Say you are writing a short story and you want to do a really good job. You might write the story on one sheet of paper, and then get another sheet of paper and write another version, and then keep writing more versions on more sheets of paper. After a while, you'd end up with lots of sheets of paper that weren't the version you were currently writing. You might put those sheets in a folder so that you can go back and see old drafts of your story.

You give that folder to your mom to keep track of, because you know that you can always ask mom for one of the versions of your story if you need it. Plus, mom keeps photocopies so she can give versions of your story to your siblings. Mom also keeps similar folders for the stories that your siblings write.

Git is the folder, and your pieces of paper are code. Your mom is GitHub.

Notice that in the story above, technical terms aren't integrated into the story, and the list of actual terms is drastically simplified (and could be omitted entirely without negatively impacting understanding).

#EXPLAINLIKEIMTEN

Ten-year-olds are ready for terminology, but they can lose a thread if you digress too much or get too technical. You can still succesfully employ an analogy, but they also tend to enjoy more integrated references to technical terms. Thing Explainer by Randall Munroe is another approach that works - explain how a thing works using universal words and concepts, because at ten, people tend to understand all of the truly important concepts already.

Think of computer apps as stories. Say you are writing a story and you want to do a really good job. You might write several revisions of your story. You don't want to lose your revisions though - what if you had a good idea earlier that you want to revisit later? So you decide to keep all of your revisions in a folder. You can think of git as the idea of folders that hold revisions, and your repository (or repo) is the folder that holds this one story you're writing, and all of its revisions.

You might want to share the story that you're writing! Maybe you want to share the writing with friends, or maybe you just want a place for folks to go to read it. GitHub is such a place - a web app that holds your repos, and gives you a web site for them to share. From GitHub, anybody you allow can get their own photocopy of your story, and make changes to their copy. In git that's called a fork. If the person who forked your repo wants to share their changes with you, so that you can have them in your repo, they can give you a pull request on GitHub - a bit like handing you a version of your story with changes in red, and asking you to use that as your story instead.

#EXPLAINLIKEIMFIFTEEN

At fifteen, people are capable of truly excelling in areas that interest them. They can handle terminology and abstract concepts: metaphor, while helpful, isn't a tool you need to lean on quite so much. But you might still be dealing with a short attention span, so keep your descriptions on-topic and don't concern yourself too much with asides.

Say you are writing some code, but you worry that when you change your code you might want to undo your changes. So you need a version control system that you can tell "I've made some changes, please remember them". Git is an example of a version control system. Your repository or repo is your code, which you check in or commit to your version control system as you work. Later, you might create other git repos for other code projects you're working on.

GitHub is an app that gives you a web site for your repo. Others can go to this web site, and download your repo to run the code, or make their own changes in their own copy (if they want to!). The idea of everybody getting their own copy of your repo and being able to make their own changes to it is called distributed version control. GitHub gives you the ability to manage pull requests. Pull requests are changes that other people have made to their copies of your repo, that they'd like you to include in (or merge into) your copy.

#EXPLAINLIKEIMTWENTY

For a twenty-year-old, you can lay out all of your terms once, and then explain the concept using those fully-explained terms. A twenty-year-old can be expected to look up words in the dictionary, or check out concepts on Wikipedia, without needing prompting, so you don't need to belabor your explanations. Your job is to explain how those terms relate to each other. You can also diverge a bit into related topics without worrying that the person will lose track of the main thread.

Git is a version control system (VCS), similar to CVS, SVN, or Mercurial. A VCS is a system for committing code changes to a code repository (repo) in an append-only fashion such that at any given point you can see the history of all changes made to the code up to that point.

Git is a distributed version control system (DVCS), meaning that if multiple people are working on the same code project, they each have an independent copy (a fork) of the repo, which can diverge from other copies. Changes from other copies (called remotes in git), are merged into a given repo through a text-based differential comparison algorithm. Changes in a given repo can also be branched and then merged later, to give the developer an opportunity to work on specific ideas without the pressure of committing them to the main branch of the repo. Mercurial is another DVCS, whereas SVN and CVS rely on a central "master" code repo on a server, which is typically referred to as a trunk.

GitHub is a web app that provides a social network around git repos. GitHub will give each repo a unique URL, which can be used for forking or downloading a repo. GitHub also provides pull requests, wherein a developer can fork a repo, create changes, and then request that those changes be merged into another fork. GitHub provides other services such as issue tracking, wikis, visualizations of code history, and more. People often use services such as GitHub or Bitbucket because they don't want to have to manage their own git server, or because they want the extra features such as pull requests or issue tracking.

Notice that this version is the first time that alternatives to Git, such as SVN or Mercurial, even get mentioned. I'm assuming that a twenty-year-old will want to do further research if they're interested, so providing bigger context such as naming alternative approaches can be very helpful toward that end.

Conclusion

Explaining things to people is fun! It is a good test of your personal knowledge on a topic, and it is also deeply rewarding to share knowledge with people who want to learn. Keeping your audience's abilities and desires in mind drastically increases the likelihood that what you say will be received and absorbed.

Do I think you should start using the hashtag #explainlikeimten or #explainlikeimtwenty just to be more precise? No, I do not! Please don't do that! I love that we have the one hashtag here which generally means "I'm new to this and would like some help." This was just a fun thought experiment meant to shed some light onto my own teaching process. I'm hoping that posting it here will be inspiring in some way!

Who have you found (here on Dev.to, or elsewhere) that's good at explaining? What is it that you like about their explanations?