DEV Community: Logan McDonald

Slack dorking for the win!

Logan McDonald — Mon, 03 May 2021 00:00:00 +0000

A while back I was on the DevDiscuss podcast to talk about being an SRE. At some point I made a comment about a pet peeve of mine: When folks post screenshots of some error they are seeing to Slack. I wanted to take some time to expand on that and talk a little bit about how I use Slack to explain why.

Slack is one of the most powerful logging systems I use. When investigating an error or incident, I’m probably using Slack search as a first move. Often I’ll find the error we’re seeing in the backlog of our chats and it often leads to valuable clues — “oh something similar happens every thrid Monday of the month???” When I find past mentions of the bug or error we’re currently seeing, I have a load of context to give me even more clues. Who worked on this last time? What was the content of the conversation? Even if I can’t figure out what’s happening from the previous chat, maybe I can contact the people involved previously.

Slack search is hugely helpful even if there isn’t something I’m actively trying to debug. There are search modifiers Slack provides that can help us find information more easily. For example, say I’m looking for any documentation our team has on our Fastly SSO because I need to add a new user, and I know that historically my teammate Steven has owned our integration with Fastly. Before pinging him, I can search from:@steven fastly login in Slack. I’ve now saved myself and Steven some time!

Other modifiers, aka “dorks”, I’ve found useful:

Searching DMs for a message someone sent me but I don’t remember who: to:logan that thing
Narrow your search to a channel or DM using in: — probably my most common dork.
Use before:, after:, on:, or during: to do time based searches. For example, if you remember an incident from March but don’t recall what: during:March in:opsalerts feature flags outage
Use has: to retrieve messages that have a specific emoji reaction. This is especially helpful if you also have any automation that adds emoji reactions to particular messages. For example we tag deploy messages with :rocket: so I could search: has:rocket in:infra on:5/1/2021 rig_controller_api to find anything that deployed mentioning that service on May 1st.

Like many organizations, my work has embraced a lot of ChatOps for organizing teams, incidents, and work in general. We send all of our alerts to Slack, work on RFCs, and keep a ton of our context there. With all that in mind, do you understand why screenshotting (i.e. destroying) context frustrates me 😬 ? I’ve actually gotten in the habit of describing what is in the screenshot in a threaded comment response when I see them, so at least now it is tagged with searchable content.

I hope this post inspires you to help an operations engineer out and copy and paste your errors, or at least add as much context as possible when posting screenshots. You are doing your team and your future self a huge favor.

What's your dream technical interview process?

Logan McDonald — Wed, 02 Oct 2019 21:28:18 +0000

We all know that the interview processes in tech can be pretty broken. But what's your dream process? Besides just being handed the job, obviously. What do you think would show off talents and skills necessary for the job you'd want and would be a fun and informative process for you?

I'll start. This would be for an infrastructure engineering position at a mid-to-senior level. My dream process would conversation with recruiter + manager to gauge fit for the team. Then, a time-boxed small code challenge in a language of my choice, anonymized. Then, I'd like to have a series of short-answer questions, also anonymous, that have to do with my specific field. This would be things like "Explain load-balancing", "In your experience, what are pros and cons of microservices", "How does an identity aware proxy work?" If it could be tailored to be resume, that'd be great!

If I passed those, then it'd be an "on site" half day of 4-5 interviews with the team. I want to meet people I'd be working directly with and mostly doing deep technical dives into my past work and comparing it to their specific challenges. I'd also like to meet with leader of the org, either CTO or VP to ask some questions about culture. I don't want to do any puzzles not related to the job. I don't want to whiteboard unless it is for either party to explain the architecture of a previous project better. I want to prove technical skills by thoroughly talking through problems I've solved in the past, though I'm willing to bring code from those projects if necessary.

So, what's your dream process?

How BuzzFeed’s Tech Team Helps Journalists Report On Technology With Authority

Logan McDonald — Wed, 25 Sep 2019 19:26:47 +0000

Reporter: Is anyone around to take a quick look at some code on codepen and help me understand what it does?

This question — and many, many others like it — arose from an initiative called “The Tech + News Working Group,” a BuzzFeed News collaboration between reporters and nearly every role across the Tech department — including SREs, SWEs, data scientists and product managers.
The BuzzFeed Tech team is afforded “7% time,” or half a day each week, to learn something new. For those of us on the Tech side of the working group, that 7% time is dedicated to bringing our technical expertise to the newsroom.

A primary value of BuzzFeed Tech is fostering an experimental and collaborative workplace; One way that’s come to life is through the Tech + News Working Group. The foundation for the group was laid by former BuzzFeed Tech SRE Sri Ray, who provided countless acts of support to the newsroom throughout his time here. Reporters began reaching out to people on the tech and IT teams to help understand terminology, dig into tips, or verify the security of a file they had received from tipsters. The partnership allowed the Tech team to learn from reporters and consider the wider implications of our work. In turn, as insiders in the tech industry, the newsroom benefitted from our tools and perspective while reporting on technology, security, and business stories.

Like other award-winning newsrooms, BuzzFeed News has a dedicated data journalism team that uses data and programming to aid in their reporting. But a year ago, we realized that there is a place for the Tech team to help with reporting, too.

We decided to expand and formalize our collaboration, with an open Slack channel that includes members from both our Tech and News organizations to support day-to-day consulting on stories and work on larger ongoing projects, along with a new mission statement:

Our mission [for the Tech News Working Group] is to act as an elevating force for the newsroom’s work through guidance and assistance using our technical skills. Our News team at BuzzFeed strives to shine light on the most important issues facing us today. Currently, this means a growing focus on the impact of the internet and technology on a range of topics covered by News. As a team of technical experts, we are uniquely equipped to assist in this realm.

Since then, we’ve worked with the newsroom to develop reporting tools, including a project called Tubeviewer which took us down the YouTube rabbit hole. Through Tubeviewer, we found that the Up Next algorithm occasionally pushes users toward hyperpartisan videos that include divisive, conspiratorial, and sometimes hateful content. Tubeviewer simulates the viewing experience of real humans watching YouTube videos given a certain set of search terms. Turn it on, walk away, and come back to a full picture of your viewing experience. We’ve used tools like this to audit algorithms across tech platforms. We’ve also developed bots that send digests of patent information to Slack or collect data from filing systems like NYSCEF. We’ve developed patterns for doing network analysis through Wireshark and done visualizations for articles with the data.

We’ve received hundreds of requests ranging from assistance with finding data from a website to explaining how authentication tokens are used:

Reporter: working on a story about token based authentication vulnerability in a video game that uses google/fb sign on — and looking for someone to comment on the limits/risks of using token based authentication in general

Sometimes we get requests for help with something that would require a lot of manual labor, were it not for a quick and easy tech solution:

Reporter: this is a weird request
that i apologize for
but can someone count the number of stories under this tag
if there is an easy way to do that…

Engineer: can do with JS, count the number of item-list classes

When we created the group, we didn’t know how popular the channel would be, but these requests are a fun break from our day to day “real” work. Sometimes the Tech team gets way more invested in the answer to a question posed by a reporter than the reporter intended. Once, we spent an embarrassing amount of time trying to determine what software Jessica Simpson used to create the graphic for this tweet (loads of EXIF digging later, we still couldn’t figure it out and the reporter had completely moved on). In the end, we’ve helped contribute to a bunch of stories and scoops, and in doing so created a bond of trust amongst reporters and technologists across our organization.

Reporter: I credit this scoop to asking the group here, so thank you

The mentality of a good engineer is similar to that of a journalist; both need to ask good questions and seek out accurate answers with persistence even when there might be bugs in a system. Over the past year, it has been fun and rewarding to build a stronger bridge between the News and Tech teams at BuzzFeed, and we only expect this work to continue. Through the collaboration, both teams thrive: members of the Tech team have the privilege of working hand-in-hand with journalists, while the newsroom is able to report on technology with more authority.

Original post was published on Tech @ BuzzFeed publication on Medium, September 4, 2019.

Working with journalists makes me a better engineer

Logan McDonald — Wed, 04 Sep 2019 18:47:27 +0000

Note: If you aren’t very familiar with my place of work, BuzzFeed, you may not know that our newsroom, BuzzFeed News, produces fantastic journalism. This isn’t a post about proving that point, so if you are thinking “Heh, BuzzFeed? News?” please first check us out 💖. Ok, read on.

I've worked at BuzzFeed as an SRE for over a year and a half. In that time, I've also worked with our Tech + News Working Group (TNWG) with BuzzFeed News. At BuzzFeed Tech, we have "7% time" to spend on any learning activity we choose, and I spend that time working with our newsroom.

We created the TNWG to pursue a mission:

Our mission is to act as an elevating force for the newsroom’s work through guidance and assistance using our technical skills. Our News team at BuzzFeed strives to shine light on the most important issues facing us. Currently, this means a growing focus on the impact of the internet and technology on a range of topics covered by News. As a team of technical experts, we are uniquely equipped to assist in this realm.

Over the last year, we've consulted with journalists on a variety of stories, and even built tools that made a few of those stories possible. In one instance, we built a bot that watches YouTube based on a set of search terms and returns a report on all the data with all the data from its trip down the recommendations rabbit hole.

While our mission and work concentrates on what the tech org can offer the newsroom at BuzzFeed, this post is about the ways the newsroom has helped make me a better engineer. I have had the privilege of observing our reporters at work and many of those lessons carry over to my job as an SRE.

Communication communication communication

When it comes to explaining complicated topics succinctly and clearly, no one has a harder and higher risk job than journalists. Often it takes our reporters just minutes to notice a breaking story and write a polished 200-word take. In these moments, they’re deploying years of hard won experience and skills. Furthermore, what’s impressive about these communication skills is that they’re balancing responsibilities among multiple reporting desks (breaking news, tech, politics, etc.) while also ensuring they meet a rigorous level of journalistic standards and ethics. The risks are far more acute than those we face on the engineering side, where colossal disasters do not generally ruin careers.

As an SRE, I'm often in a similar situation, except we face time-sensitive issues across products. If an incident happens, we have to notify a host of interested parties and we have to do it in a consistent and responsive way. When facing an incident, journalists have taught me to:

Clear out extraneous noise
Quickly identify the point people necessary to solve our problem, and
Practice our communication skills and communication plans in advance

Skills gleaned from my pals in news have helped make me and my team get better at communicating calmly, clearly, and efficiently. We work to maintain a shared language as a team, and an understanding of what we'll do if something unexpected happens.

Research is a long game

The best journalists are world-class researchers, and furthermore, expert open-source intelligence (OSINT) analysts. I've picked up lots of tips and tricks from talking to reporters about how they search for information online. When it comes to OSINT, engineers have tips and tricks of our own. Sharing these insights with one another has brought TNWG members great joy whether it’s explaining to reporters how they can run an OSINT software on their own through Docker or a journalist flagging a vulnerability in our system that they saw through their research.

The main lesson I've learned from the newsroom is that research is a long game. Not every story is going to work out, and many of the leads I've seen our journalists pursue have turned out to be dead ends. Thankfully, our news team tends to be quite good at recognizing a dead end and avoiding the sunk cost fallacy.

In my experience, we on the tech side of the TNWG seem to have a harder time halting effort in the face of unlikely returns. A reporter will come to us with a question and we'll run with it. We'll answer the question, get invested, and keep pursuing the angle. Soon after, the reporter will realize we’re at a dead end and move on to another question or story, but there are often engineers still doggedly looking at the question days later. I've learned from journalists that when a trail runs cold, it’s better to turn around than drive down the road of diminishing marginal returns. Document what you've learned and where the trail went cold, and move on.

We struggle with this immensely in our software projects. You get really invested in a piece of technology or a certain solution that it feels like there is no way to move forward unless you are on that path. This is not true. There's always another solution, another technology, and learning how to admit the trail's run cold is an immensely valuable skill.

Just talk to people

Possibly the hardest thing to imagine myself doing when I watch reporters work is that actually talking to people, like, on the phone. Reporters are constantly talking to contacts within governments and companies, keeping in touch with sources, or chatting with other reporters through text, in person, on the phone, etc. All the time. When digging into an issue with a reporter, the first time they said, "Well, why don't we just ask them if this is what's happening" I thought, "Wait, you can do that?"

Working with journalists has helped me come out of my shell a little bit in terms of my willingness to talk to people. It has also impressed upon me the importance of maintaining good relationships with everyone I meet in the tech community. I try to be kind to everyone regardless, but I think this impressed upon me the value of networking, especially in open source communities. While reporters are often seen as antagonistic or single-minded, good reporters know how carefully they have to treat relationships with their contacts and potential sources. Similarly, in the software world someone is far more likely to help you fix a bug or solve an issue if they know you.

Striving for truth isn't finding answers, it's asking questions

This is by far the most important thing I've learned. When looking into any given lead or scoop or topic, it begins with asking questions and more questions and more questions. Often the best reporting is just putting questions out there – why is this technology monetized this way, who funds this campaign, why was person X here at this time, why is the court system set up this way in the first place? I've seen reporters ask questions that I would never think to ask. These questions don't always uncover surprises or elicit confessions, but sometimes they uncover something where no one thought to look.

As a developer, a reorientation towards questioning is crucial. It often uncovers more bugs than a straightforward read of a pull request, for example. It leads to more creative solutions for tough problems. It pulls me out of biases towards certain technologies. It makes it easier for everyone in the room to understand what's going on. Focusing on the questions and coming up with the most creative questions makes your team more imaginative.

Finally, working with journalists gives you Range

I recently read David Epstein's book Range: Why Generalists Triumph in a Specialized World. In the book Epstein argues that by generalizing and studying many different ways of working, we set ourselves up for success in our specialty. From the book:

Modern work demands knowledge transfer: the ability to apply knowledge to new situations and different domains. Our most fundamental thought processes have changed to accommodate increasing complexity and the need to derive new patterns rather than rely only on familiar ones. Our conceptual classification schemes provide a scaffolding for connecting knowledge, making it accessible and flexible.

This is why I love working as an engineer on a platform that supports non-engineers. I get to talk and work with people who do things very differently from me every day. There is so much we can learn and apply from fields apart from our own and I think that's been especially true of my time working with our newsroom. I get paid to write code, but coding itself is such a small part of my job. On the other hand, the ability to communicate what's happening in my code, the ability to research how to express something I've never done before in code, the ability to reach out to people who also write code, and the ability to ask good questions about code? These are all invaluable skills that make my work better. These are all skills I am still learning, and I'm grateful that I get to learn them from the best of the best: Journalists.

Splash photo source: Unsplash

Automating delivery of US patent application alerts to Slack

Logan McDonald — Tue, 05 Mar 2019 02:37:36 +0000

Recently, I've become obsessed with United States patent applications. Specifically, I'm interested in patent applications from large tech companies. I started reading about patents when I came across some patent applications from Microsoft in an article about facial recognition software. I found the dry and legalistic nature of the patent language fascinating, given the implications of the subject matter. So I googled "how do I find out what patent applications are released" and I found a glorious piece of software: The USPTO's Patent Application Alert Service. Now, every week, I sit down with a cup of coffee and read through patent applications.

In this post we'll cover why I do this, what this USPTO service does, what patent databases are available to the public, and what I've done to make the release of patent applications more accessible for my team.

What are patent applications and why are they important?

Newsworthiness of patent applications

When patent applications are released to the public, they often result in news stories. Here are just a handful I found while researching this:

As you can see, patent applications contain useful information for the public and reveal what companies consider to be innovation worth protecting.

Importance of patent application alerts

To persuade you that patent application alerts are important, I need to talk a little bit about how patent applications work. When a company like Facebook or an individual person believes they have invented a novel "process" or "machine," among other things, they can apply for a patent to prevent another entity from making, using, or selling their invention without their consent. In order to obtain a patent, companies are forced to make a tradeoff. While they stand to make money by protecting their idea, they also have to disclose that idea to the public when they apply for a patent.

By law, patent applications must be made public within 18 months of being filed. However, there are few exceptions. First, there is something called a "provisional" patent application. A provisional patent application allows an applicant to establish an early filing date, but the USPTO does not make it public or determine whether to grant a patent unless the applicant files a regular non-provisional patent application within one year. If the applicant doesn't file a non-provisional patent, the provisional application is never made public.

There are two rarer examples of non-published applications and issued patents. First, there are those filed under foreign patent protection. Second, the USPTO may determine that an invention has implications for national security. Those examples aside, in general, after a US patent application is filed, it will be published by USPTO for the public to view (regardless of whether it has been granted yet).

USPTO databases

So, you may be thinking that means the Patent Application Alert Service (PAAS) spits out an alert for every non-foreign, non-provisional, and non-national-security related patent application before a patent is granted, but this isn't exactly true. Applications that come through the application database will result in alerts to PAAS. However, in some cases, a patent may be approved before the 18 month publication deadline.

There are two major public databases for doing full-text searches for information related to patents: AppFT and PatFT, which contain information for patent applications and patents that were approved.

There are thus applications that are published and indexed in AppFT that will eventually be granted, and those that may never be granted. However, all patents are published and indexed into PatFT (except the ones with national security implications). Thus, those applications published prior to grant and then eventually granted live in both databases.

Now you have enough background to evaluate what alerts coming through the application alert service mean:

It doesn't mean that the application is granted (and it may never be granted)
You're missing those patents whose applications were not published prior to being granted, and thus only exist in PatFT
You're missing provisional patents, some foreign related patents, and those patents that have national security implications

However, dealing directly with AppFT still gives us a lot to work with. With that in mind, let's discuss the alert service.

The Patent Application Alert Service (PAAS)

PAAS is the system that provides customized email alerts when a patent application is published. It was created to allow the public to monitor the latest patent application releases. You can use the service to add alerts for a variety of search terms.

While these alerts definitely make it easier for reporters to report on companies' patent applications, the central reason for a patent alert system is to spur innovation. USPTO thinks that if people have easy access to patent-related information, they will file higher quality patent applications and pursue novel inventions. This is because patent filings require you to prove your patent is novel using "prior art". "Prior art" is anything that can be used as evidence your invention is already known. USPTO thinks that if it can help you find "prior art" that shows your invention isn't novel, you won't apply for a patent. The more visible patent applications are, the more discoverable the "prior art".

I signed up to start getting these alerts a couple months ago when I found out about PAAS. I included the top 10 tech companies in the US as well as some ones that were recommended to me like Palantir or Clear, just to see what they were like. This is the interface for adding a new company to your alerts:

I get the best luck with just searching for terms and selecting "Applicant". All the other categories will give you too much noise.

I love reading the patent applications. To me, they seem like an unfiltered look into what these companies were working on in the language the engineers would use to describe it. Interestingly, there seems to be a disconnect between the engineering teams building these products and the PR teams who BuzzFeed reporters usually hear from. For example, while Amazon PR downplays the extent to which their facial recognition technology could be used by law enforcement, the language in their patents revels in those possibilities. This is because the incentives of the IP lawyers and the PR departments are misaligned. The lawyers' and engineers' goal is to get the patent approved, not to write the patent in such a way that hides a technology's dystopian features.

If you are thinking of working at a large company, read their patent applications. It'll give you a great look into their work.

A patent application Slackbot

I almost immediately started annoying my Twitter followers with patent-related tweets once I started getting these alerts.

EYE TRACKING USING TIME MULTIPLEXING: "system implements time-multiplexing by configuring a source assembly comprising a plurality of light sources to project at least a first light pattern towards the user's eye over a first time period"

again very cool but very creepy pic.twitter.com/4a40xToGe6
— Logan McDonald (@_loganmcdonald) February 7, 2019

Are they building Hunger Games???

"The module may [...] analyze the metrics data to generate game inputs based on the participants' physical metrics, and provide the game inputs to the game system to affect game play" pic.twitter.com/BIBQkPWwOL
— Logan McDonald (@_loganmcdonald) February 14, 2019

You may notice that the tweets above and the stories I linked earlier are all on Thursdays. PAAS releases all new patent applications to the public on Thursdays around 7am EST. Each Thursday, I get a new round of alerts from the companies to which I subscribe. This is what the looks like:

I wasn't super happy with the formatting of the email. I'd have to go through and click on every link, and go out to its individual page to read about it. So I had a thought: What if I could turn this email into data that I could display the abstracts and titles only? What would that look like? My goal was to make PAAS more accessible for people in a hurry to discover the most interesting applications right away.

I looked to see if anyone had built something that accomplished this goal on Github, but could not find anything. Furthermore, while an API exists for searching the actual patent database, none exists for querying recently updated applications. Lastly, I recognize that patents are a big business. Proprietary software to parse new applications probably exists, but I don't have access to it.

So I set out to make a new Slackbot. My general design of the Slackbot relies on incoming webhooks to Slack channels, sending the data to these webhooks when we receive the email from USPTO.

SES and SNS

The first question is how we get the data from emails to our Slack webhooks/channels.

I already have a DNS entry setup through Route53 so I thought it'd be fun to try out AWS's Simple Email Service (SES) to make an email for that entry that I could use to forward USPTO emails to. You have to set up email receiving by defining "rulesets" for emails within your account. I set it up to receive UTF8 emails to a patents@mydomain.com email.

Then, I hooked up that SES ruleset to a Simple Notification Service (SNS) action, which forwarded the emails to an SNS subscription called patents.

Then I went and signed up for PAAS using that email, and received the email on the AWS side through SNS and used the link they provided to confirm my email. Great! Now we had SES -> SNS and then we just needed to hook SNS up to our Slackbot. So there are a couple ways we could go at this point. We can either use https to forward the data to an endpoint as a POST or we could setup a lambda function that gets the forwarded data and forwards it to Slack. I decided to setup an endpoint because I didn't want to setup deploying the lambda but either would work.

Slack applications

Slack has great documentation on setting up incoming webhooks. I created a Slackbot called "PatentsBot" and gave it permissions to post to channels through certain webhooks.

I decided to set up a different channel per company I wanted to follow so I have something like:

#patents-uber
#patents-amazon
#patents-facebook
...

and expect my code to parse and understand where it should send any new applications that arrive for those companies and get them to the right channels.

Now we just needed to receive it, parse it, and forward it to our existing webhooks.

Parsing emails

Before this project, I didn't have experience with receiving and parsing emails with code so I was able to learn a lot about email encoding and the various options available for email parsing in python.

If we look back at the email above, we can see that we are given a list of ids which are hyperlinks. The links don't make a ton of sense (i.e. aren't formatted with consistent routing):

http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PG01&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.html&r=1&f=G&l=50&s1=%2220190065445%22.PGNR.&OS=DN/20190065445&RS=DN/20190065445

... great, right? However, if you click through the link you'll find a nicely formatted HTML document that looks like this:

This was enough information, formatted in ~~tolerable~~ parsable XML. I decided to pull all the hyperlinks from the email, load their associated HTML, parse out the abstract, filing date, and company name, and then forward that data as a single Slack message, rather than trying to parse out meaningful information from the email alone.

This is the code, a build_slack_msgs method, that does the parsing to build messages we'll send on to Slack. This is receiving json objects from SNS, forwarded by SES, finding all the links that being with the AppFT URL signage, http://appft.uspto.gov., requesting data from each link, parsing that data, and returning some metadata.

message = data['Message']
json_file_parsed = json.loads(message)['content']
parsed_email = email.parser.Parser().parsestr(json_file_parsed)
links = []

# Let's pull out all those nasty links and collect them
for part in parsed_email.get_payload():
    links += re.findall(r'\<(http://appft.uspto.gov.*)\>',quopri.decodestring(str(part)).decode('utf8'))

message_data = []

# Then we parse the information we can obtain from those links
for link in links:
    html = urllib.request.urlopen(link).read().decode()
    html_tree = lxml.html.fromstring(html)
    title = ' '.join(html_tree.xpath('/html/body/font')[0].text_content().strip().split())
    # regarding the following line... I'm sorry, but it works
    applicant = html_tree.xpath('/html/body/table[3]/table/tr/td/table/tr[2]/td')[0].text_content().strip()
    date = ' '.join(html_tree.xpath('/html/body/table[3]/table/tr')[3].text_content().split())
    abstract = ' '.join(html_tree.xpath('/html/body/p[2]')[0].text_content().split())
    slack_webhook = find_slack_webhook(applicant) # method to retrieve our Slack channel from secrets
    if slack_webhook:
        message_data.append(
            {
                'applicant': applicant,
                'title': title,
                'link': link,
                'date': date,
                'slack_webhook': slack_webhook,
                'abstract': abstract
            }
        )

There are a couple more special packages I want to point out I'm using to parse the emails from PAAS.

The first is email's parser. This package provides a standard parser that understands email document structures, including MIME documents. parsestr takes a string object and then reads the data from the object given.

The next package is quopri. quopri performs printable transport encoding and decoding for MIME (Multipurpose Internet Mail Extensions) documents, which is the encoding used in email. It takes the email object and decodes it.

Finally, once we have our links list that we've parsed out from the email, we'll use lxml to parse the html associated with those links. We give it our decoded string of html we get as a response from requests and it returns a nicely formatted tree we can use to grep on html properties.

After we've formed nicely formatted data blobs about our various patent applications in the email, we format the messages to deliver them to slack:

msgs = build_slack_msgs(data)
for msg in msgs:
    if msg['slack_webhook']:
        Slack_data = {
            "attachments": [
                {
                    "fallback": "Patent filing.",
                    "color": "#d32600",
                    "pretext": "New patent from {}".format(msg['applicant']),
                    "title": "{}".format(msg['title']),
                    "title_link": msg['link'],
                    "fields": [
                        {
                            "title": "Date filed",
                            "value": "{}".format(msg['date'])
                        },
                        {
                            "title": "Filing abstract",
                            "value": "{}".format(msg['abstract']),
                            "short": False
                        }
                    ],
                    "footer": "PatentBot"
                }
            ]
        }

        response = requests.post(
            msg['slack_webhook'],
            data=json.dumps(Slack_data),
            headers={'Content-Type': 'application/json'}
        )

And that completes our design:

PAAS service -> SES email -> SNS subscription -> https endpoint POST -> Slack incoming webhook POST

When we put this all together we get this:

Easily readable, automated patent alerts sent directly to our team Slack channels! Users in Slack can now subscribe to exactly the channels they want. Then, there are three simple steps to extend this code to new companies to follow:

You add a company with USPTO alerts by logging in to USPTO and selecting "add an alert"
Edit our secrets file to add another Slack webhook
Add the logic to find_slack_webhook to specify which keywords will identify the channel

Currently I have about twenty channels setup in a test slack, while I only subject my coworkers to about six of them 😅.

Future Improvements

We've been happy to get patent alerts for a variety of companies for the past month, but there are always ways to improve.

We should look into incorporating any of USPTO's APIs. I would like to query the PatFT database for information we might be missing about patents granted before the 18 month publication deadline. We could possibly use USPTO's PatentPublicData API for that.

Another API, in beta, that I'm interested in is the USPTO action rejection API, which allows for retrieval of rejected applications. It would be great to keep track of those applications accepted or rejection as follow-ups.

As always, I am open to suggestions for new avenues of research for getting updated patent information as fast as possible. Also, if there are any lawyer-devs out there, please let me know if I could expand on anything or got anything wrong. I have no legal background but am eager to learn!

Shoutout to @frewsxcv for reviewing the code here with me and debugging MINE decoding. Thanks to @mkaemingk for editing and @chrswng for chatting with me about patents.

Thanks for reading!

Questions I ask in SRE interviews

Logan McDonald — Sun, 03 Mar 2019 14:07:38 +0000

SRE, or Site Reliability Engineer{ing}, is a software discipline that applies software engineering to infrastructure and operations. Other SRE-like roles are Operations Engineering, Systems Engineering, or Infrastructure Engineering, but SRE tends to focus more on "resilience" of the operations of software systems. This post doesn't broadly cover SRE, but you can read more about it in my favorite SRE-intro post here.

I was recently asked by a friend who has been interviewing for SRE jobs for good questions to ask at the end of the interview and was surprised to not find any infrastructure-specific posts on questions to ask interviewers. In writing the email back to them I realized I had lots of ideas and it might be worth it to share a short post for them, my future self and anyone else who is interested.

These are the questions I make sure to ask when interviewing for positions on infrastructure or site reliability teams. Depending on the structure of your interviews, these may be appropriate for the recruiter, but are most likely questions for the hiring manager or other members of the team. Most of this centers around where the team is at with their tooling and how happy developers are. I'm also trying to get at the relationship between the operations engineering team and the rest of the team and org.

SRE related questions

What is the tech stack?
- I'm looking at the question from an operations perspective. Are they using a hodgepodge of languages or is the development flow opinionated? How many different technologies does the team have to support?
What is the infrastructure stack?
- Depending on what they say, we'll be talking about this for a while and will probably create a lot of other questions.
What does your metrics & monitoring setup look like? How do you debug issues with the system?
This may be a controversial one, but if the title is "SRE" I ask why the title is "SRE" and not something else (same for "DevOps"). I'm looking to see if they're being thoughtful about what the term means and how they are defining "resilience" for their systems.
Walk me through the experience a developer has on-boarding to your development environment. How long do you think it takes?
Walk me through the experience a developer has deploying with your pipeline. What would you say are the biggest pain points?
- How would you rate test coverage and do you continue to measure that? What about test coverage is important to the team?
- Do you have blue-green deployments? Do you have canaries?
- How do engineers share their work with product teammates in the QA phase? How many environments do you have?
- These questions are incredibly important to me. It could both surface fun red flags for you to discuss with your interviewer and see how receptive they are to your opinions and give you an idea of things you might be working on for them.
How would you describe the relationship between the operations team, IT, and the rest of the engineering team?
How do you handle app security? How do you encourage developers to think about the security of their services?
Do you have to be GDPR compliant? Did that process go smoothly for you?
- This may not lead anywhere, but I'm looking for a discussion about what their data auditing procedures look like, and how easy it is to answer security questions about their data quickly.
What's your on-call set up look like?
- How many times a month are you on-call?
- When you are on-call, how many times during that period are you getting paged?
- Would you say when you get paged, alerts are actionable?
- Are developers on-call for their services?
- How do you on-board people to on-call?
- How much time is spent on the team in "reactive" rather than "proactive" mode?
Are most things in the infrastructure stack self-service? Like, what's the process of setting up a new service with data stores?
Which level of Dickerson's hierarchy of site reliability do you think needs the most work in your stack?
Overall, how would you rate developer productivity?
Do you have any open source projects? If not, are you interested in open sourcing anything?

Standard interview questions

Does your engineering team have a values statement? What's in it?
What do you do to foster an environment of learning?
What does success in this role look like? What sorts of projects or accomplishments could you see being completed 3 months, 6 months, and 1 year out?
Is the working environment collaborative during work or do people mostly keep to themselves? How so? Is the office open? (I would also ask during on-sites that they show you where you'd be sitting. If you're sensitive to lots of noise while working this could be very important.)
How much of the team is distributed? What is your "work from home" or "work from X" policy? Is it flexible or set?
What is the thing you are most excited about working on or launching in the next year?
What do you like best about working here?

I want to point out there are some things I didn't ask here and that doesn't mean that I don't value them. For example, I never ask directly about diversity on the team. I hope that's something that I see around the office and in my interviews. When I go to lunch with one of the developers, I'll ask questions that indirectly get to those issues. I don't really need the PR line about diversity.

What are your questions you always make sure to ask in either general tech or SRE interviews? Let me know!

Image used from unsplash

Homographs, Attack!

Logan McDonald — Fri, 16 Feb 2018 04:15:51 +0000

We've known about Homograph attacks since the 1990s -- so you may be wondering why I'm writing about them in 2018. Don't worry, I'll get to that. In this post, we'll explore the history of homograph attacks, and why, like many of the internet’s problems that stem from path dependence, it seems like they just won’t ever go away.

Origins of my Interest

I first got interested in homograph attacks a few months back when I was working through tickets for Kickstarter's Hackerone program. HackerOne is a "bug bounty program", or, an invitation that hackers and security researchers find vulnerabilities in our site in exchange for money.

When I was looking through the tickets, one caught my attention. It wasn't a particularly high risk vulnerability, but I didn't understand a lot of words in the ticket, so of course I was interested. The hacker was concerned about Kickstarter's profile pages. (We often get reports about our profile and project pages.)

Profile pages often create vulnerabilities for websites. Whenever you are in the position to “host” someone on your site, you are going to have to think about the ways they’ll abuse that legitimacy you give them. Our hacker was specifically concerned about a field that allows our users to add user-urls or "websites" to their profile.

They thought this section could be used in a homograph attack. To which I was like, what the heck is a homograph attack? And that question lead me down a rabbit hole of international internet governance, handfuls of RFCs, and a decades-old debate about the global nature of the internet.

Internet Corporation for Names and Numbers (ICANN)

We have to start with ICANN, the main international internet body in charge in this story. ICANN makes all the rules about what can and cannot be a domain name (along with performing the technical maintenance of the DNS root zone registries and maintaining the namespaces of the Internet).

For example, say you go to Namecheap to register "loganisthemostawesome.com". Namecheap uses the “extensible provisioning protocol” to verify your name with Verisign. Verisign is the organization that manages the registry for the “.com” gTLD. Versign checks the ICANN rules and regulations for your registration attempt, tells Namecheap the result, and Namecheap tells me if I can register "loganisthemostawesome.com". Spoilers: I can!

This is great. But I primarily speak English and I use ASCII for all my awesome businesses on the internet. What happens to all those other languages that can’t be expressed in a script compatible with ASCII?

Version 1 of Internationalized Domain Names

ICANN attempted to answer this question when they proposed and implemented IDNs as a standard protocol for domain names in the late 90s. They wanted a more global internet so they opened up domains to a variety of unicode represented scripts.

What's a script? A script is a collection of letters/signs for a single system. For example, Latin is a script that supports many languages, whereas a script like Kanji is one of the scripts supporting the Japanese language. Scripts can support many languages, and languages can be made of multiple scripts. ICANN keeps tables of all unicode character it associates with any given script.

This is even better now! Through IDNs, ICANN has given us the ability to express internet communities across many scripts. However, there was one important requirement. ICANN’s Domain Name System, which performs a lookup service to translate user-friendly names into network addresses for locating Internet resources, is restricted in practice to the use of ASCII characters.

Punycode

Thus ICANN turned to Punycode. Punycode is just puny unicode. Bootstring is the algorithm that translates names written in language-native scripts (unicode) into an ASCII text representation that is compatible with the Domain Name System (punycode).

For example, take this fictional domain name (because we still can't have emojis in gTLDs 😭):

hi👋friends💖🗣.com

If you put this in your browser, the real lookup against the Domain Name System would have to use the punycode equivalent:

xn--hifriends-mq85h1xad5j.com

So, problems solved. We have a way to use domain names in unicode scripts that represent the full global reach of the internet and can start handing out IDNs. Great! What could go wrong?

Homographs

Well, things aren’t always as they seem. And this is where homographs and homoglyphs come in.

A homograph refers to multiple things that look or seem the same, but have different meanings. We have many of these in English, for example “lighter” could refer to the fire starter or the comparative adjective.

The problem when it comes to IDNs is that homoglyphs exist between scripts as well, with many of the Latin letters having copies in other scripts, like Greek or Cyrillic.

Example of lookalikes from homoglyphs.net:

Let's look at an example of a domain name.

washingtonpost.com

wаshingtonpost.com

Can you tell the difference? Well, let's translate both of these to purely ASCII:

washingtonpost.com

xn--wshingtonpost-w1k.com

Uh oh, these definitely aren't the same. However, user-agents would make them appear the same in a browser, in order to make the punycode user-friendly. But in reality, the first "a" in the fake-WaPo is really a Cyrillic character. When translated to punycode we can see the ASCII remaining characters, "wshingtonpost" and then a key signifying the Cyrillic a, "w1k".

This presented ICANN with a big problem. You can clearly see how these may be used in phishing attacks when user-agents interpret both Washington Post's as homographs, making them look exactly same. So what was ICANN to do?

Internationalized Domain Names Version 2 & 3

By 2005, ICANN had figured out a solution. They told gTLD registrars they had to restrict mix scripts. Every single registered domain had to have a "label" on it to indicate the single pure script that the domain name would use to support it's language. Today, if you went and tried to register our copy-cat Washington Post at xn--wshingtonpost-w1k.com, you would get an error. Note: There were a few exceptions made, however, for languages that need to be mixed script, like Japanese.

Problem fixed, right? Well, while mixed scripts are not allowed, pure scripts are still perfectly fine according to ICANN's guidelines. Thus, we still have a problem. What about pure scripts in Cyrillic or Greek alphabets that look like the Latin characters? How many of those could there be?

Proof of Concept

Well, when I was talking to my friend @frewsxcv about homograph attacks, he had the great idea to make a script to find susceptible urls for the attack. So I made a homograph attack detector that:

Takes the top 1 million websites
For each domain, checks if letters in each are confusable with latin or decimal
Checks to see if the punycode url for that domain is registered through a WHOIS lookup
Returns all the available domains we could register

A lot of the URLs are a little off looking with the Cyrillic (also a lot of the top 1 million websites are porn), but we found some interesting ones you could register.

For example, here's my personal favorite. In both Firefox and Chrome, visit:

https://раураӏ.com/

Here's what they look like in those Browsers.

Firefox:

Chrome:

Pretty cool! In Firefox, it totally looks like the official PayPal in the address bar! However, in Chrome, it resolves to punycode. Why is that? 🤔

User-Agents & Their Internationalized Domain Names Display Algorithms

It is because Chrome and Mozilla use different Internationalized Domain Name Display Algorithms. Chrome's algorithm is much stricter and more complex than Mozilla's, and includes special logic to protect against homograph attacks. Chrome checks to see if the domain name is on a gTLD and all the letters are confusable Cyrillic, then it shows punycode in the browser rather than the unicode characters. Chrome only changed this recently because of Xudong Zheng’s 2017 report using www.xn--80ak6aa92e.com as a POC.

Firefox, on the other hand, still shows the full URL in its intended script, even if it's confusable with Latin characters. I want to point out that Firefox allows you to change your settings to always show punycode in the Browser, but if you often use sites that aren't ASCII domains, this can be pretty inaccessible.

So, what's next?

So what, now, is our responsibility as application developers and maintainers if we think someone might use our site to phish people using a homograph? I can see a couple paths forward:

Advocate to Mozilla and other user-agents to make sure to change their algorithms to protect users.
Advocate that ICANN changes its rules around registering domains with Latin confusable characters.
Implement our own display algorithms. This is what we ended up doing at Kickstarter. We used Google's open-source algorithm and show a warning if it's possible that the url shown on the page is a homograph for another url.
Finally, we could just register these domains like @frewsxcv and I did with PayPal so that they aren't able to be used maliciously. Possibly, if we are part of an organization with a susceptible domain, we should just register it.

To summarize, this is a hard problem! That's why it's been around for two decades. And fundamentally what I find so interesting about the issues surfaced by this attack. I personally think ICANN did the right thing in allowing IDNs in various scripts. The internet should be more accessible to all.

I like Chrome's statement in support of their display algorithm, however, which nicely summarizes the tradeoffs as play:

We want to prevent confusion, while ensuring that users across languages have a great experience in Chrome. Displaying either punycode or a visible security warning on too wide of a set of URLs would hurt web usability for people around the world.

The internet is full of these tradeoffs around accessibility versus security. As users and maintainers of this wonderful place, I find conversations like these to be one of the best parts of building our world together.

Now, we just gotta get some emoji support.

Thanks for reading! 🌍💖🎉🙌🌏

Resources

Background

Browser policies

Tools

Homograph Attack Generator for Mixed Scripts NOTE: It is no longer possible to register mixed script domain names.
Homograph Attack Finder for Pure Cyrillic Scripts
Homograph Attack Finder + WHOIS lookup for Pure Cyrillic Scripts
Homoglyph Dictionary
Puncode converter

ICANN CFPs and Guidelines

ICANN, Verisign, and the Domain Registration Process

Wikipedia for TLD. Each TLD has its own Registry that manages it and defines its IDN rules.
Wikipedia for Domain Name Registry, like Verisign
Wikipedia for Domain Name Registrar, like Namecheap, Godaddy, or Gandi.net
ICANN Accreditation and Verisign Certification for distributing .com domains
Wikipedia for the Extensible Provisioning Protocol, which is used when a user on a registry requests a .com domain. The registry uses the EPP protocol to communicate with verisign to register the domain.
Verisign's IDN Policy. Verisign requires you specify a three letter language tag associated with the domain upon registration. this tag determines which character scripts you can use in the domain. presumably the language tag for https://аррӏе.com/ (cyrillic) is 'RUS' or 'UKR'.
PIR, manager of .org TLDs, IDN rules

Misc Security related to Domains

Extended Validation Is Broken by @iangcarroll

Homograph Major Site Copy-cat Examples

http://аоӏ.com/
https://раураӏ.com/
https://аррӏе.com/
http://www.спп.com/

How I Take Notes

Logan McDonald — Sun, 12 Nov 2017 18:13:43 +0000

Last month I was on jury duty in Brooklyn for two weeks. I learned a lot. However, jury duty is not being offered for the truth of the matter asserted, but merely to show what lead to this blog post. Apologies for the legalese, I was hanging out with lawyers too much!

Anyway, at jury duty we had a lot of free time where we were trapped in a courtroom with no wifi just waiting around for instructions. I decided to take that time to learn something! I downloaded The Rust Programming Language aka TRPL and grabbed my favorite notebook and took them with me to court every day for the last two weeks. On breaks I read chapters of the book and I took notes.

Why Take Notes

Taking notes doesn't work for everyone, but I find it absolutely necessary to retain knowledge. Notes can be taking in loads of different ways and it's important to find what works best for you. Particularly, I make a point to handwrite most of my notes. There is something visceral and tactile about indexing handwritten notes that helps me remember the topics better. My coworkers kid me about when I first interviewed at Kickstarter, I brought a big notebook and took copious notes during all the interviews. I take notes in lots of different situations, but namely when I'm learning something new at work, when I'm at a conference, or when I'm primarily paying attention to someone else talking/writing.

I find several advantages to taking notes. First, it gives me a written record of everything I've learned. If I timestamp it well (and I usually put hours and minutes as well as dates!) I can get a log of my progress. Second, the act of taking notes well helps me remember what I've learned.

Now, let's discussing what taking notes "well" means.

The Cornell Method

I think part of the reason we hate having to take notes is the many wasted hours being forced to take them throughout school. Personally, I always loved to handwrite notes. However, there are conflicting studies as to it's actual impact on memory retention. Regardless of if you take digital or handwritten notes, there are techniques you can use to make your notes more effective.

One that I have loved over the years is The Cornell Method. The idea behind the Cornell notes is that you keep "cues" of the main body of the notes along one edge while you go through the materials. The reason I always loved Cornell notes is because you can make the cues whatever you find helps you remember the material the best, not necessarily keywords given in a textbook or by a lecturer. For example, I have a cues for my Rust notes that reference things in the Ruby API that are more familiar to me, so I can say things "like Bundler" for different Cargo commands, which cue me to remembering the context for the notes.

Here are some cues on my notes from Megan Anctil's 2016 Monitorama talk, for example:

Lessons from Learning Science

Strengthening Recall

A reason that I love Cornell notes' cueing method is it helps with a practice in learning science called recall. The idea behind recall is that rereading information from notes or a textbook are proven to have little effect on longterm memory of the concepts in that material. However, forcing yourself to recall information with flashcards or cues does help. Also, spacing out your study of this material has been shown to help. This is called "delayed retrieval". When we force our minds to stretch for past information with the help of cues, we strengthen those memories to the information.

Low Stakes Testing

Another concept connected with notetaking that's been proven to help learning is low stakes testing. Frequent, low stakes testing allows us to practice remembering without the pressure of a live test. It gives us space to reflect on our mistakes and revist faulty places in our memory of certain content. If we take careful notes of our notes (i.e. cues) we can use these as a way of testing ourselves, which is pretty useful! All I do is go back to my Rust notes every couple of days and cover the side of the paper with the verbose notes and force myself to remember concept only from the cues.

Modeling

Finally, I'm a huge fan of utilizing different sorts of patterns to model complex relationships amongst topics in my notes. Unfortunately for me, I'm not a great artist so usually these come out as squiggly blob drawings. I was inspired to start diagramming more in my notes (and pull requests!) by my coworker Sarah. She has this amazing talk on dev diagrams. Forcing yourself to make models of what you think the connections amongst different concepts are puts you on the road to constructing strong mental models to lean on for those concepts in the future.

A good diagram is a beautiful thing, but in my opinion a series of bad diagrams are just as cool. I love to draw out how I think a system works in my notes only to find it later and be like, "Wow that's not it at all." That's what can also make taking notes wonderful. It's not only a log of your learning. It's a log of your past mistakes. It's powerful because not only is it a source of information. It's a source of your own journey to find that information.

What's your note taking style?

In conclusion, I know it's not for everyone, but learning how to take notes has helped me improve how I learn, which makes me a better developer. Do you take notes? I'd love to hear your note-taking style!

Using Bitwise Operators

Logan McDonald — Sun, 12 Nov 2017 17:10:06 +0000

A cool thing I learned about a couple years ago in school is bitwise operators. I honestly mostly forgot about them until a coworker pointed out an implementation in our codebase the other day. I want to talk about a couple "practical" uses for them in that I've seen.

Before we begin, let's do a quick math recap. An operator is a tool that acts on the operands to produce another output. Famous operators are +, -, *, etc. It is both the element or tool that acts upon operands, and what we call the thing that denotes the action. An operand is the object of the mathematical operation. So in 2 + 2, the 2s are operands, the plus sign is the operator.

Introduction to Bitwise Operators

So what are Bitwise Operators? Well, an operator that operates on bits! Basically, they are operators that move around bits in patterns (like binary numerals) at the individual level. Because they operate at such a low level, they are general very speedy. Remember operators are just like +/plus and -/minus so they also have mathematical symbols and english names. The programmatic or mathematical denotation is going to be different across languages.

I'm going to go through and explain some really common ones. Sometimes I understand code better than English so I've also just coded out in ruby what the logic means, but this is my own pseudocode, not source code! Ok, here are the main bitwise operators:

NOT

Also known as a complement, this operation performs negation on each bit so 1111 would become 0000, 10101 becomes 01010, etc. In code:

def NOT(bit)
  output = ''
  bit.each_char do |char|
    output += char == '0' ? '1' : '0'
  end
  return output
end

AND

This takes two operands and multiplies each bit. So 1111 AND 0000 is 0000 (1*0 across all bits), 10101 AND 00111 is 00101, etc.

AND reminds me of a lot of languages Boolean || operators when false || true, true || true, false || false, the bitwise AND acts in an identical way, but with binary. In code:

def AND(bit_1, bit_2)
  output = ''
  bit_1.split('').zip(bit_2.split('')).each do |arr|
    output += (arr[0].to_i * arr[1].to_i).to_s
  end
  return output
end

OR

This also takes two operands. The OR takes two bit patterns and results in 0 if both bits are 0, or else 1. So, 1111 AND 0000 is 1111, 10101 and 00011 is 10111, etc.

def OR(bit_1, bit_2)
  output = ''
  bit_1.split('').zip(bit_2.split('')).each do |arr|
    if arr[0] == '0' && arr[1] == '0'
      output += '0'
    else
      output += '1'
    end
  end
  return output
end

XOR

This is our most complicated one yet. XOR also takes two operands where the result in is 1 if only the first bit is 1 or only the second bit is 1, but is 0 if both are 0 or both are 1. Ah, confusing. Let's code this.

def XOR(bit_1, bit_2)
  output = ''
  bit_1.split('').zip(bit_2.split('')).each do |arr|
    if arr[0] == arr[1]
      output += '0'
    else
      output += '1'
    end
  end
  return output
end

Ah, that's better.

So these operators are pretty cool! They do simple math on binary and we can kind of see how they might be useful on a theoretical level. Just like we couldn't image life without multiplication, addition, subtraction, and division, bitwise operators are essential to a computer's life. From these simple building blocks, we can make some pretty powerful math.

All programming languages I've seen have bitwise methods built it, but they look a little different than what we've defined here. For example, Ruby bitwise operators:

NOT: ~
AND: &
OR : |
XOR: ^

Then there are also really cool bit shifters in ruby as well: << and >>. These shift the integers bits to the left or right by a given number of positions like

(a >> 2).to_s(2)

where a is a decimal, to_s(2) means to_binary basically, and >> 2 means, shift bits right 2 places. Example in irb:

> a = 64
 => 64
> a.to_s(2) # the binary equivalent of a
 => "1000000"
> (a >> 2).to_s(2) # the binary equivalent of a shifted to the right
 => "10000"

Anyway, we could stay here all day talking about how cool shifting bits is but I want to go into a couple cool use cases or examples I've seen of using these practically.

Tic Tac Toe

Note: In researching for this post I found this brilliant description of using bitwise operators in tic-tac-toe by Joris Labie: Using bit-logic to keep track of a tic-tac-toe game. Make sure to check it out for a more detailed description of how this works!

Building a game of tic-tac-toe is often one of the first intro level tasks a new programmer will be asked to do. It comes up often in interviews or code challenges. When I was in Ada Academy, our introduction to JavaScript was building a tic-tac-toe game in the browser. I enjoyed the project but I had a hard time formulated a good way to make the logic for winning a game. There are a lot of different possibilities for how to win across 9 different squares. Fortunately for me, I was also first learning about bitwise operators at the time, and got to thinking that Xs and Os aren't all that different from 1s and 0s. Perhaps I could use bitwise operator logic to keep track of combinations across the board.

I knew I needed to check every move for a winning score to judge the outcome, so I wrote a method checkEnvironment that would call winningScore each turn which would determine if the player that went last successful created a winning pattern of Xs or Os.

So here's how the winningScore logic worked. It relied on an array of binaryWins or 8 unique ways to win (3 horizontal, 3 vertical, 2 diagonal). These are very special numbers and taking a look at them may help to figure out why.

These are the winning combination numbers in binary:

[111, 111000, 111000000, 1001001, 10010010, 100100100, 100010001, 1010100]

and in decimal:

[7, 56, 448, 73, 146, 292, 273, 84]

And then let me express them one final way, again, in binary, but with some extraneous 0s:

[['0', '0', '0'],
 ['0', '0', '0'],
 ['1', '1', '1']],
[['0', '0', '0'],
 ['1', '1', '1'],
 ['0', '0', '0']],
[['1', '1', '1'],
 ['0', '0', '0'],
 ['0', '0', '0']],
[['0', '0', '1'],
 ['0', '0', '1'],
 ['0', '0', '1']],
[['0', '1', '0'],
 ['0', '1', '0'],
 ['0', '1', '0']],
[['1', '0', '0'],
 ['1', '0', '0'],
 ['1', '0', '0']],
[['1', '0', '0'],
 ['0', '1', '0'],
 ['0', '0', '1']],
[['0', '0', '1'],
 ['0', '1', '0'],
 ['1', '0', '0']]

Look familiar? These binary numbers are an exact expression of the wins!

It might help at this point to revisit our bitwise AND operator, which takes two operands and returns multiplies each bit. This means if you compare any combination that a current player has across the board, with each of the binary wins, any 0 bit in the players combinations, cancels out and becomes 0, but any place where the win's combination is 1 and the player has that 1 will register as a hit. For example, look at me winning diagonally on this board represented as a multi-dimensional array (I'm X):

[['X', 'O', 'X'],
 ['O', 'X', 'O'],
 ['X', 'O', 'X']]

And here's one of the binary wins:

[['1', '0', '0'],
 ['0', '1', '0'],
 ['0', '0', '1']]

If we represent each of the X's on the game board as a 1, even though there are more than the three winning Xs, they'll be canceled out by the AND operator if compared with our binary win!

This is pretty cool! Now that we have our logic figured out, the code is simple (excuse my javascript, it's not my primary language, but this works!):

Early on, set our binary wins:

binaryWins = [7, 56, 448, 73, 146, 292, 273, 84];

Write our winningScore logic:

winningScore: function(score){
  var self = this;
  for (i = 0; i < self.binaryWins.length; i += 1) {
    if ((score & self.binaryWins[i]) === self.binaryWins[i]) {
      return true;
    }
  }
  return false;
}

Each check we iterate through the possible wins, and use our score and the win as operands in the AND operator, i.e.
score & self.binaryWins[i] and return a boolean for if a winning score has been found.

So this is really neat. We've just used a bitwise operator for real! But tic-tac-toe is not business logic (unless you have a really cool tic-tac-toe business, please let me know if you do). Let's explore some other ways we could use these smooth operators.

Boolean or Bitwise Logic?

So we often use boolean logic in conditionals, but bitwise logic is just as generally powerful! These conditionals are often used in assigning colors on the RGB spectrum, for example. We can also assign a range of random elements to different binary numbers and use bitwise operators to compare them. & and | do a good job comparing if two sets have an intersection. ^ can be used to see what is different between two sets. Anytime we have sets of relationships, they can be compared using bitwise logic.

Ruby's Secure Compare

This is the secure_compare method in Ruby:

def secure_compare(a, b)
  return false if a.empty? || b.empty? || a.bytesize != b.bytesize
  l = a.unpack "C#{a.bytesize}"

  res = 0
  b.each_byte { |byte| res |= byte ^ l.shift }
  res == 0
end

secure_compare is a super cool method that protects us from timing attacks, or attacks where an attacker can measure stuff about the response of a site by how long it takes to repsond. When string equality operators work left to right (i.e. user_input == 'string') and a user is inputting something that controls that logic, they can execute a timing attack.

So to prevent this, secure_compare takes hashes of the two strings and compares those instead like:

secure_compare(
  Digest::SHA256.hexigest('a'),
  Digest::SHA256.hexigest('b')
)

If we look at the code about for secure_compare, we can see that it steps through each letter, unpacking the byte size as it goes. It then iterates on those characters byte sizes and uses the XOR operator to compare them. We can see that if two of the bytes in the corresponding hashes mismatch,res is no longer zero and the compare returns false.

Each hash pasted in is of an equal length and XOR is used to compare and find any differences in their binary equivalents. This is not only more secure because it takes constant time to compare the hashes (no ability to measure the time differences with different inputs), but it's also really fast!

Rails bitfields gem

Finally, I want to talk about a really cool gem, bitfields. It's less directly related to the operators but it uses binary logic to store things in ActiveRecord.

Say you have a table, Puppy with multiples columns has_chew_toy, likes_pets, and good_walker, which are all Boolean values. That's a lot of trues and falses for one table that seems primarily based around Booleans. What if we could save one integer to represent all these fields? We can!

The Boolean values are organized like:

true false false => 1 0 0 => 1
false true false => 0 1 0 => 2
false false false => 0 0 0 => 0
true true true => 1 1 1 => 5

And when you instantiate a new dog, you just give it the values, like

class Puppy < ApplicationRecord
  include Bitfields
  bitfield :puppy_bits, 1 => :has_chew_toy, 2 => :good_walker, 4 => :likes_pets
end

puppy = Puppy.new(has_chew_toy: true, good_walker: true)
puppy.likes_toys # true
puppy.puppy_bits # 3

This is pretty cool use of binary logic! And under the surface it's using bitwise operators to create the relationships.

Final Note

I really love learning about bitwise operators because it takes something that seems really scary and illuminates it. Bitwise operators are just like the operators we learn in elementary school, but for a numeric system we aren't familiar with. While I love this implementations, for people less familiar with the operators, the syntax can be really confusing because it looks like Boolean logic but is really different. For that reason, I really like solutions that have comments and are well documented, at least recognizing the use of a bitwise operator in a codebase.

I learned a lot writing this post! If you'd like to explore these operators more in depth I've left some resources below.

Resources

Here's a collection of some helpful resources I used in forming this blog post and some code:

How to Learn

Logan McDonald — Mon, 16 Oct 2017 20:42:19 +0000

For the past several years, I've been thinking a lot about learning science, or the systematic investigation of how we learn. This was fueled by a switch in career trajectories after I graduated college towards software engineering.

There's this really intimidating article that I often hesitantly send people looking to get into site reliability or ops work. It's A roadmap to becoming a web developer in 2017 and lays out a list of topics that one can attempt to learn in various fields of web development when they're trying to break into a field. Here is the graph of things to know for the "devops" path:

(Diagram credit to Github user @kamranahmedse)

Wow, this is making me sweat just looking at it. Needless to say, there's a lot to learn.

In May, I gave a lightning talk at Monitorama on learning and it's connections to my job as an ops engineer: "Optimizing for Learning". You can watch the talk here at 6m7s. The talk is an exploration of why memorization and learning are important whenever faced with a seeming abyss of knowledge.

Software engineering is an incredibly broad field. There are a lot of topics to just know about, before you can even get to mastering them. This is particularly true for systems and ops engineering. Entering this field really overwhelming. In the talk I have identified one source of this overwhelming feeling - Ops is based heavily on expert intuition and experience. Experience takes time, which we often cannot control. However, I decided in the last year to balance this disadvantage with something I have slightly more control over: My learning and memory. I decided to give that talk and write this post because I realized this is a challenge everyone faces. The faster you can learn new things and connect them to what you already know and the problems you’re facing, the more effective you are at your job or starting to build something useful.

Now, learning is incredibly subjective and brains are complicated. No one learns quite the same. However, I have found some patterns that have made me better at it, and I'd love to share them.

Hierarchies of Learning

I start theoretically with some conceptuals diagram that I learned all the way back in the 9th grade, Gagne's hierarchy of learning and Bloom's taxonomy of learning. There have been many adaptations and addendums to these hierarchies since Robert Gagne and Benjamin Bloom came up with them in the 1950s, but I find the general path of learning to be an effective mental constructs for me.

The idea is that learning increases in complexity up a hierarchy and learning can often be more successful when focusing on the lower rungs, listed here lowest to highest. Here's Gagne's hierarchy:

Signal learning
Stimulus-response learning
Chaining
Verbal association
Discrimination learning
Concept learning
Rule learning
Problem solving

And Bloom's:

Remembering
Comprehending
Applying
Analyzing
Synthesizing
Evaluating

You can see we start at the lower levels with stuff like route memorization and pattern matching. This is what I like to call the "copypasta code" level of learning. I want to be clear that I'm not knocking this type of learning or copypasta code. In fact, we'd better not call it "lower" at all -- it's foundational. And many experienced programmers or subject-matter experts will often have to relearn foundational tools to learn.

In both these paradigms, longterm memory building is prized at the foundational level. Built upon these skills, pattern matching and chaining information is formed. Then learning what something is NOT (analyzing, discrimination learning) starts to become helpful. For example, when I started learning a functional programming language for the first time after programming only on object-oriented languages, I was surprised how much it strengthened my understanding of OOP. Finally, when we have a good grasp of the basics, we can start to form mental models or concepts generally applicable across different types of examples. Finally, we can create. Honestly, I think there should be a final item with both of these hierarchies -- opinions. They should be a thing you can hold at the highest rung. Unfortunately, many ranks these much lower in their learning calculus.

While I definitely (and I assume most people do) interact with these hierarchies in very non-linear ways, they can be incredibly helpful when setting out to learn anything new. And they'll often help explain to me why, when learning Go for the first time for example, I couldn't right away start creating or solving complex problems in it.

Over the last year, in order to inch closer towards the top of these hierarchies, I've become reacquainted with some old enemies: quizzes and flashcards.

This is because I realized early on in my first year as an ops engineer that the more I knew about the tools as my disposal, the faster I could recognize the constraints of any given project I was assigned. This is because once you have knowledge (no expertise needed!) you can start to ask better questions. When you have good memory, it allows you to be more creative in solving problems.

I found success learning about the things that inhabit the ops universe in a couple ways:

Practicing Retrieval

Continuous development and testing for your brain!

Simply re-reading material has been proven to not work. However, practicing retrieval of that material can. Retrieval is just recalling something from memory.

Low stakes testing

Try testing yourself after you read something. Retrieval is proven to force the brain to make strong connections to concepts. While short term memory kind of sucks, our longterm memory is incredibly plastic. But don’t test yourself on things right away. Difficult learning leads to stronger memory. By spacing out your retrieval you make your brain strain more.

Reflection

Writing about something after you learn it. Reflection helps consolidate ideas into longterm memory because it links them to experience we already have. It’s as simple as asking yourself what went well, could have gone better, could do next time.

Memory Palaces

I’ve used memory palaces to remember concepts around networking, HTTP codes, linux features, and AWS services.

Builds on a cool feature many humans have called imagination inflation. This is the ability for us to imagine things so strongly we trick our brains into believing they are real and saving them in memory as real.

You just take a place you are already familiar with and place pieces of information throughout it while imagining something visceral or absurd in the place to make you remember. Like tarantulas passing notes in your living room to remind yourself about TCP.

Constructing Mental Models

Mental models are the result of the extraction process where you take key concepts from new material and organize them. As we start to gather information about the ops universe, and monitoring systems for example, we are building up abstractions of those monitoring systems that will allow us to design other types of systems that aren’t exactly the same. Mental models, in other words, are what makes up expert intuition.

Having a Growth Mindset

The final memory tool that has been essential in this year is having, and working with people who have, a growth mindset.

There was a 2010 study done on 5th graders by the researcher Carol Dweck in New York where researchers gave them puzzles. After completing the puzzles, they told them either that they were very smart or that they worked very hard. They gave them another round of harder puzzles. The kids told that they worked hard did much better.

Here's a quote from the study:

Emphasizing natural intelligence takes it out of the child’s control, and it provides no good recipe for responding to a failure.

Learning requires failure. Ops engineering definitely requires failure.

Surround yourself with people who believe this. I am lucky to work with a team the praises each other for working hard on difficult problems, not for being “smart” or naturally great engineers.

If any of these strategies are interesting to you, I took a lot of suggestions from Make it Stick, a book about learning science.

Final Note

As a final note, I want to say that while learning how to learn can be a great way for people new to any topic to level-up, it's also a powerful thing for anyone in a technical leadership position to embrace as well. First, it'll allow you to mentor and train new engineers faster. Second, recognizing the fundamental importance of pattern matching and copying should force us to make code bases, runbooks, and documentation that is kind and considerate towards newcomers. We are all constantly learning across different levels of understanding. We are all trying to take our learning to the level of craft. Understanding how we learn can help not only ourselves learn faster, but be more empathetic in general.

I hope to share more learning tips and tricks as I come across them. Also, I'm using this blog as a sort of longform of reflection and retrieval, to help myself learn as well.