DEV Community: Caleb Collins-Parks

15Five's response to the Log4J RCE

Caleb Collins-Parks — Sat, 29 Jan 2022 05:55:21 +0000

When we learned about the Log4j RCE in early December we quickly patched our servers, working on the weekend the day after the CVE was reported to get it done ASAP before hackers could strike.

We started by escalating the incident to our security officer to get the appropriate attention on it. He immediately stopped the vulnerable server to prevent any exploits. This gave us time to do a thorough review of our services for use of Log4j. We used https://www.techsolvency.com/story-so-far/cve-2021-44228-log4j-log4shell to see what services were affected. Fortunately, only the aforementioned server was affected.

Updating the server was easy thanks to Docker. A quick increment to the tag version and a run of Ansible was all it took. Fortunately our other software was unaffected, so that was all the work needed. In less than a day the server was back online.

To verify that Log4j no longer affected us we used https://log4j-tester.trendmicro.com, which reported a clean result. Problem solved! 🎉

Lessons learned:

I learned about the Log4J vulnerability via Reddit. Therefore, browsing Reddit has a business purpose; learning about security vulnerabilities. That's totally what I'm doing on there. Any adorable cat gifs on my screen is 100% related to security.
You have more vendors and packages than you realize. When you see a vulnerability, think carefully about what it might affect.
Containerization is good for security not just because it isolates attacks but also because the ease of updating makes attack response fast.

Migrating to Terraform: a retrospective

Caleb Collins-Parks — Fri, 09 Apr 2021 16:17:39 +0000

Summary:

At 15Five we migrated to Terraform in 8 months with a 3-person team. It went smoothly and gave us more confidence in our AWS infrastructure. When it came to setup a new VPC for a customer, the hours we invested returned to us in a speedy and simple setup, a process that would have been a massive pain if we had done it manually.

The Stats:

Lines of code: ~16,000
Number of commits: 903
PR's: 514
The team: Paul (principal engineer), James (devops manager), Caleb (SRE)

The Timeline:

2019.09 - I played around with Terraform for the first time in a company hackathon.
2019.09 - I used Terraform to do my first AWS change.
2019.11 - James joins team & starts work on terraform. Repo is split into a main terragrunt repo and a terraform repo for our various modules. At this point we only have two modules. CI is setup.
2020.01 - we migrated our first environment
2020.04.28 - preview environment migration attempt 1
2020.04.29 - preview migration attempt 2 ✔️
2020.05 - staging migration
2020.06 - solitary customer VPC migration
2020.06 - production migration

Takeaways:

Caleb:

Pain points:

Having two different repos is a bit awkward. I feel that having a monorepo or more CI would have been better, as the lack of CI feedback when changing code in the modules repo slowed things down. We'll probably be moving to a monorepo in the future.

What went well:

Hackathons are good for inspiring change.
Checklists are great. It simply feels good to check things off, for one. It also makes you think of all the items to do ahead of time and makes sure that you don't forget anything. Hospitals have used checklists to massively lower infection rates, although the results depend on how effectively they are used.
Terraform is great for developing a deep understanding of all the services involved in your infrastructure. By describing your infrastructure in code you develop a mental model of all the components involved, and have a written record of the tiny fiddly settings that might normally be hidden away behind some unintuitive UI icons.
Migrating production went pretty smooth, that's worth its weight in gold.

James:

Pain Points:

The monolith structure of Terraform modules results in very large artifacts - it would have preferable to break it down into multiple git repositories, one repo for each module.

What went well:

We have been able to upgrade Terraform without incident.
We were able to develop 30-40% faster because of having a small agile team without overly restrictive bureaucracy.

Paul:

Pain points

We should have pinned the Terraform version in version control to prevent accidental upgrade and to keep everyone on the same version (we ended up doing this later on with tfenv and a .terraform-version file).
None of the solutions for handling multiple environments seemed ideal, but James ended up solving this with Terragrunt once he came on board.

What went well?

It was a strong relief to have Terraform handle the setup of the environment. You just had to click a button, walk away, and the infrastructure would be ready by the time you came back.
Using Terragrunt was a very clean way of configuring infrastructure across multiple environments.
It checked off a lot of boxes for compliance and disaster recovery.

What could we have done better?

In retrospect I could have handed off more product work to other people to have more time to write Terraform and ensure best practices.
We could have had nightly test runs - spinning up and tearing down infrastructure in another region to be confident it would always work. Metrics such as timing and money spent in setup would have been useful. The more metrics the better.

Any other thoughts?

I wish I knew Terraform wrappers like Palumi were an option. However, it probably would have been overkill at the time.
I have found that it saves more time to configure a cloud service with Terraform first and then checking UI. Doing it in the UI first and then translating that to Terraform is slower.

Managing health checks at scale

Caleb Collins-Parks — Fri, 05 Feb 2021 17:39:00 +0000

At 15Five, like many other companies, we use cron to schedule regularly occurring jobs. If a job fails to complete at the expected time or exits with the error code then we get an alert via healthchecks.io.

For a while this worked pretty well.

But soon we faced a challenge. We wanted to expand health checks to all of our testing environments, but that would require creating hundreds and hundreds of new checks. That's a lot of manual work, and why work when you can automate?

I pondered different solutions like writing a script to copy the checks from one project to another, or a script to parse all the ping endpoints into a format that could easily be inserted into our codebase, or managing the health checks via terraform. But none of these solutions were perfect. I wanted something to completely automate the process.

Fortunately, after talking with the Healthchecks maintainer, it turns out they already had a basic script for that. Every time a job completed, it would do a simple call to the Healthcheck API, creating the check if it didn't already exist and returning the check endpoint. Ping the endpoint, and you're done! Easy peasy, no human work involved.

#!/bin/bash

API_KEY=your-api-key-here

# Check's parameters. This example uses system's hostname for check's name.
PAYLOAD='{"name": "'`hostname`'", "timeout": 60, "grace": 60, "unique": ["name"]}'

# Create the check if it does not exist.
# Grab the ping_url from JSON response using the jq utility:
URL=`curl -s https://healthchecks.io/api/v1/checks/  -H "X-Api-Key: $API_KEY" -d "$PAYLOAD"  | jq -r .ping_url`

# Finally, send a ping:
curl -m 10 --retry 5 $URL

I expanded on this to work with alert channels, measure start and end times, and to record failures.

Install jq, replace healthchecks_api_key with your API key and ship the file with your cron jobs. Now you can pass the cron command to the script and it will do all your work for you. Creating a new environment? Just update the API key and you're good to go!

We use Ansible to install jq and templatize the crontab and healthcheck file. Feel free to use Puppet instead, or manual distribution, or docker, or heck, redstone blocks in Minecraft, who am I to judge?

This has been working well in our production environment for months now. As a bonus when we go into Healthchecks.io we can see a detailed history of job start and end times:

It's great to be able to set up a new datacenter and take comfort in a vibrant field of solid green checks, and the only thing I had to do to set it up was to update an API key!

100,000 lines of Typescript: A data-backed retrospective

Caleb Collins-Parks — Tue, 19 Jan 2021 17:16:18 +0000

PROLOGUE:

If you're more interested in the results of the migration than the journey itself, scroll on down to DATA & RESULTS.

At first, types seemed tedious, and not worth the effort. I remember thinking, back in college, why was I specifying that a string (as indicated by the name foo_str) was a string (as indicated by the quotation marks) was a string (as indicated by the type)? Then Typescript came with a revelation. Type inference¹! The computer would do the grunt work for you, leaving you free to wander outside and sing and dance and skip and shout joyfully into the sunlight². I would never use Javascript again.

REGEXES EVERYWHERE

I was using Javascript again.

Or to be more accurate, my coworkers at my new company 15Five were using Javascript. Those poor souls. I nobly volunteered (was assigned a ticket) to save them from their struggle (they were doing okay, really), by migrating the codebase to Typescript (by migrating the codebase to Typescript).

This was my first time doing a migration of this size, and it was a daunting task. There were over 300 files and literally thousands upon thousands of Typescript errors. I knew migrating it by hand, alone, was out of the question, so I checked to see if there were any auto-migration tools. There was one! Yes! It failed to run! Yes! Wait, No!

Unfortunately, that was the only one I found. Other large migrations I read of were done by teams of people with a solid chunk of time dedicated solely to the migration, and the assistance of specialized tools.

Fortunately, we already used React PropTypes, which are somewhat similar to Typescript types. In some cases, very similar. A simple search/replace across the entire project (VSCode made this very easy) sufficed to take care of basic cases like converting PropTypes.string to string. This alone was a massive time saver as I could change ~300 files at once.

But plain text replacement can still be slow, as you have to go through each scenario one by one. With regex you can handle multiple scenarios at once:

PropTypes.(string|number|object|symbol|any).isRequired
Substitute with $1 to replace all the primitive PropTypes at once!

PropTypes.arrayOf$(\w+)$.isRequired
Substitute with $1[] to replace one-line array types.

([^:]+): PropTypes.array,
Substitute with $1?: any[]; to replace bare arrays.

Regexes aren't as powerful as codemods, but they were enough for me. Within a week I had a PR merged that converted the entire react codebase to Typescript.

TSC-SILENT

But I wasn't done. Not by a long shot. Although the PropTypes were in Typescript, and the files were technically Typescript files with a .ts extension, there were still thousands of miscellaneous type errors that polluted a Typescript run. Devs could still have the benefits of Typescript on new code, but on old code it wasn't as powerful. Furthermore, the type errors prevented us from running Typescript in our continuous integration (CI) environment.

Fortunately, I found a tool called tsc-silent, that allowed you to run Typescript while silencing certain types of errors. That way the CI could catch any new errors that appeared, while also giving us a progress report on how many old errors were left. Tsc-silent was highly configurable, letting us silence certain errors in directors of the old codebase while catching all errors in other directories.

Another option I could have done but didn't think of at the time was a snapshot test. By saving Typescript errors to a file and failing CI if the run doesn't match the file, you can catch any new errors introduced and keep track of existing errors. For more details see https://dylanvann.com/incrementally-migrating-to-typescript.

THE LONG GRIND

For a while I tried tackling the remaining Typescript errors by myself. This went slowly, given my limited time and lack of expertise in the codebase. Eventually, I realized I needed subject matter experts, so I created an every-other-week meeting with a few generous volunteers from the frontend team. Thanks to the expertise of the volunteers, we made good headway at first, but as the months went by each type error took longer and longer to solve. We were so close - only 500 errors left! Yet fixing just a couple errors could take an entire hour.

For example, certain errors may involve your code's use of a foreign package, so in order to fix it you need a good knowledge of both your code and the package's API. If the library doesn't have good types then the issue can take a while to untangle. The fix may pretty easy (one PR literally just added <Date>) but knowing how to fix it is a whole other story.

As 2020 neared its end (good riddance) I decided to switch strategies.

TS-MIGRATE

Airbnb open-sourced their automatic JS->TS migration tool, a project aptly called ts-migrate. Although our codebase was already migrated, the feature to automatically add ts-expect-error comments to all existing errors seemed useful. Using that to ignore existing errors would let us Typescript directly in our CI instead of tsc-silent and prevent new errors from creeping in. Running it took a while, but soon tsc produced a clean run! I added it to our CI, and a simple grep allowed us to keep track of the remaining type errors.

- run:
    name: Leftover type errors per file
    command: grep -cR "// @ts-expect-error ts-migrate" your/path/here | grep -v :0
- run:
    name: Number of leftover type errors
    command: grep -R "// @ts-expect-error ts-migrate" your/path/here | wc -l

There's now ~200 errors left, but a lot of them will be gone once we refactor out our usage of Redux. I would like to say we got down to 0 errors, as that would make for a neat and tidy ending to this story, but the real world is not so perfect. When Redux is gone we can go through the remaining errors and decide what is worth fixing.

DATA & RESULTS:

Team size:

11 Frontend engineers in total (not directly involved with TS)
3-4 engineers directly involved in migration
1 Project lead (not included in above)

For context, most of our employees are senior or mid-level coders. We have no interns.

Codebase Size:

Jun 19th, 2019: 291 files, 25,866 lines
Dec 7th, 2020: 1560 files, 126,816 lines

PR's: 47

Person-hours:
very rough estimates follow

Assuming each PR ~= 1 hour, that's 47 hours
Plus 10 hours for meetings
Plus 5 hours for PR reviews
Total: 62 hours

If we had ts-migrate at the start, that might have saved 9 hours or so.

Number of TS Errors Fixed: 3,168

Time:

Jun 19th, 2019: Started work
Jun 25th, 2019 (6 days): Migrated to TS
Sep 30th, 2019 (3 months): TS enforced in CI
Dec 7th, 2020 (1.5 years): Vast majority of errors fixed, TS used directly in CI

Note that most of the benefit of the migration was realized within the first few months. This follows the Pareto Principle - 80% of the outcome results from 20% of the work.

Bugs: 0 bugs, thanks to great reviews by Razh!

Survey data:

Although the sample size was too small to be statistically valid, the survey results were very good. Most developers were happy with Typescript. It caught bugs, improved intellisense, and made refactoring easier. The survey did reveal mixed results on making development faster, but that is no surprise. The work it takes to add types somewhat negates the improved development speed from refactoring and intellisense. The benefit you are left with is catching bugs.

SUMMARY:

Converting to Typescript with ts-migrate is a quick and easy win to improve your frontend code. Fixing all Typescript errors on a large codebase, on the other hand, is not easy nor quick. Going through them all can be a slog, so you should maintain inertia by holding type-fixing meetings or hackathons. Certain type errors may be rather difficult to resolve, so you should decide ahead of time the maximum amount of effort you are willing to spend. You can realize the majority of benefit from Typescript without fixing all the errors, so you don't need to worry about fixing everything in one long all-nighter. You can take your time - so why not start today?

I don't mean to suggest Typescript was the first to come about with type inference, just that it was the first to bring it to Javascript at a large scale. ↩
Or to stay inside and write articles about how free you are ↩

Announcing Rocket Releaser

Caleb Collins-Parks — Tue, 15 Dec 2020 17:15:38 +0000

15Five has open-sourced a python package for generating release notes and sending them to Slack! Take a look below:

15five / rocket_releaser

Python library for release notes / labeling upon deploys

Rocket Releaser

Library for creating release notes based on PR descriptions for a set of git changes.

Installation:

pip install rocket-releaser

Usage:

python -m rocket_releaser github_token start_sha end_sha github_org github_repo -r repo_location --vpc_name staging --env_name staging --search_branch staging --slack_webhook_key slack_key

This would label PR's inbetween first and second sha with "staging" and send the release notes to slack If you are using a personal github account the "github_org" would be your github username.

The github token must have "repo" scope access and the token's user must have write access to the repo.

You can also pass in Jira paramaters to label Jira tickets.

--jira_token jiraToken --jira_username bob@company.com --jira_url https://company.atlassian.net

PR format:

To label PR's and tickets your PR's should be formatted like so:

<!-- Detailed PR description for reviewers goes here... --&gt
RELEASES
Change default avatar image #public Closes [ENG-1234]
<!--
- Newline after RELEASES (need this for proper formatting

…

View on GitHub

This allows our QA and other teams to easily see what changes have been deployed and what needs testing. The package also labels PR's and tickets, so everyone is kept in the loop as to what stage of the deployment process their code is in.

The repository is based off of the open-source circleci boilerplate announced earlier.

CI/CD python packaging through CircleCI

Caleb Collins-Parks — Mon, 16 Nov 2020 17:36:10 +0000

15Five has open-sourced a boilerplate template for a python package with continuous integration and delivery through CircleCI. Take a look below!

15five / python-circleci-package-boilerplate

boilerplate for a python package with CI/CD through circleci

Features:

FULL CI/CD through Circleci with the works (black linting, pip caching, tests against multiple python versions, test summary, coverage results, automatic package deploy each push to master)
Package directory structure already laid out
Makefile and other sensible files and configuration already present

If you're using Travis I suggest using the popular python package cookie cutter template.

Using DjangoQL for a better django admin search

Caleb Collins-Parks — Mon, 05 Oct 2020 18:45:00 +0000

A reasonable Django admin search looks like this:

Python:

class MusicianAdmin(admin.ModelAdmin):
    search_fields = ['first_name', 'last_name']

User search input:
john lennon

Resulting SQL:

WHERE (first_name ILIKE '%john%' OR last_name ILIKE '%john%')
AND (first_name ILIKE '%lennon%' OR last_name ILIKE '%lennon%')

Not too bad, right?

But now imagine your search fields look like this:

class ReviewerReporterAdmin(admin.ModelAdmin):
    search_fields = ('reporter__first_name', 'reporter__last_name',
                     'reporter__email', 'reporter__company__name',
                     'reporter__company__subdomain',
                     'reviewer__first_name', 'reviewer__last_name',
                     'reviewer__email',)

The resulting SQL will look like this:

WHERE (reporter__first_name  ILIKE '%john%' OR reporter__last_name ILIKE '%john%' OR reporter__email  ILIKE '%john%' OR reporter__company__name ILIKE '%john%' OR reporter__company__subdomain ILIKE '%john%' OR reviewer__first_name  ILIKE '%john%' OR reviewer__last_name ILIKE '%john%' OR reviewer__email ILIKE '%john%')
AND (reporter__first_name  ILIKE '%lennon%' OR reporter__last_name ILIKE '%lennon%' OR reporter__email  ILIKE '%lennon%' OR reporter__company__name ILIKE '%lennon%' OR reporter__company__subdomain ILIKE '%lennon%' OR reviewer__first_name  ILIKE '%lennon%' OR reviewer__last_name ILIKE '%lennon%' OR reviewer__email ILIKE '%lennon%')

Yikes. But computers are powerful. That's still manageable.

Now try this one on for size:

WHERE (reporter__first_name  ILIKE '%john%' OR reporter__last_name ILIKE '%john%' OR reporter__email  ILIKE '%john%' OR reporter__company__name ILIKE '%john%' OR reporter__company__subdomain ILIKE '%john%' OR reviewer__first_name  ILIKE '%john%' OR reviewer__last_name ILIKE '%john%' OR reviewer__email ILIKE '%john%')
AND (reporter__first_name  ILIKE '%lennon%' OR reporter__last_name ILIKE '%lennon%' OR reporter__email  ILIKE '%lennon%' OR reporter__company__name ILIKE '%lennon%' OR reporter__company__subdomain ILIKE '%lennon%' OR reviewer__first_name  ILIKE '%lennon%' OR reviewer__last_name ILIKE '%lennon%' OR reviewer__email ILIKE '%lennon%')
AND (reporter__first_name  ILIKE '%is%' OR reporter__last_name ILIKE '%is%' OR reporter__email  ILIKE '%is%' OR reporter__company__name ILIKE '%is%' OR reporter__company__subdomain ILIKE '%is%' OR reviewer__first_name  ILIKE '%is%' OR reviewer__last_name ILIKE '%is%' OR reviewer__email ILIKE '%is%')
AND (reporter__first_name  ILIKE '%a%' OR reporter__last_name ILIKE '%a%' OR reporter__email  ILIKE '%a%' OR reporter__company__name ILIKE '%a%' OR reporter__company__subdomain ILIKE '%a%' OR reviewer__first_name  ILIKE '%a%' OR reviewer__last_name ILIKE '%a%' OR reviewer__email ILIKE '%a%')
AND (reporter__first_name  ILIKE '%great%' OR reporter__last_name ILIKE '%great%' OR reporter__email  ILIKE '%great%' OR reporter__company__name ILIKE '%great%' OR reporter__company__subdomain ILIKE '%great%' OR reviewer__first_name  ILIKE '%great%' OR reviewer__last_name ILIKE '%great%' OR reviewer__email ILIKE '%great%')
AND (reporter__first_name  ILIKE '%singer%' OR reporter__last_name ILIKE '%singer%' OR reporter__email  ILIKE '%singer%' OR reporter__company__name ILIKE '%singer%' OR reporter__company__subdomain ILIKE '%singer%' OR reviewer__first_name  ILIKE '%singer%' OR reviewer__last_name ILIKE '%singer%' OR reviewer__email ILIKE '%singer%')
AND (reporter__first_name  ILIKE '%and%' OR reporter__last_name ILIKE '%and%' OR reporter__email  ILIKE '%and%' OR reporter__company__name ILIKE '%and%' OR reporter__company__subdomain ILIKE '%and%' OR reviewer__first_name  ILIKE '%and%' OR reviewer__last_name ILIKE '%and%' OR reviewer__email ILIKE '%and%')
AND (reporter__first_name  ILIKE '%songwriter%' OR reporter__last_name ILIKE '%songwriter%' OR reporter__email  ILIKE '%songwriter%' OR reporter__company__name ILIKE '%songwriter%' OR reporter__company__subdomain ILIKE '%songwriter%' OR reviewer__first_name  ILIKE '%songwriter%' OR reviewer__last_name ILIKE '%songwriter%' OR reviewer__email ILIKE '%songwriter%')

That's what happens when the user searches for "john lennon is a great singer and songwriter". Each word is searched against every field, leading to the giant SQL block above! Here's a real-world example of what this looks like:

(table and field names obfuscated):

SELECT
"banana_table"."id", "banana_table"."create_ts", "banana_table"."type", "banana_table"."sasquatch_id", "banana_table"."yolo_id", "banana_table"."to_scale", "banana_table"."extra", "ff_orange"."id", "ff_orange"."spiders", "ff_orange"."last_epic_win", "ff_orange"."is_superorange", "ff_orange"."passport", "ff_orange"."number_swords", "ff_orange"."scim_orangename", "ff_orange"."orangename", "ff_orange"."mango", "ff_orange"."canonical_mango", "ff_orange"."yolo_id", "ff_orange"."global_id", "ff_orange"."first_kiss", "ff_orange"."last_kiss", "ff_orange"."knightlyTitle", "ff_orange"."IVotedDidYou", "ff_orange"."IVotedDidYou_length", "ff_orange"."aspirational_knightlyTitle", "ff_orange"."aspirational_knightlyTitle_length", "ff_orange"."aspirational_IVotedDidYou", "ff_orange"."aspirational_IVotedDidYou_length", "ff_orange"."vulnerabilities", "ff_orange"."vulnerabilities_length", "ff_orange"."banana_mansion", "ff_orange"."banana_mega_mansion", "ff_orange"."IEnjoyThinkingOfRandomNames", "ff_orange"."ButImNotSureAnyoneWillActuallyReadThem", "ff_orange"."MakingUpRandomNamesIsItsOwnReward", "ff_orange"."is_yolo_admin", "ff_orange"."is_pineapple_admin", "ff_orange"."can_view_all_oranges", "ff_orange"."require_spiders_change", "ff_orange"."is_vip", "ff_orange"."is_vip_bossperson", "ff_orange"."is_a_bro", "ff_orange"."is_not_a_bro", "ff_orange"."saml_orange_id", "ff_orange"."first_mango_eaten", "ff_orange"."last_spiders_change_ts", "ff_orange"."create_ts", "ff_orange"."update_ts", "ff_orange"."num_dirty_mangoes", "ff_orange"."howdouspellmangos", "ff_orange"."last_seen_ts", "ff_orange"."ithinkitswithae", "ff_orange"."ahhhhh", "ff_orange"."imrunningoutofrandomnames", "ff_orange"."howmanymorefields", "ff_orange"."arethere", "ff_orange"."uggghhh", "ff_orange"."is_ides_of_march", "ff_orange"."biweekly_is_ides_of_march", "ff_orange"."biweekly_is_ides_of_march_which_week", "ff_orange"."monthly_is_ides_of_march", "ff_orange"."monthly_is_ides_of_march_which_in_month", "ff_orange"."prefs", "ff_orange"."toilet_paper", "ff_orange"."skip_yolo_wide_questions", "ff_orange"."blorp_type", "ff_orange"."bloop_type", "ff_orange"."number_angry_zebras", "ff_orange"."can_manage_oranges", "ff_yolo"."id", "ff_yolo"."name", "ff_yolo"."sick_hair_yo", "ff_yolo"."IEnjoyThinkingOfRandomNames", "ff_yolo"."prefs", "ff_yolo"."is_dev", "ff_yolo"."apocalypse_start_date", "ff_yolo"."apocalypse_length_days", "ff_yolo"."mango_domain", "ff_yolo"."hypeperson_id", "ff_yolo"."toilet_paper", "ff_yolo"."create_ts", "ff_yolo"."update_ts", "ff_yolo"."arethere", "ff_yolo"."num_dirty_mangoes", "ff_yolo"."converted_to_paying_ts", "ff_yolo"."stopped_paying_ts", "ff_yolo"."country", "ff_yolo"."state", "ff_yolo"."is_ides_of_march", "ff_yolo"."biweekly_is_ides_of_march", "ff_yolo"."biweekly_is_ides_of_march_which_week", "ff_yolo"."monthly_is_ides_of_march", "ff_yolo"."monthly_is_ides_of_march_which_in_month", "ff_yolo"."uggghhh", "ff_yolo"."is_vip_bossperson", "ff_yolo"."is_a_bro", "ff_yolo"."smart_banana_access", "ff_yolo"."happiness_access", "ff_yolo"."smart_potato_access", "ff_yolo"."attachments_access", "ff_yolo"."smart_potato_access", "ff_yolo"."is_sad_banana", "ff_yolo"."is_angry_banana", "ff_yolo"."is_secret_banana", "ff_yolo"."is_nice", "ff_yolo"."is_delicious", "ff_yolo"."carrot_hypeman", "ff_yolo"."potato_hypeman", "ff_yolo"."number_of_people_with_eyebrows", "ff_yolo"."banana_owner_id"
FROM "banana_table"
LEFT OUTER JOIN "ff_orange" ON ("banana_table"."sasquatch_id" = "ff_orange"."id")
LEFT OUTER JOIN "ff_yolo" ON ("banana_table"."yolo_id" = "ff_yolo"."id")
WHERE (
    (UPPER("banana_table"."type"::text) LIKE UPPER('%this%') OR UPPER("ff_orange"."first_kiss"::text) LIKE UPPER('%this%') OR UPPER("ff_orange"."last_kiss"::text) LIKE UPPER('%this%') OR UPPER("ff_orange"."mango"::text) LIKE UPPER('%this%') OR UPPER("ff_yolo"."name"::text) LIKE UPPER('%this%') OR UPPER("ff_yolo"."sick_hair_yo"::text) LIKE UPPER('%this%'))
    AND (UPPER("banana_table"."type"::text) LIKE UPPER('%is%') OR UPPER("ff_orange"."first_kiss"::text) LIKE UPPER('%is%') OR UPPER("ff_orange"."last_kiss"::text) LIKE UPPER('%is%') OR UPPER("ff_orange"."mango"::text) LIKE UPPER('%is%') OR UPPER("ff_yolo"."name"::text) LIKE UPPER('%is%') OR UPPER("ff_yolo"."sick_hair_yo"::text) LIKE UPPER('%is%'))
    AND (UPPER("banana_table"."type"::text) LIKE UPPER('%a%') OR UPPER("ff_orange"."first_kiss"::text) LIKE UPPER('%a%') OR UPPER("ff_orange"."last_kiss"::text) LIKE UPPER('%a%') OR UPPER("ff_orange"."mango"::text) LIKE UPPER('%a%') OR UPPER("ff_yolo"."name"::text) LIKE UPPER('%a%') OR UPPER("ff_yolo"."sick_hair_yo"::text) LIKE UPPER('%a%'))
    AND (UPPER("banana_table"."type"::text) LIKE UPPER('%test%') OR UPPER("ff_orange"."first_kiss"::text) LIKE UPPER('%test%') OR UPPER("ff_orange"."last_kiss"::text) LIKE UPPER('%test%') OR UPPER("ff_orange"."mango"::text) LIKE UPPER('%test%') OR UPPER("ff_yolo"."name"::text) LIKE UPPER('%test%') OR UPPER("ff_yolo"."sick_hair_yo"::text) LIKE UPPER('%test%'))
    AND (UPPER("banana_table"."type"::text) LIKE UPPER('%query%') OR UPPER("ff_orange"."first_kiss"::text) LIKE UPPER('%query%') OR UPPER("ff_orange"."last_kiss"::text) LIKE UPPER('%query%') OR UPPER("ff_orange"."mango"::text) LIKE UPPER('%query%') OR UPPER("ff_yolo"."name"::text) LIKE UPPER('%query%') OR UPPER("ff_yolo"."sick_hair_yo"::text) LIKE UPPER('%query%'))
)
ORDER BY "banana_table"."id" DESC

If your eyes glazed over that example, don't worry. The exact syntax doesn't matter - the point is that such a large query is not a theoretical example, it can actually happen!

Still, when running against a local database the query only took roughly 4 milliseconds. Fast, but the table in my local database only has 70 rows. Slow code will remain silent at that scale, lurking in the background and waiting to pounce. And it likes to rear its head at the worst possible moment - production.

In production we have over 3 MILLION rows. Divide 3,847,951 by 70 to get the ratio of how much larger production is. Multiply the time it took with 70 rows (4ms) by the ratio and we can predict that the same search in production would take 219,882 milliseconds, or 219 seconds*! Ain't nobody got time for that. Seriously, our pages time out after 30 seconds, so there's literally not enough time for that.

You could simply use less search fields. But what if the user wants to search against a field you removed? You're reducing functionality to improve performance, which may or may not be an acceptable tradeoff.
You could buy a better database. But you might not have the funds for that yet.
You could add indexes to your tables. But this requires careful consideration for each table.
You could add a dedicated mirror database for django admin. But you might not have the funds for that yet. (This was one of the improvements we made)
You could switch to Elasticsearch. But that will take time to implement, add complexity, and again, take funds to host.

If only the user could easily choose what fields they want to search against, that would help relieve the problem. Luckily, there's a library that does exactly that; enter DjangoQL!

By letting users choose what fields to search against you can massively speed up searches. Remember that SQL query I showed earlier?

WHERE (first_name ILIKE '%john%' OR last_name ILIKE '%john%')
AND (first_name ILIKE '%lennon%' OR last_name ILIKE '%lennon%')

If the user selects a specific field the query becomes simpler, and thus faster:

WHERE (first_name ILIKE '%john%')
AND (first_name ILIKE '%lennon%')

If you have a lot of search fields and data, this can result in a large performance improvement:

Page      |  # Items   |  Performance
--------------------------------------------------------------------------
XXXX      |  ~1,000    |  no performance difference
YYYY      |  ~100,000  |  up to ~5x increase in search speed
ZZZZ      | ~1,000,000 |  searching can take so long it times out unless DjangoQL is used

This is also dependent upon network speed, hardware speed, database indexes, and a whole host of other factors. But generally speaking if you have over 100k rows and a bunch of search fields you'll probably see a performance increase.

The neat thing is that the performance improvement is really just a side-benefit of the library. The main benefit is giving your users a query language that is much more powerful and flexible than just being able to do text matches.

The final benefit is that DjangoQL is a very straightforward addition to your code. You just make a simple class that inherits from DjangoQLSearchMixin and admin.ModelAdmin and use that instead of admin.ModelAdmin

Repeat changes in the bottom file for the rest of your files, and that's basically it!

This isn't going to solve your performance problems, but it should help a bit assuming you educate your users on how to use it. It's a easy yet powerful band-aid you can apply while you work on more long-term fixes. Even if you don't care about speeding up your code it's still useful, so it's a win-win!

How to (temporarily!) ignore errors during a TypeScript migration

Caleb Collins-Parks — Fri, 14 Aug 2020 15:56:05 +0000

Update: You might want to skip this approach and use ts-migrate instead. It can automatically migrate your JS to TS and add @ts-expect-error comments to type errors you need to resolve by hand.

Last year at 15Five we started migrating our frontend react codebase to TypeScript. I went wild with regexes and Vscode's search/replace all feature, adding types and changing syntax in mass. But regex can only get you so far. There was still thousands of type errors, with each type needing to be reviewed manually. Some were easy to fix, others were much harder. This was going to take a lot of time. Time that I didn't have - my responsibility was DevOps after all, and I was migrating code to TypeScript in my free time. Frontend engineers also worked on the migration but they needed time to ship features.

I could've abandoned the migration and came back to it later. But without validation in our CI, untyped code would sneak in and leave a even bigger problem in the future. Luckily I came across tsc-silent!

evolution-gaming / tsc-silent

`tsc` with `--suppress` flag

`tsc-silent`

The purpose of the wrapper is to execute TypeScript compiler but suppress some error messages coming from certain files/folders. For example, this can be used to enable noImplicitAny in some parts of the project while keeping it disabled in others.

Installing

npm install -g tsc-silent

Usage

tsc-silent --project <path> [--suppress config | --suppressConfig path] [--compiler path] [--watch]

Synopsis

--project, -p    Path to tsconfig.json

--compiler       Path to typescript.js.
                 By default, uses `./node_modules/typescript/lib/typescript.js`.

--suppress       Suppressed erros.
                 E.g. `--suppress 7017@src/js/ 2322,2339,2344@/src/legacy/`.

--suppressConfig Path to supressed errors config.
                 See documentation for examples.

--watch, -w      Run in watch mode.

--stats          Print number of suppressed errors per path and error code.

Suppress config

You have to pass either --suppress or --suppressConfig.

`--suppress`

Let's ignore error 7017 in src/js/ directory and errors 2322, 2339, 2344 in /src/legacy/:

tsc-silent -p tsconfig.json --suppress 7017@/src/js/ 2322,2339,2344@/src/legacy/

or, ignore all errors in /src/legacy/ folder

tsc-silent -p tsconfig.json --suppress

…

View on GitHub

Tsc-silent lets you ignore certain TypeScript errors. By ignoring the errors that came from code not yet migrated, I could add TypeScript to our CI without error! For the errors that were fixed, tsc-silent would raise an error if they came back again.

This was quite easy to configure:



yarn tsc-silent --project tsconfig.json --suppress 2339,7006,2345,2739,2532,2307,7031,2741,2554,7053,7005,2740,2538,2531,2551,2571,2533,2305,2322,7015,2305,2722,7034,2724,2349,7016,1208,2749,2604,2363,2362,2605,2555,2464,2525,2559,2362@ff/static/modules 2305@node_modules --stats

At this point you might be thinking "Darn, that's a lot of errors you're ignoring! Isn't that bad practice? Is it even worth it if so many errors are ignored"?

A valid concern, but even catching just a few errors is better than catching no errors at all. And this is the dumb approach - our resident code ninja (or "Frontend-related human being" as he humbly calls himself) Razh came up with a much smarter solution.

By passing in a configuration file to tsc-silent, we can choose what errors are ignored on which paths!



yarn tsc-silent --project tsconfig.json --suppressConfig tsc-silent.config.js --stats

tsc-silent.config.js



module.exports = {
  suppress: [
    {
      pathRegExp: '/ff/static/modules/components',
      codes: [2322, 2339, 2345, 2362, 2363, 2532, 2555, 2739, 2769, 7016],
    },
    {
      pathRegExp: '/ff/static/modules/realtime_feedback',
      codes: [2322, 2345, 2769],
    },
    # other suppressions not shown
  ]
}

Now for new folders we have full TypeScript error checking! For old folders we can be more selective about what errors we ignore too. As a bonus the tsc-silent output gives you information on exactly how many errors are left that you need to fix if you use the --stats option. In the example below you can see that there are a couple errors with only 1 instance left - those would be easy targets to fix and start validating against.



  {
    "codes": {
      "2322": 4,
      "2339": 2,
      "2345": 3,
      "2362": 2,
      "2363": 2,
      "2555": 1,
      "2739": 2,
      "2769": 11,
      "7016": 1
    },
    "pathRegExp": "/\\/ff\\/static\\/modules\\/components/",
    "total": 28
  },

In conclusion, if you're migrating a large project to typescript tsc-silent can be quite useful. It lets you validate errors in CI even when your migration is not finished yet. Just don't use it as a excuse to ignore errors forever - there's a reason TypeScript raises an error, and even if it's not a bug it's still good to fix to increase the quality of your code. Someday we will finish up our migration and our need for tsc-silent will be gone. Until then, it's served us well.

How to never forget you're in production

Caleb Collins-Parks — Tue, 11 Aug 2020 02:57:14 +0000

The Scenario

You now have production access!
You shell into the server and .... huh

Welcome to Ubuntu 50.01.1

  System information as of Sun Aug  9 20:29:10 UTC 2050

  System load:  0.2                Processes:           149
  Usage of /:   42.4% of 16 bits   Users logged in:     0
  Memory usage: 67%                IP address for ens5: 8.8.8.8
  Swap usage:   0%

 * Are you ready for Kubernetes 5.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.

 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:
     https://ubuntu.com/livepatch

0 packages can be updated.
0 updates are security updates.


Last login: Sun Aug  9 20:22:21 2050 from 192.16.184.0
bob@dns0:~$

It looks exactly the same as any server†. Pretty boring, really. Months go by, and shelling into production becomes old hat to you. One day you're tasked with deleting everything in a test server. A quick rm -rf *, and bam, it's done.

Oh wait.

You were on the production server.

"Maybe if I control-z I can undo it"

This was just a hypothetical example, but it could happen! It's important to visually make your production servers distinct as to reduce the chance of error. Doing so is very easy, and can potentially stop a total catastrophe.

The Tutorial

Put the below line in your ~/.profile file:
export PS1="\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h.{{ domain }}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ "

Replace {{ domain }} with the name of your environment.

Now you can see the environment in your prompt:

caleb@web0.cloud20:~$

For production environments at 15Five we use a distinct color and put the environment name in uppercase:

export PS1="\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h.\[\033[01;31m\]{{ domain }}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ "

Feel free to customize this to your liking - you can even put emojis in your prompt!

You can automate this for all your servers by using Ansible:


# key_names is a list of users

# https://serverfault.com/questions/322997/how-do-you-make-it-obvious-you-are-on-a-production-system
# http://abload.de/img/bash-color-chartmxjbp.png

- name: Set terminal prompt to have VPC name
  lineinfile:
    dest: /home/{{ item }}/.profile
    regexp: '^export PS1='
    line: 'export PS1="\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h.{{ domain }}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ "'
  with_items: "{{ key_names }}"
  when: env_name != 'production'

- name: Set terminal prompt to have red uppercase VPC name in production
  lineinfile:
    dest: /home/{{ item }}/.profile
    regexp: '^export PS1='
    line: 'export PS1="\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h.\[\033[01;31m\]{{ domain|upper }}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ "'  # noqa: 204
  with_items: "{{ key_names }}"
  when: env_name == 'production'

But what if I accidentally run Ansible against the wrong environment?

You can catch that too by using a Ansible callback plugin! Assuming you have a environment variable in your terminal for the environment you are running against, you can verify that the Ansible inventory matches the environment variable.

You probably have to adapt the file for how you handle secrets and deployments via Ansible.

† Assuming your other servers are google DNS servers from the future that work with just 16 bits of storage. You know, the usual.

Fun fact: You can actually recover data from a rm -rf!