DEV Community: scarlett

Debugging Python code in the terminal (with examples)

scarlett — Sun, 29 Dec 2019 17:46:11 +0000

For one of my projects, I wrote a new endpoint in a Flask application. My project had many pieces that built on each other, so I wanted to test in various ways and expand my backend debugging tool belt. Here are 3 tools I learned to use and now regularly use for debugging Python code. 🐍

1. curl

curl is a command line tool for making HTTP requests. I like it because it’s a quick way to test endpoints are doing what I expect, but I found writing a curl confusing at first.

Breaking down the anatomy of a curl

Example GET request to retrieve a specific student’s info

-v
is optional and stands for verbose. With it, you’ll see all the details of the request and response that are normally hidden, which can be useful for debugging.

-H
stands for header, and it’s how to specify extra headers to include in the request, such as an authentication token. In my example, the endpoint can only be accessed by school employees, so I’ve added a header to indicate that I am making an internal origin request. Otherwise, the request response would say I’m unauthorized. (It's not necessary to know what Envoy is for the example, but here’s more info if you’re curious: 1 & 2)

You don’t always have to specify extra headers (depends on the API or type of request), and you can include many headers in your request.

Example POST request to make a new student

-X POST
will perform a POST request. -X allows you to change the HTTP method used because the default is GET.

-d
stands for data. With this option, you can specify what data you’re sending to the HTTP server, in the same way that a browser does when you fill out a form and click the submit button. There are various ways to pass data in a curl POST request; in my example, I’m sending JSON.

Content-Type
Since I’m sending data, I need to let the server know the format of the data.

2. Python Interpreter

The Python interpreter is a tool that comes with your Python installation. It’s an interactive REPL in which you can type and execute Python code. I use it often to import and interact with the files I’ve written.

Getting Acquainted

service_venv ipython
is how I launch the interpreter. But it’s more commonly invoked by typing python or python3 (depending on your installation). Note: IPython provides extra features than the standard Python interpreter, but the examples I’m showing can be performed in the standard interpreter as well.

from app.models.student import *
imports the specific file to use in the interpreter. I am specifying the directory path of the student.py file: there’s a folder app at the top level directory which contains the folder models which contains the file student.py. import * states that I want everything (*) in that file. Note: when importing a file, do not add the file ending.

The import statement is necessary; without it, in my example, StudentModel and .save() would be undefined and the Python interpreter would throw an error message.

Everything’s setup, and now I can test if my code is working as expected!

In this example, I'm testing a specific method get_students in the Python interpreter.

What are some investigations I could do? Perhaps I can test what happens when a user passes in a string value. Or I can see what happens if a user passes in a valid integer value but a student with that id doesn't exist. Am I returning a helpful error message? In the method get_students, I haven't done error handling...yet! In my experience, testing in the Python interpreter has been great for "ooooh yeah need to add that" realizations.

3. Pytest

Pytest is a Python testing framework that can be setup quickly to run tests and report the results. It’s also another command line tool!

Here’s an example of an integration test that tests that a GET call returns the expected data.

I learned how to write tests by studying examples, and I was confused by what values were in a test, like json_body. So, I'd take tests a part and add print statements, like in the example below. But when running the test, I wouldn't see the output of the print statements. 🧐

`-s`

python -m pytest
is how to invoke testing through the Python interpreter. service_venv is specific to my development environment.

tests/integration/test_student_resource.py
is the directory path of the file I want to test.

-s
is the star of this example. Adding this option allows all print statements in tests to get printed in the terminal when the tests is run. Now I'd be able to see the value of print(json_body).

Pytest is a powerful tool with powerful features, but this small -s flag has helped me to debug and learn how to write tests.

And that's it! I'm excited to see what other debugging tools I pick up as I continue to work on backend projects.

Deploying to Production 101: deploying safely and putting out fires

scarlett — Wed, 30 Oct 2019 16:27:21 +0000

What can make deploying to production scary? Perhaps it’s your first time going through the process. Or you’re deploying to a service with high user traffic. Or your Pull Request introduces potentially risky changes. Check, check, check - these scenarios applied to my first major deploy. I was also hit by chaos monkey during it (more info on chaos engineering).

Things went sideways during my first major deploy to production, but from the experience, I learned how to conduct better deploys and gained a stronger understanding of what and how to monitor.

What is "deploying"?

Your API or frontend service on localhost:8080/3000 can’t be accessed by others if they go to that URL from their computer. It lives on your computer and can be accessed only by you.

Generally speaking, deploying is the process of taking your local development environment and making it publicly available on a remote host, allowing other users with the URL to access your application. The deployment process is continuous; as you make updates to your application locally, you deploy the new code to make it publicly available.

How to deploy?

Deployment processes vary.

I deployed my Flatiron School Ruby on Rails, React final project to Heroku with a few terminal commands.

At work, my deploys are a multi-step process in which I release new code in stages and to different environments, as opposed to releasing it fully to production right away. The environments/steps are:

Staging: A sandbox environment that is nearly identical to production. In this step, the new code is tested and monitored to ensure everything is working in harmony under a production-like environment.
Canary: A single production node. In this step, the new code is released to a small percentage of real users and monitored for any issues that might not have been caught earlier.
More subsets of production: In this step, the new code is released to more production nodes but not yet all of production. And (as you've guessed by now) it is monitored, monitored, monitored before released to 100% of production.
Production 🎉

The multi-step process is great because it’s a series of gates to catch any bugs before the new code is put in front of users. The deploy doesn’t run on its own, however. At each step, engineers approve if the new code gets to pass onto the next, so the success of a deployment process like this depends on understanding how to monitor and following best practices.

Best Practices

From my first major deploy that went awry, I learned the key things to practice in future deploys. Here’s my advice, which hopefully can be applied to your process.

(1) Let the new code “bake” longer in each deployment phase

At work, teams define how long you must wait on staging, canary, etc. before moving on to the next step. We call this wait period “baking”, in which you see if any errors manifest from and/or alarms are set off by the new changes you’ve introduced to the application. Once you’ve waited the bake period and verified that all is well, you can proceed to the next deployment phase. I like to wait an additional 10-15 minutes in each step for extra monitoring and QAing to make sure I feel confident that the new changes are safe to roll out to production.

(2) Monitor, monitor, monitor, and then monitor some more

A successful deploy is hinged on thorough monitoring, and it starts with understanding how the code you're going to deploy interacts with the rest of the application and what kinds of errors/issues you want to be on the lookout for. Also, it’s important to get comfortable with your team’s monitoring tools and understand what purpose they serve. For example, the tools I use during deploys fall into two investigation modes: is something broken? how is it broken? For the former, I look to Grafana for a big picture overview on whether the system is healthy and changes to key stats (ie. number of 5xx responses). For investigating how something is broken, I use Kibana to look at individual logs and do queries (ie. filtering logs by whether they are classified as a warning, error, or critical) to trace where in the application the issue is happening.

(3) Actively communicate with teammates when there is an issue
Seeing your deploy go on 🔥 can be scary, but everyone has gone through/will go through a problematic deploy. Alert your team if something goes wrong and include any helpful information in the message, like links to graphs or logs that show what/where the issue is. Get confirmation on what to do next if you’re unsure - pause your deploy? rollback? An application is owned by a team, and everyone shares the responsibility to keep it healthy and running.

Anything can happen during a deploy—related or unrelated to your code changes, which is why I find the process intimidating. But learning how to make the deployment process in itself a safeguard has made me feel more equipped to conduct deploys successfully.

Linting an Existing Application: 3 Key Things I Learned

scarlett — Sun, 08 Sep 2019 21:43:31 +0000

I’m here to write on something I’m really proud of doing at work: linting an un-linted production codebase! For one of my onboarding tickets, I set up TSLint in the build of a React project - meaning code with linting errors can’t be merged to master. Linting would be 100% necessary! Yay! What this also meant for me was fixing a ton of existing linting errors.

I had never linted my personal projects, so this process was new to me. I also wasn't too familiar with the codebase and it used TypeScript, which I don’t know. #challengeaccepted. Throughout the linting process, I thought a lot about linting best practices and researched many of the TSLint error messages to understand what was wrong with the code. Here are 3 key things I learned while working on this project:

Enable TSLint to Auto Fix Errors

Some of the linting errors were easy fixes such as missing semicolons, trailing whitespaces at the end of a line, consecutive blank lines, etc. Though easy to fix, it would have been time consuming to fix them manually. There’s always a better way, and I learned that you can enable your IDE to automatically fix eligible linting errors.

Here’s how I set it up in VS Code:

Download the TSLint extension
In VS Code's settings.json file, add:

”editor.codeActionsOnSave”: { “source.fixAll.tslint”: true }

(`Prefences` -> `Settings` -> search for "settings.json" -> select `Edit in settings.json`)

To get the automatic TSLint fixes, I went directly to the file with linting errors and just saved it; I didn’t need to make any changes to the file. I preferred this method over running the terminal command tslint —fix (which would automatically fix all eligible linting errors across the whole application) since I wanted to get more familiar with the codebase and understand the code in those files.

Check out TSLint core rules to see what linting errors are eligible to be fixed automatically; search for the “Has Fixer” flag.

Silence Some Errors

For some of the errors, I wondered about their severity and what linting errors are fine to ignore? I searched for articles on this topic and asked around and ultimately learned this is subjective. I started looking at other React projects that used Typescript at the company and compared tslint.json files (where linting rules can be silenced) to see if there was a standard practice. Here's an example of how to silence TSLint rules in tslint.json:

 "rules": {
      "object-literal-key-quotes": false,
      "no-any": false
}

Silencing a rule in the tslint.json file ignores the rule overall but you can also silence a single instance of a linting error. The latter is a great alternative if generally you think the TSLint rule is important to address but want to make an exception for a certain line of code. Perhaps there’s a reason to keep the line of code the same or perhaps TSLint incorrectly flagged there was an error (this happened on a few occasions). To ignore an instance of a linting error, place this above the flagged line of code:
// tslint:disable-next-line: <name-of-linting-rule>

// tslint:disable-next-line: react-unused-props-and-state

Carefully Review Lint Fixes

I used Version Control in VS Code to quickly and easily review the changes made, especially if they were from automatic fixes. I was concerned about over-linting, breaking a feature, or introducing a new bug, so as I fixed linting errors I tested that the application was still working as expected. Through linting the application slowly, I also noticed a few instances where TSLint mistakenly flagged an error. For example, a few lines were flagged with react-unused-props-and-state but the props were being used in the file.

For me, addressing linting errors in small batches and in a controlled manner was the best way to make sure I was linting properly and to review and test changes.

I learned a lot through this experience and am happy that I contributed in an important way to ensure consistent coding style and improve the development process for my teammates.

I’ll end this post with a linting playlist. Fixing linting errors can take a while and listening to music helps keep your mood 🌞🌞🌞.