DEV Community: jess unrein

Launching a django webserver in Docker

jess unrein — Tue, 05 Dec 2023 22:54:14 +0000

Every single time I have to set up Docker for a new web server I forget a couple key things on how to set it up. This little post is mostly a note to my future self that this is how you do it and why.

For a detailed breakdown of common keywords and concepts of putting together a Dockerfile, check out my previous post on Dockerizing a simple python process.

I found myself needing to run a django server on Docker today because it turns out that you can't set memory limits using the python resource library on processes on macOS since about version 10.14. Solution? Run it on a Python Docker Image that runs on Linux. Easy enough.



FROM python:3.11
RUN pip install poetry
COPY . .
RUN poetry install
CMD ["python", "./myapp/manage.py", "runserver"]

Quick and dirty. I could probably be more careful with what I'm copying, where, and such. But I'm in a rush and this will do the job.



docker build . -t myapp

Builds just fine, I see poetry installing the packages that it should. Beautiful!
However, when I try to run the image...



docker run -it -p 8000:8000 myapp
Traceback (most recent call last):
  File "//./myapp/manage.py", line 11, in main
    from django.core.management import execute_from_command_line
ModuleNotFoundError: No module named 'django'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "//./myapp/manage.py", line 22, in <module>
    main()
  File "//./myapp/manage.py", line 13, in main
    raise ImportError(
ImportError: Couldn't import Django. Are you sure it's installed and available on your PYTHONPATH environment variable? Did you forget to activate a virtual environment?

That's annoying. I just saw poetry install all the libraries. And yes, I went and checked, and django is present in my poetry.lock and pyproject.toml files. What gives? Seems like poetry is creating a virtualenv somewhere that needs to be activated, but that's really not necessary since Docker is already isolating the project and having a virtualenv is redundant. So let's try add a poetry config command to the Dockerfile.



FROM python:3.11
RUN pip install poetry
COPY . .
RUN poetry config virtualenvs.create false --local
RUN poetry install
CMD ["python", "./cohortcreation/manage.py", "runserver"]

Now, the --local tag isn't strictly necessary here, because, again, Docker is isolating this project. But just in case I copy paste this somewhere where it might mess with poetry configurations, let's make sure this config value stays local to whatever project I'm working with.

Next, rebuild the docker image. Loads just fine. Let's try running it.



docker run -it -p 8000:8000 myapp
2023-12-05 22:33:32,585 INFO autoreload django.utils.autoreload Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
December 05, 2023 - 22:33:32
Django version 4.2.4, using settings 'myapp.project.settings'
Starting ASGI/Daphne version 4.0.0 development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
2023-12-05 22:33:32,806 INFO server daphne.server HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2023-12-05 22:33:32,806 INFO server daphne.server Configuring endpoint tcp:port=8000:interface=127.0.0.1
2023-12-05 22:33:32,807 INFO server daphne.server Listening on TCP address 127.0.0.1:8000

Hey! That looks like what I'd expect from running ./manage.py runserver on an average django project. Success! Now I just need to open my browser and get on with my testing.

What? No. Why? I set up the ports when running the container. I even remembered this time to add the -p 8000:8000 flag to the run command the first time! I even remembered that the first port listed is for what port you want to access on the host, and the second port is the port you want Docker to forward, even though that always feels backwards to me.

Okay, let's check the container and see what's up.



docker ps

. . .

d8ccf5c1aa7a myapp "python ./myapp/ma…" 6 seconds ago Up 5 seconds 0.0.0.0:8000->8000/tcp priceless_pascal

That all seems correct. Should be able to access localhost:8000 to get to whatever port 8000 on the docker container is doing. So what's happening here?

I kill the container, run the webserver in a different terminal tab, just to make sure I'm not going crazy. It loads fine. I keep the local server running, but also start up the docker container again, and now I can't access the app front end through the browser anymore. So clearly docker is doing something with port 8000. But letting me access my server isn't it.

I'll spare you the roughly three thousand google searches I did before arriving at the answer. Long story ~short~ somewhat less long, Python thinks we're exposing our API at 127.0.0.1:8000. This is what we want if we're just running the server locally. However, if we look at the ports output from docker ps, we're not exposing 127.0.0.1:8000. We're exposing 0.0.0.0:8000. One last tweak to the Dockerfile should do the trick.



FROM python:3.11
RUN pip install poetry
COPY . .
RUN poetry config virtualenvs.create false --local
RUN poetry install
CMD ["python", "./myapp/manage.py", "runserver", "0.0.0.0:8000"]

Building the image works. Running the image works. Now I just need to cross my fingers and check my browser.

Thank god. Time for a nap.

idle thought on my declining participation in dev communities

jess unrein — Fri, 03 Sep 2021 21:40:56 +0000

It’s been a good long time since I’ve posted here, attended a meet up (digital or otherwise), or engaged on dev Twitter. I was wondering why that is, and I realized that this is the happiest I’ve ever been in my job, and when that happens, my urge to do side projects and write about software dries up a bit. Not sure I have a grand thesis here. It’s great to have a job that’s fulfilling and allows work life balance in a way that I’m unused to. However, my lack of drive makes me a little sad because in the past I’ve gotten a lot of value out of my interactions with the community.

Has anyone else experienced something similar? How does having a comfortable job versus an overly stressful job impact your community involvement, home life, or other behaviors? If you’re now primarily work from home and weren’t before, how does that impact where you spend your time? I’m eager to hear other perspectives.

how is everyone holding up?

jess unrein — Tue, 21 Apr 2020 03:34:28 +0000

hey folks. turns out having a soul crushing job and lack of interesting projects to work on can really decrease your blogging output. also, I had a very bad mental health 2019.

i was laid off at the end of march due to covid related sales pipeline issues, leading to a lack of cash in hand, and it turns out to have been the best thing to happen for me in months. don’t get me wrong, it was incredibly nerve wracking but i didn’t realize how drained i was until I didn’t have to go in to work anymore. i start my new job next monday and i’m feeling more empowered and excited than i have in a while.

one thing i wasn’t expecting about my job search is that, because i’d done so little quality coding over the last few months, i was terrified of coding exercises in the job process. my job search probably took longer than it needed to because i would just stare at those codility emails in fear, wondering how badly i was about to screw up my future.

good news- once i started doing them it jumpstarted my brain and i was like “oh, i know how to do this. this isn’t so scary.” so for anyone out there who is also job searching amid the layoffs and procrastinating due to fear of failure, know 1. you’re not alone and 2. it’s unlikely to be as bad as you think. the waiting is the hardest part.

anyhow, that was a whole lot of rambling but i missed Dev and the community here, so i thought i’d dip my toe back in and see what’s up. what’s changed since i’ve been on hiatus? how are you holding up through everything happening with covid? come say hi in the comments. we’re all in this together.

programming languages as crafts and toys

jess unrein — Sat, 27 Jul 2019 17:31:50 +0000

Java: LEGO
Python: Knitting
Ruby: Crochet
JavaScript: Cross stitch
HTML/CSS: Sand Art
Web Assembly: Embroidery
Go: K’nex
SQL: Silly Putty
PHP: Pony Bead Melting
C: Glassblowing
COBOL: blacksmithing

I will not be explaining further or taking questions. It’s not rational. This is just something I feel deeply in my soul.

A PSA for Well Meaning Coworkers This International Women's Day

jess unrein — Fri, 08 Mar 2019 16:42:41 +0000

I usually try not to post twice in one day, but this is timely. My standup this morning started with the following exchange.

Dude: Oh, yeah, Carol is at that women's thing. Jess, why aren't you there? What do you have against women?
Me: Well, I'm not a woman.
(group laughter)
Me: Yeah. I'm nonbinary.
(group silence)
Dude 2: I.....didn't know.

Now, I'm not militant at work about enforcing correct pronoun usage, and I don't remind people that I'm nonbinary all the time. But I do have it in my public profile at work and on slack, and it stings when the people around me put in so little effort.

But even if I weren't nonbinary, there are a few pieces of advice I'd like to give.

1.) Don't ask women why they are or aren't participating in a woman focused event. Not all women focused events are safe for every woman, and not every woman wants to center her career on her gender. This is a rude and uncomfortable question.

2.) Don't ask women to explain why women-centric spaces are necessary, or why they're okay when a "men's group" would be given side-eye. That's a question you can google for, or you can reach out to group organizers in private if you have earnest follow up questions.

3.) Don't ask women or nonbinary people what they have against women or why they hate women, even in jest. Don't make jokes at your coworkers' expense that are designed to make them defensive or feel bad. That's a shitty way to be a coworker, an ally, or a friend.

This has been a public service announcement

###

Driven mostly by spite, jess coded

jess unrein — Fri, 08 Mar 2019 16:09:34 +0000

I continued to code in 2019 because...

Coding for a living is like getting to solve puzzles for a living. I love that.
My first VP of Engineering told after only three months on the job that I just didn't seem "technical enough," and that I should consider a "people management" or product manager track instead. I hope he sees my entire career as a "fuck you" direct from me to him.
Gotta keep a roof over my head.

I deserve credit for...

Prioritizing my health and work life balance over the last 14 months. It's easy to fall into a rut, especially at a desk job where you don't have to interact with people as much as many other professionals do.
Taking time for myself.
Coming out as nonbinary at work.

I hope to see my tech community...

Start treating nonbinary people as legitimate, rather than "woman-lite."
Start including more WOC and nonbinary POC voices in leadership.
Commit to pay transparency.

3 Tricky Python Nuances

jess unrein — Wed, 16 Jan 2019 20:57:47 +0000

This blog post is adapted from a talk I gave last week at ChiPy, the Chicago Python User Group. For more great content like this, I highly recommend your local Python meetup, and PyVideo, the community project aimed at indexing every recorded Python meetup or conference talk.

Nuance Number One

Lists Don't Contain Objects

I know this sounds like a ridiculous statement. Everything in Python is an object. If declare a list, I'm filling it full of objects. Collecting objects is what a list does!

Kind of. Let me back up a bit.

Over Advent of Code a couple friends and I were sharing code snippets and debugging things together in our Friendship Discord. One of the problems involved creating a giant matrix that represented cloth, and you had to figure out which squares of cloth were being used by manipulating individual elements in that matrix. I have a friend who is learning Python right now, and setting the problem up turned out to be more difficult than they thought.

Here's how they set up their matrix initially. Simple enough, right?


l = [[0] * 5] * 5

print(l)
[[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]]

So far so good. But when they went to change an element in the first nested list...

l[0][3] = 1

print(l)
[[0, 0, 0, 1, 0],
[0, 0, 0, 1, 0],
[0, 0, 0, 1, 0],
[0, 0, 0, 1, 0],
[0, 0, 0, 1, 0]]

😧

They posted in our discord group, asking what the heck was going on. My answer, as usual, was pretty much to shrug and ask why they weren't using a list comprehension. Another of our friends said something to the effect of "seems like it could be pointers?"

My second friend was right. Python doesn't store actual objects inside a list. It stores references to objects. And it's hella weird.

A reference points to a specific location in memory. That specific location contains the object, and if you change the object stored at that location in memory, you change every instance where it appears.

So each of the five lists inside the original matrix declared by l = [[0] * 5] * 5 are actually just five separate references to the same list. When you change one, you change them all.

So wait, how do I make them different objects?

Great question! In order to accomplish the goal of having a matrix where each element can be manipulated independently without inadvertently affecting other elements, we need to make sure we're creating a new object at a new location in memory for each nested list. In this case, we can use the copy library to create a new container object that we can change independently.

from copy import copy
a = [0, 0, 0, 0, 0]

b = []

# intentionally verbose example to illustrate a point :)
for _ in range(5):
    b.append(copy(a))

But of course, we can make this much cleaner. I was right the first time. Mooooooost of your Python problems can be solved with a well placed list comprehension (satisfaction not guaranteed).

l = [[0 for i in range(5)] for j in range(5)]

l[0][3] = 1

print(l)

[[0, 0, 0, 1, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]]

Nuance Number Two

Default arguments are evaluated when the function is defined, not each time a function is called

What does this mean? It's easier to explain through an example. Let's pretend that, for some reason, we want to write a function that takes in an element and appends it to an existing list without using the append function.

def list_append(element, input_list=[]):
    input_list.extend([element])
    return input_list

Now, this is a completely contrived example that we would never implement in practice, but it's useful for demonstration. Let's try calling that function a few times.

print(list_append(3))
[3]
print(list_append([5, 7]))
[3, 5, 7]
print(list_append("asdf"))
[3, 5, 7, "asdf"]

When we call the function multiple times, it seems that it's always adding to and returning the same object, rather than starting with a fresh empty list each time the function is called.

This is because the default argument for a function is created and evaluated when the function is first declared. So it would get evaluated and you'd start with a fresh list every time you restarted your app, for example. But each time your function is called while the app is running it's going to be working off of the same mutable object, and will just keep growing and growing until the app is restarted again.

Here's how we fix this:

def list_append(element, input_list=None):
    if not input_list:
        input_list = []
    return input_list.extend([element])

print(list_append(3))
[3]
print(list_append([5, 7]))
[5, 7]
print(list_append("asdf", ["bcde"]))
["bcde", "asdf"]

Elements declared within a function's scope are evaluated every time the function runs, so in the above example you'll be starting with a new empty list every single time, instead of passing the same default argument from function call to function call.

This isn't just a problem with lists, by the way. You should always be careful about passing in mutable objects as default arguments to a function; all mutable objects will have this same problem.

Nuance Number Three

`==` vs `is`

is and == are both operators in Python that evaluate two objects against one another and return a boolean value based on the results. This is a little tricky, but == checks for equality while is checks for identity.

Remember back to nuance number one when we talked about references, or specific locations in memory that contain an object? They're important again here!

Equality means that the values of two objects match. Identity means that the two things you're comparing are the exact same object and exist at the same location in memory. You can check the location of an object in memory by using the builtin Python function id(). So when you compare two objects using the is operator, you're checking that the two objects have the same id, not the same value.

A quirk of this, that you'll often see in ideomatic Python, is that there is only one True boolean object and only one False boolean object in memory. Each appearance of True has the same id as every other appearance of True. However, there are some values that evaluate as being equal to True without sharing its identity. That's why Pythonistas often use the is operator when comparing things to True or False rather than the == operator.

This can be a bit confusing and jargon filled when explained in words. Here are some examples.

Equality vs identity for boolean values:

print(1 == True)
True
print(1 is True)
False
a = True
print(a is True)
True


print(0 == False)
True
print(0 is False)
False
b = False
print(b is False)
True

Equality vs identity - equivalent variables

# a and b variables point to the same id
a = [1, 2, 3, 4, 5]
b = a
print(a == b)
True
print(a is b)
True
print(id(a), id(b))
4437740104, 4437740104

# a and b are different container objects, but the values of their contents are identical
a = [1, 2, 3, 4, 5]
b = [1, 2, 3, 4, 5]
print(a == b)
True
print(a is b)
False
print(id(a), id(b))
4437740104, 4442640968

Conclusion

I'm not sure I have a unified conclusion here. These are just three things that might trip you up in Python that are hard to google for, since they lead to unexpected behaviors but don't give you a nice error message.

I suppose if I were to give you a takeaway it's that
1.) references are kind of wonky. You usually don't have to think about them, but they're worth understanding
2.) Always use list comprehensions

As always, let me know if you have any questions in the comments, or if you think I missed anything important regarding these three topics. Let me know if there's a Python quirk you don't get and would like an explainer on. Thanks!

What Even Is Data Engineering?

jess unrein — Wed, 12 Dec 2018 16:25:42 +0000

I started my first job as a Data Engineer back in September. I wasn't exactly sure what the job entailed, but I was ready to shift my focus and learn new things. I'd spent the previous four years on back end application development, and while I learned a lot, I was ready to shift away from primarily consuming and constructing APIs.

When I was interviewing for the position, I read up on DAGs, ETL pipelines, and specific common database patterns like the star schema. However, in my interview I was asked things "how do you optimize this SQL query" and "assuming you don't have an automated load balancer in place, how do you decide which cluster to create a new database in?" They were interesting questions, so I decided to take the job. But it can be difficult to explain, even to other engineers, what the scope of my job is.

My team is responsible for database infrastructure. We make sure that database clusters aren't overloaded. We're concerned with internal data security. We manage database users and permissions, and field requests from different teams. We're the information gathering arm for the Business Intelligence team; if they need a dump of data from a specific API every 24 hours, we write the ETL pipeline to do it, and manage the data warehouse where all that information lives. My job is kind of a grab bag of things that need to get done but might not have an obvious owner.

I know I probably haven't done a great job of answering the question "What even is data engineering?" because I'm not exactly sure I know myself! I recently met up with Ali at a local Python meetup and we bonded over the fact that the discipline of data engineering seems utterly made up.

In this series, however, I'm going to introduce different tools that I use in my work through the lens of data engineering. If you have specific questions, or want to know about specific tools commonly used by data engineers, let me know in the comments. Data engineering is a fairly new field in software development, and the boundaries are still being drawn. Let's suss it out together!

Where do you find conference CFPs?

jess unrein — Mon, 10 Dec 2018 21:42:34 +0000

I did quite a bit of conference and meetup speaking in 2016 and 2017, but decided to take 2018 off of speaking due to some burnout and health concerns. But now I'm back! But the landscape seems to be totally different from the last time I was CFP hunting.

Lanyrd is permanently down, the Technically Speaking newsletter is now defunct, and Papercall.io continues to be an unhelpful mess.

For those of you who do conference speaking, which channels do you keep an eye on for finding new CFPs?

Different types of testing explained

jess unrein — Thu, 06 Dec 2018 21:47:54 +0000

In standup the other day, my team's DBA was talking about running smoke tests for his most recent project. I've heard people talk about smoke tests before, but for some reason it never really clicked that I have no idea what a smoke test is. How is it different than a unit test? An integration test? A regression test?

It feels a little embarrassing at this point that I can't articulate the difference between these things, so I decided to do a little research and write up an explainer so that I can reference it in the future and not feel like an ignorant dingus. I figured, since I've been working as a dev for almost 5 years and had this question, there are probably others out there who are similarly too shy to ask.

After reading a bunch of different blog posts, stack overflow questions, and random resources I've constructed a Frankenstein approximation of a consensus for several different categories of tests. After a little bit of time spent googling, I think there are three good things to think about to understand different kinds of testing.

1.) What kind of thing do they test?
2.) When are these tests written and run?
3.) What information does a test failure provide?

Different people have different definitions, and a single test suite might include multiple types of tests. For example, you might have a set of tests you run that combine integration tests and regression tests into a single suite. That's fine. There are grey areas, and teams have a habit of developing their own, team-specific vocabulary. You don't need to have a comprehensive suite for each of these categories. You should test at the level that makes sense for:

the complexity of your app
the amount of traffic your app sees
the size of your team

If you think I've radically mischaracterized or omitted something important, especially if you work in testing, please let me know in the comments!

Unit tests

What do they test?

Unit tests evaluate that each atomic unit of code performs the way it's supposed to. Ideally, when you're planning and writing unit tests, you should isolate functionality that can't be broken down any further, and then test that.

Unit tests should not test external dependencies or interactions. You should definitely mock out api calls. Unit test purists would also have you mock out database calls and only ensure that your code operates correctly given correct inputs from outside sources. Depending on your existing codebase or your manager's preferences, this might not be possible. If you aren't able to exclude database functionality from your unit test suite, make sure you are mindful of performance and look for potential optimizations. I can tell you from experience that long running unit test suites are extremely unpleasant and slow down development significantly.

When do I run them?

You should write and run unit tests in parallel with your code. When people refer to Test Driven Development, they're referring to unit tests, and using the tests as the spec for what your code should accomplish.

What happens when they fail?

A failing unit test lets you know that a specific piece of code is busted. If you've broken it down far enough, your failure should zoom in on the exact piece of code that isn't working as intended.

Failures should help you identify and fix problems quickly, and let you know when your specs need to be updated. They're probably a good guide for when to update your code documentation as well.

Integration tests

What do they test?

Integration tests check the interaction between two or more atomic units of code. Your application is composed of individual units that perform specific small functions, and each of those small functions might work in isolation but break when you knit them together.

Integration tests also test the integration of your code with outside dependencies, like database connections or third party APIs.

When do I run them?

Integration tests should be the next step after unit tests.

What happens when they fail?

When an integration test fails, it tells you that two or more core functions of your application aren't working together. These might be two modules you've written that clash in some complicated business logic, or a failure resulting from a third party API changing the structure of their response. It might alert you to bad error handling in the case of a database connection failure.

Failures might be easy to identify, or they might require some manual validation and experimentation to identify. Difficult to solve integration test failures are an indication of where you can improve your logging and error handling.

Regression testing

What do they test?

Regression tests check a set of scenarios that worked in the past and should be relatively stable.

When do I run them?

You should run your regression tests after your integration tests pass. Do not add your new feature to the regression test suite until existing regression tests pass.

What happens when they fail?

A regression test failure means that new functionality has broken some existing functionality, causing a regression.

The failure should let you know what old capabilities are broken, and indicate that you need to write additional integration tests between your new feature and the old, broken feature.

A regression test failure might also indicate that you have inadvertently reintroduced a bug that you fixed in the past.

Smoke testing

What do they test?

Smoke tests are a high level, tightly curated set of automated tests that live somewhere in the space between integration and regression tests. They're there as a sanity check that your site's core functionality isn't wrecked.

The term smoke test seems to be a holdover from plumbing. If you could see smoke or steam coming out of a pipe, it was leaky and needed to be fixed.

When do I run them?

Smoke tests should be a test of your whole system together, ensuring that core functionality remains intact. These shouldn't be comprehensive. These are your significant, big picture, no-go test failures. You should run them early and often, ideally daily, in both staging and production environments.

What happens when they fail?

If a smoke test fails there's a significant problem with your site's functionality. You should not deploy the new changes until these failures are addressed. If they fail in production, fixing these should be very high priority.

Acceptance testing

(I've also heard this called QA/BV/Manual testing, etc.)

What do they test?

Acceptance testing is usually a set of manual tests performed after the end-to-end development is finished. They check to make sure that the feature as written actually meets all of the initial specifications, or acceptance criteria.

What happens when they fail?

Looks like you missed a bit of functionality when writing your code. You'll need to go back to development and fix that. :(

If acceptance tests fail you probably need to decide on acceptance criteria earlier in your planning process next time.

When do I run them?

Since these are manual tests, not tests run as code, the timing is a little different. You and your project owner should draft a set of acceptance criteria before work begins on a project. Any additional scope that's discovered or added to the project should be reflected in the acceptance criteria.

Acceptance tests should happen fairly quickly after development is complete so that you can go back and iterate quickly if something isn't quite right. It makes sense to do these right after unit or integration testing, before you've gone too far in the testing process before significant changes need to be made.

Performance testing

What do they test?
Performance tests check stability, scalability, and usability of your product and infrastructure. You might check things like number of errors per second or how long it takes to load a page. There isn't necessarily pass/fail criteria associated with a performance test. This stage is more about data gathering and looking for areas of improvement.

What happens when they fail?
Performance tests don't exactly fail in the same way that a unit test suite would fail. Instead you collect a set of benchmarks and assess them against where you want those numbers to be. If your performance test fails, it might tell you that you need to pay more attention to infrastructure scaling, database query time, etc.

When do I run them?
Performance tests are a good idea after major releases and refactors.

Load testing

What do they test?
Load testing is a kind of specialized performance test that specifically checks how your product performs under significant stress over a predetermined period of time.

What happens when they fail?
Load tests assess how prepared you are for a significant increase in traffic. If a load test fails, it doesn't mean that your site is broken, but it does mean that you aren't prepared for a viral hit or a DDOS attack. This is probably not a big deal for small products just starting out, but failure should be a concern as your userbase starts to scale.

When do I write them?
Load tests should not be your first concern right out of the gate, but as your product becomes bigger and more established, you should probably run load tests on new features to see if they will affect the overall performance of the site and see if they can be optimized.

I can no longer say I don't know what a smoke test is, and hopefully you learned something along the way too! As I mentioned above, I am not a tester, so if you notice something I've missed or misinterpreted, let me know in the comments!

Papers We Love: Exploring Gamification Among Elderly Persons

jess unrein — Mon, 03 Dec 2018 17:08:50 +0000

Welcome back to Papers We Love! Sorry for the absence last week - the holidays kind of took the pepper out of me and I fell behind. BUT I picked out a paper on gamification among frail elderly patients that I think will lead to a really good discussion (and isn't too long of a read :) ).

I picked this paper somewhat selfishly. My dad worked in geriatric practice when I was growing up. As a result, I spent a lot of time tagging around with him to nursing homes and interacting with patients. I'm very interested in the intersection of elder care, healthcare, and technology. It's almost like this paper was tailor made for me.

I think the paper did a great job of outlining possible gains as well as technical, cultural, and accessibility challenges. However, I was a bit surprised and dismayed that the paper didn't touch on possible ethical and privacy ramifications of this kind of data gathering. Sure, you might be able to observe a decline and medically intervene earlier. But how is this information being collected and stored? Who has access, and is participation voluntary for the patient? Will this integrate with an Electronic Medical Record, and will performance in gamified activities impact insurance rates or treatment options?

I am incredibly wary of policies and technologies that treat elderly patients as having fewer rights than other people. It's a trap that can rob patients of their dignity at the same time they're being robbed of their mobility, senses, and ability to live autonomously, and I think it's imperative to keep these questions in mind when integrating technology with elder care.

Dockerizing a Simple Python Process

jess unrein — Fri, 16 Nov 2018 16:14:49 +0000

This is part two in a series on taking a simple Python project from local script to production. In part one I talked about a gotcha I ran into when converting an old project from Python 2 to Python 3.

This part will go over how I put my Python process, its inputs, and its outputs into a Docker container and made an image publicly available on Dockerhub.

Requirements that I will not go over here. Go to Docker.com and follow the instructions there

Download docker
Create a docker id
Log in with your docker id on Dockerhub

What is Docker?

Docker is a containerization platform. Containerization is a way to package units of code with their dependencies so that they have everything they need to run in isolation.

Using Docker can help fix the "it works on my machine" problem, and writing dockerized code is a great way to encourage thoughtful code practices. Docker containers should be simple, responsible for as little as possible, and dependent on as few externals as possible.

Docker image vs docker container

Throughout this post, and online, you'll see the terms container and image. An image is basically a snapshot of your dockerized code that is created when you use the docker build command - more on that below. Docker images start a container when you use docker run on that image. So a container is a running instance of an image.

Anatomy of a Dockerfile

I decided to dockerize my csv writer from the previous post in this series so that I could move it between environments easily.

For this I needed a Dockerfile. A Dockerfile is a text file that does not have a file extension.

Here's what the dockerfile for my Python code looks like:

FROM python:3.7
ARG export_file=goodreads.csv
COPY $export_file goodreads_export.csv
COPY converter.py /
CMD ["python", "./converter.py"]

FROM

The FROM keyword here indicates a dependency. Docker containers don't have languages automatically loaded. To access Python to run the code, we need to instruct the image to include python:3.7.

A note on Docker registries:
the default Docker registry is Dockerhub. If a docker image is available on Dockerhub, you don't need to specify a url when pulling or pushing from a docker repo. You just need the author's username and the repo name. For example, you can pull the docker image from this post with the command docker pull thejessleigh/goodreads-libib-converter. If you're using a different registry you'll need to tell Docker where to go. For example, if you're using Quay you'd do docker pull quay.io/example-username/test-docker-repo.

The python dependency in my Dockerfile doesn't have a username because it's an official repo hosted on Dockerhub.

ARG

ARG declares an argument. It is the only instruction in a Dockerfile that can precede FROM, although I prefer to have FROM come first for the sake of consistency.

In the above example, I declare an ARG export_file and give it a default. It expects a file called goodreads.csv in the same directory as the Dockerfile. If I want to pass in something different, I instruct it to use a different filename with --build-arg=export_file=my_goodreads_export.csv when building the image.

COPY

COPY and ADD duplicate the contents of a file into the docker image. This is where I'm importing the input file and also the actual Python code that the Docker image executes.

COPY takes two arguments:

the location of the file you're putting into the image
the location of the file inside the docker image

So whatever file I include as the CSV to convert will be referred to as goodreads_export.csv inside the Docker container. This is nifty, because it means that no matter what I build the docker image with, the filename will always be consistent. I don't have to worry about making the Python code handle different filenames or paths. It can always look for ./goodreads_export.csv.

There are some subtle differences between COPY and ADD that @ryanwhocodes has already written about, so I'll leave his post here.

Update September 2019: It appears that this post is no longer available on dev.to so I have replaced the embedded post with an archive.org link.

RUN

RUN issues an instruction that is executed and committed as part of the image. If I were dockerizing a Python project that needed to install external packages, I could use RUN to pip install those dependencies. However, converter.py is a very simple process that doesn't need external packages, so I don't need to run anything as part of my build process.

CMD

There can only be one CMD instruction per Dockerfile. If the Dockerfile contains multiple CMDs, only the last one will execute.

CMD is the command you intend the image to do when you run an instance of it as a container. It is not executed as part of the build process for an image. CMD is different from RUN in this way.

Building a docker image

Now we have everything necessary to build a Docker image for our Python code from the Dockerfile.

As stated above, a Docker image is an inert snapshot of an environment that is ready to execute a command or program, but has not yet executed that command.

To build using the above Dockerfile, we run

docker build --build-arg=export_file=goodreads_export.csv -t goodreads-libib-converter .

--build-arg tells Docker to build the image with a file called goodreads_export.csv, overriding the default expectation of goodreads.csv.

-t goodreads-libib-converter "tags" the image as goodreads-libib-converter. This is how you create your container with a human readable REPOSITORY name.

. tells Docker to look for a Dockerfile to build in the current directory.

After I do this, I can see that the image was successfully created by checking my image list.

> docker image list
REPOSITORY                 TAG       IMAGE ID       CREATED             SIZE
goodreads-libib-converter  latest    1234567890     12 seconds ago      924MB

Running a Docker container

Now that I have an image, I have a standalone environment capable of running my program, but it hasn't actually executed the core procedure specified with CMD yet. Here's how I do that:

docker run goodreads-libib-converter

I see the print debugging statements I have in my converter.py file execute, so I know how many CSV rows are being converted. When I ran the program locally, it created an output file called libib_export.csv. However, when I check the contents of my directory now, it's not there. How is that useful!?

Accessing Files Written Out

I'm no longer running the Python code in the directory I was before. I'm running it inside the Docker container. Therefore, any files that are written out will also be stored inside the Docker container. The output file doesn't do me much good in there!

I'm running the Docker container locally, so all I have to do is find the container and copy the output file from it's dockerized location to the place I actually want it.

docker cp container_id:/libib_export.csv ~/outputs/libib_export.csv

This extracts the resultant CSV output from converter.py and puts it somewhere I can access it.

I can figure out the container_id (or the human readable name) with

> docker ps -a
CONTAINER ID  IMAGE                   COMMAND                  CREATED             NAMES
e00000000000  goodreads-libib-export  "python ./converter.…"   24 seconds ago      naughty_mcclintock

Yes, naughty_mcclintock is actually the procedurally generated name for the container I've been working with locally.

Copying a file from a container to my desired location is fine for a local environment, but has limited uses if I ever want to take this project to production. There are other, better options for dealing with output files from Docker containers, but we'll get into that ✨ in another installment in this series ✨

Committing a docker image

After we've run the container to confirm that it works, we probably to create a new image based on the changes it made when it executed. We're preparing the image that we want to push up into an external Docker registry, like Dockerhub.

When committing a Docker image, we need to specify the registry (if it's something other than dockerhub), the author name, the repository name, and the tag name.

docker commit -m "Working Python 3 image" naughty_mcclintock thejessleigh/goodreads-libib-converter:python3

My docker commit was successful, so I see a sha256 hash output in my terminal. Creating a commit message is, of course, optional. But I like to do it to keep organized.

A note on Docker image tags:
When you pull a Docker image and you don't specify a tag it will use the default tag (usually latest). Tags are the way you can keep track of changes in your project without overwriting previous versions. For example, if you (for some reason) are still using Python 2, you can access the Python 2 image by running docker pull thejessleigh/goodreads-libib-converter:python2. Right now the :python3 and latest tags on my rocker repo are the same, but you can pull either one.

Pushing a docker image to Dockerhub

Now that I have an image I want to put out into the world, I can push it up to Dockerhub.

First, I need to log into Dockerhub and create a repository. Repositories require a name, and should have a short description which details the purpose of the project, and a long description that explains dependencies, requirements, build arguments, etc. You can also make a Docker repository private.

Once I've done that, I run docker push, which sends the latest commit of the project and tag I've specified up to the external registry. If you didn't specify a tag, this push will override the latest tag in your repository.

docker push thejessleigh/goodreads-libib-converter:python3

If you go to my Dockerhub profile you can see the goodreads-libib-converter project, and pull both the Python 2 and Python 3 incarnations.

Next Steps

Now that I have a working Docker image, I want to put it into production so that anyone can convert their Goodreads library CSV into a Libib library CSV. I'm going to go about this using AWS, which requires a bit of setup.

The next installment in this series will go over setting up an AWS IAM account, setting up awscli and configuring your local profiles, and creating an s3 bucket that your IAM account can access.

EDIT: Never did get around to that next post in the series. I should do that someday.

DEV Community: jess unrein

Launching a django webserver in Docker

idle thought on my declining participation in dev communities

how is everyone holding up?

programming languages as crafts and toys

A PSA for Well Meaning Coworkers This International Women's Day

Driven mostly by spite, jess coded

I continued to code in 2019 because...

I deserve credit for...

I hope to see my tech community...

3 Tricky Python Nuances

Nuance Number One

Lists Don't Contain Objects

So wait, how do I make them different objects?

Nuance Number Two

Default arguments are evaluated when the function is defined, not each time a function is called

Nuance Number Three

== vs is

Equality vs identity for boolean values:

Equality vs identity - equivalent variables

Conclusion

What Even Is Data Engineering?

Where do you find conference CFPs?

Different types of testing explained

Unit tests

Integration tests

Regression testing

Smoke testing

Acceptance testing

Performance testing

Load testing

Papers We Love: Exploring Gamification Among Elderly Persons

Dockerizing a Simple Python Process

What is Docker?

Docker image vs docker container

Anatomy of a Dockerfile

FROM

ARG

COPY

RUN

CMD

Building a docker image

Running a Docker container

Accessing Files Written Out

Committing a docker image

Pushing a docker image to Dockerhub

Next Steps

`==` vs `is`