DEV Community: Charles Landau

GitOps Superpowers

Charles Landau — Wed, 19 May 2021 21:23:16 +0000

I've been using GitOps for the past few years and I wanted to reflect briefly on what I've learned, how it helps, and how it breaks down.

The term GitOps was developed by Weaveworks in 2017. It's a technique where you put the desired state of your system in a version controlled repository. When changes are made in the code, those changes get applied to your system automatically (they are "deployed"). There are lots of tools that implement this, including Weaveworks' own FluxCD tool, Terraform Cloud, and Vercel.

For a deeper dive on GitOps, gitops.tech is a great place to start.

Since I started using it, GitOps has made it considerably easier for me to build. Considerations like governance, tracking changes, and testing are much less daunting. Generally, what I like to do now is require changes to get a peer review before being accepted. For more sensitive environments some providers will let you assign ownership to specific parts of the system, so that those owners must be the ones to approve changes. This is great for enforcing the "four eyes principle" - that changes can't get made by a single person. Sometimes it can be good to define the desired infrastructure alongside software, as with Vercel or many projects that use Helm. In other cases it's much better to avoid this, such as when you're using GitOps to manage a cloud account.

All the benefits of defining systems with code roll up to the GitOps approach. Changes in the repository can be tested automatically, you can write reusable libraries, you can try your changes in test/QA environments before accepting them. The nice thing here is that you can accomplish this using git branches! Your CI tool probably has prebuilt jobs that can cover many of these use cases.

Unfortunately, not everything can always be defined declaratively and deployed automatically and on time. There are a lot of factors that could drive you back to "ClickOps". The most obvious one is when the code doesn't support a resource you need (e.g. there is no Terraform implementation for what you're doing.) Two different problems arise when this happens. The first is that the steps you took manually all add up to a debt, and they need to be taken again if you ever need to redeploy. They need to be documented, and they need to be automated. Failing to document them is very bad, and the task of automating them has a way of languishing in the backlog. The other problem is that you have chipped away at one of the benefits of GitOps. We want the team to trust that the source repository describes the desired state very well. If we have enough manual workarounds tacked on to that desired state it becomes much harder to look at the source repository and understand what's going on.

In addition to the problem of manual workarounds, there can be some other challenges with GitOps having to do with bootstrapping and kickstarting.

Bootstrapping a GitOps-managed system can be the cause of a lot of navel-gazing and Rube Goldberginess. There is usually no practical reason to automate every aspect of the bootstrapping, and it's turtles all the way down so why bother? (I have a strong urge to automate all the things, so this is mostly advice to myself.) An illustrative example: You may want to make an AWS Account Factory, but you wouldn’t write a puppet to automate the root account provisioning.

It can sometimes be necessary to "kickstart" resources as well, wherein we delete something that's in the desired state definition, like a container or storage volume, so that the automation pipeline can detect the missing resource and recreate it. There are a lot of different reasons this can end up being needed, often tied to implementation details that are at a lower level of abstraction than you're dealing with. (Immutable infrastructure methodologies seem to have a bunch of related problems here.)

Despite the challenges, I don't think I'll be transitioning out of GitOps any time soon, and in fact I want to manage more things with GitOps. It feels faster to click through the AWS dashboard and make an S3 bucket, but it's fast in the same way writing code without tests feels like saving time. I think there's more opportunity for data and analytics resources to be managed by GitOps, and treating configuration like data opens up all kinds of new possibilities.

Cover Photo by Zach Reiner on Unsplash

Things I Wish I Knew as a New Python User

Charles Landau — Fri, 26 Mar 2021 18:04:26 +0000

This post originally appeared on the The Slip Box

Last week I responded to the weekly #DevDiscuss with a long thread about Python, and this post is the refinement, expansion, and all-around long-form expression of those same ideas. These are all the all the things I wish I knew when I was just starting to use Python, with the benefit of hindsight.

Basics

There are a million and one tutorials out there to get you up and running with the syntax.

Pick any tutorial written for Python 3.6+ and ignore anything written for Python 2. Python 2 is deprecated. They are not all the same, so sample some, and devour at least one deeply. Revisit a few times. FreeCodeCamp wasn't on my radar when I learned Python, but it comes highly recommended. They seem to have a very robust Python course.

Do not expect of yourself to memorize every syntax rule. Many (most?) developers still have to look things up after many years of working with a language, including me.

Read PEP 8 and PEP 20.

PEP 8 is the style guide. Tools like Flake 8 and Black will help you adhere to it easily. Don't follow it blindly!

A foolish consistency is the hobgoblin of little minds. — PEP 8

PEP 20 is the Zen of Python, which you can refer to whenever somebody tells you something nonsensical with the word "pythonic" mixed in. (We can joke, but the truth is that Python has the power to be beautiful, and that matters to me.)

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than right now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

It has been years, and I am still learning to write code that embodies these ideas.

Tools

Spread your investment in your environment out

I will never settle on a "perfect" IDE configuration.

DO: Install the basic Python extensions for your code editor, plus some productivity/workflow extensions like Git

DO: Try out package-specific extensions for packages you'll be using

DO NOT: Get stuck editing your IDE into a complex configuration that distracts from writing code

Use this excuse to learn git if you aren't already

Eventually, you will want to share, collaborate, or reuse your code, and then you will want something like Git. I have never heard a developer say "Dang, I wish I had never learned git". In the past, I have been a partisan against anything but the git CLI, but I was wrong. I still think it's hard to learn the concepts without going to the command line, but if you can do it in the IDE or a desktop client then go for it.

The Standard Library, and Other Packages

stdlib is a gold mine. sys, os, json, asyncio, http, uuid, unittest, and so many more! Find them all here. These libraries are preinstalled with Python!

Beyond stdlib there's a whole universe of packages. Python has one of the most vibrant and awesome ecosystems around. This awesome list organizes a bunch of those libraries into categories for you, and there are many more out there for you to find on PyPI. You can do it with pip!

pip install mypackage1 mypackage2 mypackage3 to download your packages
pip freeze > requirements.txt to save your installed packages to a file (the file name is conventional)
pip install -r requirements.txt to install all the packages in requirements.txt

Practices

Use and dispose of virtual environments often

Python uses "virtual environments" to isolate dependencies. You can use anything you want, and I like Poetry a lot, but on Linux I generally start a new project with a simple set of commands:

git init my-py-project
cd my-py-project
python3 -m venv venv
source ./venv/bin/activate

There is a repo of starter .gitignore files maintained by GitHub including one for python. It already includes your venv file. Copy it into every project. Never look back.

Last thing to note: you probably don't need conda. It's a really great tool, but some folks start out going straight to conda but I think Poetry or venv is enough for now.

Static Type Checking

With mypy, you can add static typing to Python. Famously, Guido Van Rossum, the creator of Python, led a team at Dropbox to add mypy type checks to four million lines of code. New versions of Python have been adding language features that complement type checking.

For quick, one-off projects, you might skip mypy. If you're going to be working with a project for a significant period of time, static type checking can eliminate entire categories of errors from your Python.

Documentation is a Superpower

PEP 257 and a few others define "docstrings". Leverage them to make full use of autodoc tools. pdoc is a pretty fun tool that "just works". Build good habits from the start. Projects that have great documentation are just more attractive to me. If I come across a project that seems to do what I need, but has crappy documentation, I keep looking.

Any code of your own that you haven’t looked at for six or more months might as well have been written by someone else. — Eagleson's Law

Thank You Python

Python has truly changed my life. No kidding. I love python, it has given me so much joy and I owe it so much. I hope more people learn and love python.

Photo by Prasad Panchakshari on Unsplash

The Wrong Question About Docker and Kubernetes

Charles Landau — Mon, 18 May 2020 21:17:35 +0000

Not too long ago I discovered the work of a very smart and snarky cloud economist by the name of Corey Quinn. There’s a lot to admire in how Corey delivers content, his perspective on the industry, and his sense of humor. Having said that, I’ve been on the k8s learning curve for a little while now, and Corey’s pretty bearish on the topic.

Corey Quinn

@quinnypig

If you run Kubernetes in production, you must be incredibly intelligent.

That’s not a compliment; it’s a requirement.

17:13 PM - 25 Apr 2019

Corey is one of many cloud computing advocates taking this position, offered here as an example. He’s not the only one, and I even see Kelsey Hightower, one of the original evangelists for k8s, urging some caution. “Up and Running”, a book Kelsey co-authored, was my introduction to k8s, and I’ve tried to pay close attention to Kelsey ever since.

Kelsey Hightower

@kelseyhightower

So you want to roll your own application platform. All you need is:

Linux
Docker
Kubernetes
Istio
Prometheus
Fluentd
Grafana
Jaeger
Harbor
Open Policy Agent
Vault
Spinnaker and Jenkins

Oh, almost forgot, you're also going to need servers, people, and glue. Bring lots of glue.

01:32 AM - 03 Apr 2020

I think all of the above is fair, and I think the point about k8s complexity is hard to argue with. It’s easy for somebody to read expert opinions about this technology and get the wrong signal, because you might be starting with the wrong question. In this post, I’m going to urge you not to begin with the question “do I need docker or kubernetes?” Instead, begin with the question “could we get value from the technical and management practices required to express a Dockerfile? A k8s podspec?”

The iceberg of business value

Very Scientific Chart

As my question suggests, I have come to believe that the business process required to write a Dockerfile is more valuable than the tooling itself, and can require much more effort than the actual investment in Docker. Is there scripting in place to automate the build and run processes in a reproducible manner? Are all the dependencies fully documented? Have you followed the 12 Factor App framework or similar? Addressing these questions is where most of the value comes from, and Docker happens to be a particularly successful technology that helps organizations deliver after they’ve done so.

The same can be said about your services. The business and technical processes can go beyond expressing their build and run steps in a Dockerfile and can be extended to expressing their services as k8s podspecs.

Lastly, don’t trivialize that this is a genuine question about value! Maybe it makes more sense for your team to go the way of the Majestic Monolith. Go where the value is regardless of hype. If the technology you want to work with isn’t valuable to the shop you work in, perhaps that’s a signal for you to consider other employment, not a signal for the business to change value streams.

Closing thoughts

As I mentioned, I’m still on the learning curve for these technologies, so please take these words with a grain of salt. My hope is that this post has given you an alternative point of view on value creation in the container and container orchestration space. It’s hard to stay grounded when you’re learning about very hyped technologies, and refocusing on business value is how I’ve learned to cope.

Greedy Learning and Rapid Adoption

Charles Landau — Wed, 25 Mar 2020 19:46:48 +0000

A long time ago I gave up on knowing anything about coding, tech, or development. Knowing things takes a long, long time.

My memory is not great and even if it was, it's hard to justify technical decisions with "...well I remember the docs said xyz." So I gave up on remembering things too.

Nonetheless, I want to explore and learn about new tools constantly. I want to be unconstrained in the solutions I compose. The only way I can live like this is to cheat a little bit.

The Greedy Part

"Greedy learning" is a mental model for rapid adoption of new technologies. It essentially means that we start with the laziest, most intuitive search, and graph everything from that point on.

The root node of the graph is you, because we're being greedy. The only intermediate node before we do a search is your intent. Let's say that we want to perform web requests in Python.

Our initial knowledge graph is:

self -> intent:"web requests python" -> search

Most of the time, you will perform the search on a public search engine, the intranet at your job, or a REPL. If you are very fortunate, you may have an "enterprise search" tool like Spotify Lexikon that makes intent-based search trivial. Regardless, at this step you must become as lazy as you possibly can.

For example, I have found that if I am interested in a topic, I can write awesome at the beginning of a search and I will often find an awesome list about it. Just like that, we have many subgraphs to explore.

search results "awesome data engineering" ->
        intent:"explore_subgraph" ->
            root_graph_update:new_search

Likewise, if I think I can guess the answer, my search may begin at the REPL, where I test my guess, and the error messages I invariably generate become new searches.

search results "print foo" ->
    errors ->
        intent:from_error ->
            root_graph_update:new_search

We want to do all of this fast and lazy. We're speed-reading in the F-shape pattern. Everything should make intuitive sense. If it doesn't make intuitive sense, that indicates that we're missing something upstream in our knowledge graph.

To extend our intent:"web requests python" example, something about query parameters may come up which doesn't make intuitive sense. We don't abandon our scanning pattern and start deeply reading, but instead we generate a new intent to update our query parameters subgraph.

Rapid Adoption

At this point you may be thinking that I've created a long-winded manual for competitive Wikipedia surfing. Or a deep rationalization for my NADD. I hope I've done neither, and I think "rapid adoption" is the missing piece that sets this apart.

In as few iterations of greedy learning as possible, we need to learn a schema to satisfy our root intent. Fundamentally, this schema defines an actionable subgraph of nodes. Ideally, each root_graph_update modifies the schema, or we come to our problem with clear expectations. As quickly and as lazily as possible, we have a minimum actionable subgraph. Again, our "no memory or knowledge" assumption doesn't require us to start from scratch. It is perfectly valid to start with a schema that is intuitive so long as you update it frequently based on search results and your use case.

A Complete Example

(Full graph on Whimsical)

We'll begin with a basic root_graph as described above, for our example of a web request client in Python.

The root graph when we look for "python web request" tools

We have an intuition that a good tool for us will have some obvious qualities:

It will have a strong community of support, such as lots of answers on Stack Overflow, or frequent releases on GitHub.
It is written in, or has binding for, the language we care about.
It has the required level of performance.
It is productive. I certainly struggle to measure the productivity of software libraries. The number of lines of code required to complete a basic task is, perhaps, a good proxy.
The interface fits our use case. For web requests, we're looking for a way to make the request, and read the response.
- We may also intuit that the response object should have methods like .status_code or .json but let's keep our example simple for now.

With our schema prepared, we proceed to search.

For me, the simplest and most intuitive search for this intent is to just restate the intent and append the word "library", ensuring that I get more results that can be imported, rather than tutorials.

The first thing that jumps out at us is stdlib.http.client — given that we are exceedingly lazy and greedy, we have a strong bias for standard library tools which require no installation.

Upon arriving at the docs, we discover the following callout:

See also: The Requests package is recommended for a higher-level HTTP client interface.

We don't need to read any further. In the terms of Kent Beck's 3X framework, we are firmly in the "explorers" camp. We leave the tab open so we can always come back to this page if the high-level interface doesn't meet our needs, and we move on to the requests package.

The requests docs immediately tell us a couple of things:

The GitHub for this tool has more than 42,000 stars from developers and it's maintained by the Python Software Foundation, who also maintain Python itself. Strong community ✔️
That this tool has a Request object and Response object ✔️

So, we go hunting for the interface docs, and scanning through the quickstart. Reading a quickstart tells us which features the maintainers consider "core" to their user's experience. This matters because, as explorers, we have a strong preference for finding tools that support the features we care about as a core value, rather than kludges.

From our search, multiple subgraphs

With the complete documentation of our Request and Response objects in hand, we are ready get our hands slightly dirty. We can perform a simple validation that the core functions work in a REPL:

A smoke test of our search results

At this point we are ready to check our subgraph against the schema. In the schema I outlined, we would do this by asking: Is it productive, performant, and Python? Does it have an interface to the features we need? Is it supported by a healthy community?

In your schema, perhaps none of these questions will matter, and you'll care about something else.

If our subgraph is valid, we can move to adoption. "Adoption" means:

Slow way down. Up until now we've been trying to go as fast as possible. You could even write unit tests if you want.
- You might even want to stand up and stretch for a while before starting this part. Get into a different mindset.
With the tabs for each part of the interface open, write the interface calls into our real use case.
Refer to the appropriate docs actively as you write.
Run the solution and refer to the open tabs for errors.
If the solution doesn't work, update the root_graph accordingly and resume your search or generate a new search.

Arguments and Persuasion

I should end by warning you that this method has never worked for me where depth is the point. The way I think about this is as a distinction between arguments and persuasion.

Arguments: making strong logical connections in support of a thesis (as with a knowledge graph). "These are the reasons for this change, and how."
Persuasion: knowing the audience deeply and speaking to them from a place of radical empathy. "I know you, and what you want is this change."

If there's anything that fits this mental model of arguments, it's interfaces. An API documents a clear argument: Given xyz parameters and abc state, an endpoint /api/v2/users should return an instance of Response/Users.schema. Another example is troubleshooting.

Recording all of this has certainly been interesting for me as an exercise, and I hope you've found it useful. I'm always trying to learn more, so if you have feedback, let me know!

Data Engineering vs Data Science

Charles Landau — Mon, 27 Jan 2020 12:12:23 +0000

I was in a local technology Slack and saw the question:

Tell me, what is data engineering and how is it different than data science? What [Python] tools do you use?

It's a good question and it's worth considering. Here's what I think:

Ted Danson reading a slip of paper. Caption: "Okay, here we go"

Data engineers are concerned with how data:

Lands in a system
Moves through a system
Interacts with business processes and application logic
Is stored
Is governed

Definitions differ, but I think that's a good starting place.

There is a huge and diverse ecosystem of tools out there. I would highlight the following as tools with strong Python tooling:

Kafka (for pub/sub)
Spark for data processing (including the Spark Streaming API for stream processing)
Airflow for workflow management
Pandas for data "wrangling"

And you can find tons of tools on the awesome list.

Lastly, how do data scientists differ from data engineers? I would argue that they're both roles that are concerned with connecting data to business processes. They differ the same way their names differ:

A data engineer is concerned with designing and building systems that make data available and actionable in a cost-effective manner.
A data scientist performs experiments, the results of which are (sometimes) actionable insights or automation.

I hope that explanation helps you. If you see something I didn't explain very well, or got wrong, please chime in with that 👇

What do you think about the "Shape Up" project management framework?

Charles Landau — Tue, 26 Nov 2019 23:05:37 +0000

And if you haven't checked it out, here it is.

How does open source impact your local community?

Charles Landau — Wed, 06 Nov 2019 14:40:16 +0000

For the past few months I've been participating with Code for DC, a Code for America brigade. As part of this work I even joined a civic tech podcast for a discussion about one of the projects we're working on (I'll post that in the comments).

Like most volunteer organizations, I've seen how the leaders of the brigade struggle to understand the impact of their work. Anyway, all this led me to wonder how open source is impacting other communities. So, I thought it might be worth asking:

How has open source impacted your local community (if at all)?
Has your journey as a developer given you new tools you can use to help your community?
What are some ways that open source COULD help your local community, but currently doesn't?

Python 3.8 Has Been Released 🚀🚀 Let's Take a Tour

Charles Landau — Wed, 16 Oct 2019 15:55:56 +0000

Cover Image by Onur Ömer Yavuz from Pixabay

!python --version

Python 3.8.0rc1

Launch Page

Python 3.8 is a major release of Python. Here's a roundup of some new features.

New Syntax Rules!
Parallel Data Ops Improvements!
Static Type Checking Features!
CPython Stuff!

Let's start with a new syntax rule:

Assignment in Expressions (PEP 572)

Python now allows you to create variables inside of expressions (e.g. the body of a list comprehension.)

def f(x):
    for k in range(10000000):
        x*x*x/x+x+x+x+x
    return x

x = 2

# Reuse a value that's expensive to compute
%timeit [y := f(x), y**2, y**3]

# Without reuse
%timeit [f(x), f(x)**2, f(x)**3]

1.78 s ± 13.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
5.45 s ± 93.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Doing this in a "naked" expression is strongly discouraged in the PEP. See: Exceptional Cases

PEP 572

Here's one more new syntax rule:

Positional-Only Args (PEP 570)

This one confused me until I scrolled down to the syntax section, so let's just see it in action:

def prelaunch_func(foo, bar, baz=None):
    print(f"\nFoo:{foo}\nBar:{bar}\nBaz:{baz}")

def pep570_func(foo, /, bar, baz=None):
    print(f"\nFoo:{foo}\nBar:{bar}\nBaz:{baz}")

The / in the call signature of pep570_func denotes where the positional-only args end. These functions will let us see the practical difference PEP 570 causes.

prelaunch_func(foo=1, bar=2, baz=3)

Foo:1
Bar:2
Baz:3

# Violating positional-only
pep570_func(foo=1, bar=2, baz=3)

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-4-9e9851275486> in <module>
      1 # Violating positional-only
----> 2 pep570_func(foo=1, bar=2, baz=3)


TypeError: pep570_func() got some positional-only arguments passed as keyword arguments: 'foo'

# This is fine, as bar is right of the "/" in the call signature
pep570_func(1, bar=2, baz=3)

Foo:1
Bar:2
Baz:3

# This was wrong before PEP-570 of course
pep570_func(foo=1, 2, baz=3)

  File "<ipython-input-6-adcd14865e94>", line 2
    pep570_func(foo=1, 2, baz=3)
                       ^
SyntaxError: positional argument follows keyword argument

What I like best about this PEP is that you can also have positional-only args with default values, as shown here:

def pep570_func(foo, bar=None, baz=1, /):
    print(f"\nFoo:{foo}\nBar:{bar}\nBaz:{baz}")

pep570_func(10)

Foo:10
Bar:None
Baz:1

# But don't mistake them for kwargs
pep570_func(10, bar=1)

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-8-05c1c4b07ca6> in <module>
      1 # But don't mistake them for kwargs
----> 2 pep570_func(10, bar=1)


TypeError: pep570_func() got some positional-only arguments passed as keyword arguments: 'bar'

For more details check out the PEP: PEP-570

Now let's move on to some parallel data processing stuff:

Shared Memory and New Pickles

Two important improvements in 3.8 were inspired by libraries like dask that try to solve the problem of passing data between processes. One is to create a new version of pickle that can pass data objects between processeses using zero-copy buffers, for more memory efficient sharing.

The common theme of these third-party serialization efforts is to generate a stream of object metadata (which contains pickle-like information about the objects being serialized) and a separate stream of zero-copy buffer objects for the payloads of large objects. Note that, in this scheme, small objects such as ints, etc. can be dumped together with the metadata stream. Refinements can include opportunistic compression of large data depending on its type and layout, like dask does.

This PEP aims to make pickle usable in a way where large data is handled as a separate stream of zero-copy buffers, letting the application handle those buffers optimally.

PEP-574

The other biggie in this category is that processes have a new interface for sharing memory: SharedMemory and SharedMemoryManager. The docs feature a very exciting example:

import multiprocessing
from multiprocessing.managers import SharedMemoryManager

# Arbitrary operations on a shared list
def do_work(shared_list, start, stop):
    for idx in range(start, stop):
        shared_list[idx] = 1

# Example from the docs
with SharedMemoryManager() as smm:
    sl = smm.ShareableList(range(2000))
    # Divide the work among two processes, storing partial results in sl
    p1 = multiprocessing.Process(target=do_work, args=(sl, 0, 1000))
    p2 = multiprocessing.Process(target=do_work, args=(sl, 1000, 2000))
    p1.start()
    p2.start()  # A multiprocessing.Pool might be more efficient
    p1.join()
    p2.join()   # Wait for all work to complete in both processes
    total_result = sum(sl)  # Consolidate the partial results now in sl

# `do_work` set all values to 1 in parallel
print(f"Total of values in shared list: {total_result}")

Total of values in shared list: 2000

It remains to be seen if these improvements create a happy path that parallel ops focused libraries will adopt. In any case, it's exciting. Check out SharedMemory here.

Now let's look at some staic type checking stuff:

Typed Dictionaries (PEP 589)

Type hints don't cover nested dicts and dataclasses don't parse to JSON well enough, so now Python will support some static type-checking for dictionaries with a known set of keys. Consider the script below:

# scripts/valid_589.py
from typing import TypedDict

class Movie(TypedDict):
    name: str
    year: int

# Cannonical assignment of Movie
movie: Movie = {'name': 'Wally 2: Rise of the Garbage Bots', 'year': 2055}

mypy will pass this with no issues because the dictionary is a valid implementation of a type.

!mypy scripts/valid_589.py

[1m[32mSuccess: no issues found in 1 source file[m

Now consider the invalid script below -- it has values of the wrong types:

# scripts/invalid_values_589.py
from typing import TypedDict

class Movie(TypedDict):
    name: str
    year: int

def f(m: Movie):
    return m['year']

f({'year': 'wrong type', 'name': 12})

!mypy scripts/invalid_values_589.py

scripts/invalid_values_589.py:10: [1m[31merror:[m Incompatible types (expression has type [m[1m"str"[m, TypedDict item [m[1m"year"[m has type [m[1m"int"[m)[m
scripts/invalid_values_589.py:10: [1m[31merror:[m Incompatible types (expression has type [m[1m"int"[m, TypedDict item [m[1m"name"[m has type [m[1m"str"[m)[m
[1m[31mFound 2 errors in 1 file (checked 1 source file)[m

You can also check for missing values, invalid fields, or create TypedDicts that will accept missing values by using the total=False with the constructor.

PEP 589

Finality (PEP 591)

3.8 will also implement finality. We can prevent objects from being overriden or inherited. The @final decorator can be used with a class definition to prevent inheritence, and the Final type will prevent overrides. Here's two examples from the PEP:

# Example 1, inheriting a @final class
from typing import final

@final
class Base:
    ...

class Derived(Base):  # Error: Cannot inherit from final class "Base"
    ...

# Example 2, overriding an attribute
from typing import Final

class Window:
    BORDER_WIDTH: Final = 2.5
    ...

class ListView(Window):
    BORDER_WIDTH = 3  # Error: can't override a final attribute

Finality can be applied to methods, attributes and inheritence. Check out all the features in PEP 591

Literals (PEP 586)

The type hinting call signature def f(k: int): doesn't help us if the function expects ints in a range. open requires a mode: str argument from a specific set of strings. Literals to the rescue!

# scripts/problematic_586.py
def problematic_586(k: int):
    if k < 100:
        return k
    else:
        raise ValueError('Gotta be less than 100')
problematic_586(144)

!mypy scripts/problematic_586.py

[1m[32mSuccess: no issues found in 1 source file[m

Instead, we can pass a Literal to the type hinting for our argument k.

# scripts/valid_586.py
from typing import Literal

def valid_586(k: Literal[0, 1, 2, 99]):
    if k < 100:
        return k
    else:
        return float(k)
valid_586(43)

!mypy scripts/valid_586.py

scripts/valid_586.py:8: [1m[31merror:[m Argument 1 to [m[1m"valid_586"[m has incompatible type [m[1m"Literal[43]"[m; expected [m[1m"Union[Literal[0], Literal[1], Literal[2], Literal[99]]"[m[m
[1m[31mFound 1 error in 1 file (checked 1 source file)[m

There's a bit of nuance to using Literal so if you decide to explore further start with PEP-586.

And that's it! I'm going to hold off on writing about the CPython features because, frankly, I would prefer not to write about them until I have my arms more firmly around CPython generally and those features in particular.

Thanks for reading!

P.S. If you would like to play with any of the examples in this blog or test drive Python3.8 without installing it, I'm providing the source repository for this blog below. It includes a Dockerfile that spins up a Python 3.8 jupyterlab environment.

CharlesDLandau / python38_blog

A repo with the source material for a blog

Python38_Tour

This is the source repo for this blog, and it also includes a Dockerfile for simply launching python 3.8 without having to install it locally. So:

//build
docker build -t <img_tag> .

//run on *nix
docker run -p 8888:8888 -it --rm -v $(PWD):/code --name dev <img_tag>

//run on windows
docker run -p 8888:8888 -it --rm -v %CD%:/code --name dev <img_tag>

If all goes well you'll be prompted to copy a URL into your browser, which will point to your local port 8888 with a token to authorize access to the Jupyter instance.

View on GitHub

Data Munging with Common Pandas Operations

Charles Landau — Wed, 09 Oct 2019 13:10:37 +0000

Why?

Why am I writing the billionth article on Pandas data munging? Aren't the other articles enough? Maybe! If you already read a few of those and you feel like you know a lot about Pandas maybe your time would be better spent moving on to some of these materials:

Getting started

Quick note: throughout this article I'm going to refer to features and observations. When you start working with multidimensional problems it's an important distinction, but in this article a feature is the same as a spreadsheet column, and an observation is the same as a spreadsheet row.

import numpy as np
import pandas as pd

We're going to work with some random data, which you can construct directly. Pandas DataFrames are constructed from Pandas Series objects, which in turn use numpy objects and operations for many of their behaviors.

# Create a DataFrame with some randomized columns
df = pd.DataFrame({
    "str_feature": [np.random.choice(["dog", "cat", "snake"]) for _ in range(10000)],
    "int_feature": np.arange(10000)
})

The .head method can grab the top n rows, five by default.

df.head()

	str_feature	int_feature
0	snake	0
1	dog	1
2	cat	2
3	cat	3
4	dog	4

df.head(10)

	str_feature	int_feature
0	snake	0
1	dog	1
2	cat	2
3	cat	3
4	dog	4
5	snake	5
6	snake	6
7	dog	7
8	cat	8
9	dog	9

Of course in some circumstances we don't want to take the first 5 or 10. We can take a pseudo-random sample of the data using .sample. Note that .sample defaults to 1 observation.

df.sample(5)

	str_feature	int_feature
2476	cat	2476
2793	dog	2793
3120	snake	3120
9338	snake	9338
3593	dog	3593

We can get the underlying types by accessing the .dtypes attribute.

df.dtypes

str_feature    object
int_feature     int32
dtype: object

It should be stressed that object features are less memory efficient than int features. A discussion of the basic dtypes is included in the official getting started guide.

Descriptive statistics can be reported using .describe -- note that only numerical features will be summarized.

df.describe()

	int_feature
count	10000.00000
mean	4999.50000
std	2886.89568
min	0.00000
25%	2499.75000
50%	4999.50000
75%	7499.25000
max	9999.00000

The descriptive statistics in this report have their own methods, for example here's std:

df.std()

int_feature    2886.89568
dtype: float64

pandas-profiling is a new-ish package that creates much more detailed reports from a DataFrame. You can check it out here.

.size returns the number of observations multiplied by the number of features.

df.size

.info returns metadata about the DataFrame

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 2 columns):
str_feature    10000 non-null object
int_feature    10000 non-null int32
dtypes: int32(1), object(1)
memory usage: 78.2+ KB

Selection and Assignment

We tend to access data from the DataFrame using the following patterns:

df[condition]
df.iloc & df.loc
df.at & df.iat

# First pattern
df[df['str_feature'] == 'snake'].head()

	str_feature	int_feature
0	snake	0
5	snake	5
6	snake	6
14	snake	14
18	snake	18

# Second pattern
df.loc[:, "int_feature"]

0          0
1          1
2          2
3          3
4          4
        ... 
9995    9995
9996    9996
9997    9997
9998    9998
9999    9999
Name: int_feature, Length: 10000, dtype: int32

Note that here the lone : means to access everything on that axis, the same way my_list[:] tells a plain Python list to access everything.

.iloc behaves similarly but works on axes only.

df.iloc[0:5]

	str_feature	int_feature
0	snake	0
1	dog	1
2	cat	2
3	cat	3
4	dog	4

df.iloc[[0,5]]

	str_feature	int_feature
0	snake	0
5	snake	5

.at and .iat are used similarly to .loc and .iloc but they can't retrieve more than one observation.

df.at[0:5, 'str_feature'] will give a ValueError because of 0:5.

df.at[0] will give a TypeError because there is no col specified.

df.at[0, 'str_feature']

'snake'

There is a discussion on SO about how you can get some performance gains by accepting the limitations of .at.

There's one more method I want to cover for accessing data from our DataFrame:

df.int_feature.head()

0    0
1    1
2    2
3    3
4    4
Name: int_feature, dtype: int32

This works fine until:

Your feature is named my super feature and you try to access df.my super feature
You forget that the reverse operation is illegal.

That's right, you cannot do this:

df.new_feature = df[df['int_feature'] == 1]

c:\users\cdl\documents\personalproject\common_pandas_blog\.venv\lib\site-packages\ipykernel_launcher.py:1: UserWarning: Pandas doesn't allow columns to be created via a new attribute name - see https://pandas.pydata.org/pandas-docs/stable/indexing.html#attribute-access
  """Entry point for launching an IPython kernel.

Trying to will trigger a warning, at least in in newer versions of Pandas. We can confirm that the feature was not added to the DataFrame:

df.head()

	str_feature	int_feature
0	snake	0
1	dog	1
2	cat	2
3	cat	3
4	dog	4

Instead, let's add a new feature like so:

df['new_feature'] = df.loc[:, 'str_feature'] == 'snake'

Now we can see that the new feature was added properly:

df.head()

	str_feature	int_feature	new_feature
0	snake	0	True
1	dog	1	False
2	cat	2	False
3	cat	3	False
4	dog	4	False

Manipulation

Let's perform some common tasks to manipulate our data. Namely:

Filter values
Handle missing or invalid values
SQL-like join two DataFrames
SQL-like groupby

1. Filtering values

To filter values, we can simply overwrite our df variable like so:

# We're cat people now.
df = df[df.loc[:, 'str_feature'] == 'cat']

df.head()

	str_feature	int_feature	new_feature
2	cat	2	False
3	cat	3	False
8	cat	8	False
10	cat	10	False
11	cat	11	False

Since we've just filtered our DataFrame to use only use cat rows, our snakes-only new_feature will be full of False values and str_feature will be all cats. Let's filter our columns.

df.drop(["new_feature", "str_feature"], axis=1, inplace=True)
df.head()

	int_feature
2	2
3	3
8	8
10	10
11	11

As discussed here, the axis argument tells pandas to drop on the basis of the features axis, and the inplace argument makes this equivalent to df = df.drop(...).

As a final note, don't make the mistake of assuming that any of these methods will automatically free up memory. Instead, use tools like Python del and pandas I/O features like low_memory or memory_map.

2. Handle missing or invalid values

Another common data munging task is to handle cases where the data is invalid in some way. Let's make a new, messy DataFrame:

# Create a DataFrame with some randomized columns
df = pd.DataFrame({
    "str_feature": [np.random.choice(["dog", "cat", "snake"]) for _ in range(10000)],
    "int_feature": np.arange(10000),
    "messy_feature": [np.random.choice([1, None, np.nan, False, True, -1]) for _ in range(10000)]
})

This simplest thing we could do is clean our data by filtering all observations with np.nan, using .isna:

# Note the rarely used ~ negation operator below
clean_df = df[~df['messy_feature'].isna()]
clean_df.sample(10)

	str_feature	int_feature	messy_feature
4134	cat	4134	-1
2127	snake	2127	-1
7014	dog	7014	False
922	dog	922	False
5451	dog	5451	True
5059	cat	5059	-1
1289	snake	1289	False
5016	dog	5016	-1
9765	snake	9765	True
7094	cat	7094	1

As you can see, this method will clear just the NaN values. That's great, but what if we only want values that are truthy in Python?

df[df['messy_feature']] won't work, because it's a ValueError to filter using a column with NaN values. Instead, we have to filter the NaN, then check for truthiness. Just like with a plain Python conditional and statement, the conditions are checked left to right.

clean_df = df[~df['messy_feature'].isna() & df['messy_feature']]
clean_df.sample(10)

	str_feature	int_feature	messy_feature
1049	snake	1049	1
4769	snake	4769	-1
4924	cat	4924	-1
6907	snake	6907	-1
5914	dog	5914	True
6291	cat	6291	1
5872	dog	5872	1
1794	cat	1794	1
8987	cat	8987	1
3848	dog	3848	True

Imputing values is easy too:

clean_df = df.copy()
clean_df['messy_feature'].fillna(clean_df['str_feature'], inplace=True)
clean_df.sample(10)

	str_feature	int_feature	messy_feature
8063	snake	8063	snake
3901	cat	3901	cat
3702	dog	3702	-1
906	dog	906	1
8039	dog	8039	dog
2433	dog	2433	-1
4996	snake	4996	False
3015	snake	3015	snake
8307	cat	8307	1
1904	cat	1904	cat

And we can go beyond simple .fillna:

clean_df = df.copy()
clean_df['messy_feature'] = clean_df.apply(
    lambda row: row['messy_feature'] if row['messy_feature'] == -1 else row['str_feature'],
    axis=1
)
clean_df.sample(10)

	str_feature	int_feature	messy_feature
9217	snake	9217	snake
5018	snake	5018	snake
6383	dog	6383	dog
1297	cat	1297	-1
9701	dog	9701	dog
5192	cat	5192	cat
4018	dog	4018	dog
7619	snake	7619	snake
9890	dog	9890	dog
5102	snake	5102	snake

Just be careful with .apply as it has the potential to significantly degrade performance. See: enhancing performance in the Pandas docs.

3. SQL-like joins

I would definitely recommend that you learn about SQL joins and then just map that knowledge to Pandas joins. The arguments like on, how, left, and right all make sense if you just keep in mind how the joins would work in SQL.

merged_df = df.copy().join(df.copy(), how='left', lsuffix="_l", rsuffix="_r")
merged_df

	str_feature_l	int_feature_l	messy_feature_l	str_feature_r	int_feature_r	messy_feature_r
0	snake	0	False	snake	0	False
1	cat	1	NaN	cat	1	NaN
2	snake	2	False	snake	2	False
3	dog	3	False	dog	3	False
4	snake	4	None	snake	4	None
...	...	...	...	...	...	...
9995	snake	9995	False	snake	9995	False
9996	dog	9996	True	dog	9996	True
9997	snake	9997	NaN	snake	9997	NaN
9998	dog	9998	NaN	dog	9998	NaN
9999	dog	9999	1	dog	9999	1

10000 rows × 6 columns

4. SQL-like groupby

Like the SQL JOIN advice, I strongly recommend learning about SQL GROUP BY and letting that be the basis of your Pandas .groupby knowledge. Here's a simple GROUP BY SUM in Pandas

df.groupby('str_feature').sum()

	int_feature
str_feature
cat	16476822
dog	16729175
snake	16789003

Note that .groupby must be called by a method that aggregates the grouping. Let's see what happens if we neglect that part...

df.groupby('str_feature')

<pandas.core.groupby.generic.DataFrameGroupBy object at 0x0A6F91D0>

.groupby is particularly powerful when combined with the .apply method...but my earlier warning about the performance of .apply still holds.

# Simple sum/size calculation
df.groupby(['str_feature']).apply(lambda x: x.sum()/x.size)

	int_feature	messy_feature
str_feature
cat	2485.191855	0.088688
dog	2509.627213	0.086709
snake	2504.326223	0.089499

Moving Windows

Often we want to calculate rolling values (like rolling sums):

df['rolling'] = df.rolling(3).sum()
df.head(10)

	str_feature	int_feature	messy_feature	rolling
0	snake	0	False	NaN
1	cat	1	NaN	NaN
2	snake	2	False	3.0
3	dog	3	False	6.0
4	snake	4	None	9.0
5	snake	5	NaN	12.0
6	snake	6	None	15.0
7	cat	7	-1	18.0
8	cat	8	NaN	21.0
9	snake	9	NaN	24.0

Indexing by Time

A related task is to use datetime, which we accomplish using pd.to_datetime. There are helper methods like .astype for casting to datetime types, as well as .set_index if we want to use a datetime column as our index.

import time
df['Datetime'] = pd.to_datetime([time.time() - (86400*x) for x in range(df.shape[0])], unit='s')

# Round to nearest day
# see https://stackoverflow.com/a/13788301/10553976
df['Datetime'] = np.round(df['Datetime'].astype(np.int64), -9).astype('datetime64[ns]')

# Set as index
df = df.set_index('Datetime')

I/O

One of the most attractive features of Pandas is the robust I/O capabilities. The whole list is here but let's take a look at two important ones, CSV and SQL.

CSV

Writing a CSV in Pandas couldn't be easier:

df.to_csv('./data.csv')

Reading it back in is just as simple:

df = pd.read_csv('./data.csv')

But wait: let's look at our df now...

df.head()

	Datetime	str_feature	int_feature	messy_feature	rolling
0	2019-10-08 21:51:50	snake	0	False	NaN
1	2019-10-07 21:51:50	cat	1	NaN	NaN
2	2019-10-06 21:51:50	snake	2	False	3.0
3	2019-10-05 21:51:50	dog	3	False	6.0
4	2019-10-04 21:51:50	snake	4	NaN	9.0

We can't infer that any particular column is the index, so .read_csv created a DataFrame with a simple index instead of using the DatetimeIndex we set before writing to a file. Of course we could always use .set_index after the fact, but in resource constrained scenarios we want control of how large the created DataFrame will be. Searching for a solution, we come to the .read_csv documentation and the power of this tool immediately becomes apparent:

Read CSVs from URL (!!!!!), path, or file-like
Specify the exact window in a CSV file to read from with kwargs like header, index_col, usecols, nrows, skiprows, and skipfooter
Manage NaN with kwargs like na_filter and na_values
sep for handling tsv and other delimiter situations
Kwargs to help save memory like low_memory, memory_map, and dtype

Armed with tons of new knowledge about how to read csv files with Pandas, we can confidently read our data.csv file with the DatetimeIndex specified.

df = pd.read_csv('./data.csv', index_col='Datetime')
df.head()

	str_feature	int_feature	messy_feature	rolling
Datetime
2019-10-08 21:51:50	snake	0	False	NaN
2019-10-07 21:51:50	cat	1	NaN	NaN
2019-10-06 21:51:50	snake	2	False	3.0
2019-10-05 21:51:50	dog	3	False	6.0
2019-10-04 21:51:50	snake	4	NaN	9.0

Just the way it originally was! Let's clean that file up:

import os
os.remove('./data.csv')

Note: if you aren't using Pandas features for data manipulation, consider using the stdlib csv library for I/O instead of Pandas. Pandas objects create some overhead in memory, and Pandas itself is a nontrivial dependency.

SQL

If you want to read and write data that lives in a SQL database, Pandas uses the pd.read_sql and .to_sql methods to interface with databases. In this example we'll use the stdlib sqlite3 dbAPI, but Pandas can integrate with any kind of DB that is supported by sqlalchemy. Also note that we're going to use a try...finally block to enforce .close always being called.

import sqlite3

# DB operations
try:
    # Connect to db
    conn = sqlite3.connect('data.db')
    # Write dataframe to table 'df'
    df.to_sql('df', conn, if_exists='replace')

    # Read a SELECT * query of the same table, specifying index column
    sqldf = pd.read_sql('SELECT * FROM df', conn, index_col='Datetime')

# Ensure the connection closes
finally:
    conn.close()

# Cleanup file
os.remove('data.db')

sqldf.head()

	str_feature	int_feature	messy_feature	rolling
Datetime
2019-10-08 21:51:50	snake	0	False	NaN
2019-10-07 21:51:50	cat	1	None	NaN
2019-10-06 21:51:50	snake	2	False	3.0
2019-10-05 21:51:50	dog	3	False	6.0
2019-10-04 21:51:50	snake	4	None	9.0

Once again our data has made a round trip from memory to disk and back again. Well done Pandas and well done to you, dear reader!

Wrapping Up

In this article you've learned how Pandas implements some basic data munging tasks, specifically:

Extracting metadata and descriptive statistics
Filtering, imputing, joining and grouping data
Windowed functions (like rolling sum)
Using Datetime data and indexing by time
Reading and writing data to files and databases

Happy wrangling!

Image by Cimberley on Pixabay

React useContext and Pre-Initialized Providers

Charles Landau — Thu, 03 Oct 2019 19:12:34 +0000

On a recent project I started using the React Context API instead of Redux, and I like it. Here's how you can get started quickly.

Setup a project

I like Next for a variety of reasons, particularly the router. Let's create a new project like so:

git init context_demo && cd context_demo
npx create-next-app
npm run dev

localhost:3000 will now display the template app. Let's "clear the table":

// pages/index.js
import Head from '../components/head';

export default () => (
  <div>
    <Head title="Home" />
  </div>
);

The view should go blank now.

With that out of the way, let's create a context in a new folder contexts:

// contexts/contextcontroller.js
import { createContext } from "react";

// Dummy data
let ctxObj = {
  lastUpdated: new Date(),
  nestings: { nest: {nest: {nest: null}}},
  foo: 'bar',
  arr: [1, 2.0, 'baz']
};

const ControlledContext = createContext({});

// A provider with all the methods and states set up for you
class InitializedProvider extends React.Component {
  constructor(props) {
    super(props);
    this.pushFoo = () => {
        this.setState({
                arr: [...this.state.arr, 'foo'],
                lastUpdated: new Date()
            });
      };
    this.state = {
        ...ctxObj,
        pushFoo:this.pushFoo
        };

  };

  render() {
    return (
      <ControlledContext.Provider value={this.state}>
        {this.props.children}
      </ControlledContext.Provider>
    );
  };
};

export { ControlledContext, InitializedProvider };

As you can see all this does is pack a method into its state and pass the state to a context provider. Everything is all in one place, but we could be importing functions from all over the place if we wanted to.

Let's build some consumers.

// components/buttoneer.js
import React, { useContext } from 'react';
import { ControlledContext } from '../contexts/contextcontroller';


function Buttoneer(){
    let value = useContext(ControlledContext);
    return(
        <div style={{height: 100, width:100}}>
            <button onClick={value.pushFoo}>Click me!</button>
        </div>
    );
};

export default Buttoneer

This one just consumes our pushFoo method from the context. Now let's imitate significant nesting in the component tree.

// components/nestingdoll.js
import React from 'react';


function NestingDoll(props){
    return(
        <div style={{border: '1px solid black', padding: '20px'}}>
            {props.children}
        </div>
    );
};

export default NestingDoll

As you can see, this one just draws a box around its children. Now lets make a component that consumes the array from our context.

// components/listview.js
import React, { useContext } from 'react';
import { ControlledContext } from '../contexts/contextcontroller';


function ListView(){
    let value = useContext(ControlledContext);
    return(
        <div>
        {
        value.arr.map((item, idx)=>{
        return (
            <div 
            key={`${idx}-listitem`}
            style={{border: '1px solid black', padding: '20px'}}>
            <p>
                {`${item}`}
            </p>
            </div>
                )
            })
        }
        </div>
    );
};

export default ListView

With these components let's edit our index to render our Buttoneer in one branch of the component tree, and then nest Listview under several layers of NestingDoll

// pages/index.js
import Head from '../components/head';
import { InitializedProvider } from '../contexts/contextcontroller';
import Buttoneer from '../components/buttoneer';
import NestingDoll from '../components/nestingdoll';
import ListView from '../components/listview';

export default () => (
  <div>
    <Head title="Home" />
    <InitializedProvider>
      <Buttoneer />
      <NestingDoll>
        <NestingDoll>
          <NestingDoll>
            <ListView />
          </NestingDoll>
        </NestingDoll>
      </NestingDoll>
    </InitializedProvider>
  </div>
);

The result should look like it does in this demo: controlled-context-demo.charlesdlandau.net

And you can see the repo here.

I think that this pattern improves the organization of projects because global app state is all in one place. I know that there are some drawbacks to the Contexts API (e.g. rerenders) but it works for me.

Thanks for reading! I'd be interested to hear your thoughts. Is this an anti-pattern? Do you dislike the Context API and think I should just use Redux?

🤖➯🌩 (Machine Learning In the Cloud)

Charles Landau — Tue, 01 Oct 2019 14:17:14 +0000

I've been preparing a talk for my local ML user group and as a result I've had to synthesize a lot of ideas. In the talk I introduce a wide variety of "Machine Learning as a Service (MLaaS)" services and talk about how they fit in your toolkit. It's supposed to be a 45 minute talk, so the deck is pretty long... much longer than you would want for a blog post. In this post I'm going to:

Summarize the heck outta this talk.
Share the slides at the end.

Note: I am NOT affiliated with ANY product mentioned ANYWHERE in this post.

MLaaS...

...because what the world really needs is another aaS right?

Machine Learning as a Service is a tautology -- it's a label for any service that delivers machine learning. These services offer a variety of advantages to businesses and data scientists, such as:

Near-State of the Art performance in some cases, without the up-front cost (e.g. training on many GPUs in parallel for many hours). Big Four firms (and their cloud plays) obviously have advantages when it comes to this space, especially in terms of their access to data. (See: Google Natural Language)
“No Data Science Knowledge Required” and “AutoML” services that enable rapid prototyping and other optimizations. (See: H20 Driverless AI)
Platforms that enhance time-to-market, optimize cost, or offer simplified integrations with supporting services. (See: Azure ML Studio)
APIs that map endpoints to ML behaviors like training or prediction. (See: wit.ai)
Devices that embed ML capabilities (See: Hover2)

Worth it?

There are several considerations to take into account when you're evaluating MLaaS versus "rolling your own" solution:

Cost: MLaaS can seem to offer a way to mitigate the cost of building a Data Science team. Data Scientists, Big Data Engineers, and ML Engineers attract great compensation and benefits, and building a competitive recruitment process for these teams can also be difficult.

Related note: you may avoid a lot of upfront costs with some MLaaS offerings, but per-call or per-use costs tend to bake that in.

Ownership: In many cases taking an MLaaS path means that you don’t fully own and control your models, or that you’re locked in to a platform. Rolling your own can mean having a greater degree of control. On the other hand, a proprietary model from a Big Four shop may be exactly what your organization wants.

Rapid Prototyping: MLaaS offerings can enable a low-cost rapid prototyping approach, especially for organizations that do not have a mature rapid prototyping practice (a skunkworks). In this way MLaaS can extend rather than displace data science capabilities.

Culture: Many organizations seek to build not only a data science team but a "culture of innovation" of which their data science practice is a part. Having a data science team creates an in-house nexus that can advocate for itself and identify lines of business that may not be apparent to other teams.

It's also worth noting that all the typical limitations of each delivery model apply:

Typical limitations of SaaS: you only get their features when they are up (SLAs), supported (LTS), you are budgeted for them ($$$), et cetera

Typical limitations of in-house production: growing pains, recruitment costs, cruft...

Service Categories

To give a better overview of the landscape I pulled out a few categories of services that are out there for you to use. These are non-exhaustive and overlapping categories that map to common use cases.

1. NLP Services

NLP is an area where state of the art results are expensive, use cases may lend themselves to generic out-of-the-box solutions, and large enterprises often have better access to data than market entrants. Some services in this category include:

2. AutoML

AutoML and "No Data Science Required" services are developed using techniques like transfer learning, heuristics, Bayesian hyperparameter tuning, platform optimizations, and more. They're probably doing proprietary stuff that we may never even hear about. Services in this category include:

Services in the AutoML space also have begun trying to distinguish themselves using "explainable AI" features. These are accomplished with techniques like LIME, which perturb the inputs and use the perturbations to infer how much the model values that input towards the final prediction.

Image from Marco Tulio Ribeiro and Pixabay

3. ML Data Services

There's a wide world of "managed Apache" out there, for example Spark as a Service products are available at all major cloud providers. One place that's looking pretty interesting right now is data labeling, with services like:

The idea behind these services is to use human labelers frequently until a model can begin to infer the labels. As the model gets better at labeling the data, only the observations where the model has low confidence get referred to human labelers.

4. ML as an API

This might be the category with the greatest overlap to other categories, but it's the easiest to grasp. Any API that serves predictions or training endpoints fits in this category. Some notables include:

5. Embedded AI

I'm definitely stretching the definition of an "embedded system" when I use this category, but the basic idea is just devices that ship with built-in ML features, or that plug in to your devices to imbue them with ML magic. Such products include:

Bonus: JaaS

Jupyter as a Service products like Google Colab and Datalore have been around for a long time now, so if you haven't checked them out yet, then now's the time. They're very mature web applications that act like a REPL on steriods. They don't replace an IDE, but have some great features and can be a great option for when you don't actually need a full-fledged IDE. Tools like JupyterHub that host and provide compute instances to Jupyter web clients can create a unified experience with reproducible experiments available alongside their documentation.

Slides

If you'd like to see the whole deck, complete with notes, here you go!

I'll be giving this talk for the first time in a couple weeks so I reserve the right to have errors in the slides. If you have any feedback I'd love it if you share that with me.

Thanks for reading!

🤖➡🌜 (Teach Bots to Land on the Moon)

Charles Landau — Tue, 24 Sep 2019 02:30:51 +0000

Tinker with this notebook in Google Colab

GitHub

In the past few episodes we've gone over design princples for reinforcement learning, introduced reinforcement learning with OpenAI Gym, and trained a deep learning model to navigate an OpenAI Gym environment. To accomplish this, we leveraged the agent-environment relationship:

The agent takes actions in the environment. These can be random, or guided by heuristics, or by a model, or some combination. It recieves data about the environment, like a picture of the environment or a string, and then it takes an action. Afterwards, it recieves a reward from the environment, which amount to positive or negative "reinforcement" of the selected behavior.
The environment tracks the state, possible actions, rewards and reward function, and optionally renders the environment to a display. It receives actions from the agent, and it can be configured to persist data (e.g. a video of the agent performing in the environment.)

Better Agents

In previous episodes we also implemented some janky, simplistic reinforcement learning. Without convolutional layers or a policy engine the network could only learn simplistic strategies (like turn left).

To solve this without reinventing the wheel we can use libraries like Tensorflow Agents, or OpenAI Baselines which have some battle-tested implementations of the most successful agents. I've worked with a few of these libraries and I mostly came to the same conclusions as this excellent article by Thomas Simonini. Namely, I like Stable Baselines, a fork of the OpenAI Baselines library.

Getting Ready

Setting up Google Colab with Stable Baselines and its dependencies, and figuring out how to get gifs working, took me longer than I would have liked. You do it with the following lines:

# see: colab.research.google.com/drive/1GLlB53gvZaUyqMYv8GmZQJmshRUzV_tg
!apt-get -qq -y install libcusparse8.0 libnvrtc8.0 libnvtoolsext1 > /dev/null
!ln -snf /usr/lib/x86_64-linux-gnu/libnvrtc-builtins.so.8.0 /usr/lib/x86_64-linux-gnu/libnvrtc-builtins.so
!apt-get -qq -y install xvfb freeglut3-dev ffmpeg> /dev/null

!pip install stable-baselines==2.7.0

# box2d fork until SWIG dependency is replaced
!pip install box2d-py

# pyglet later versions break gym apparently
!pip install pyglet==1.3.2

# Set up display; otherwise rendering will fail for
# classic control type envs, apparently
import os
import Box2D
import pyglet
os.system("Xvfb :1 -screen 0 1024x768x24 &")
os.environ['DISPLAY'] = ':1'


# See https://github.com/ipython/ipython/issues/10045#issuecomment-522697219
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
from IPython import display

With that out of the way, you can implement a high-quality Advantage Actor Critic (A2C) model in just two lines. Simple Baselines is great because it has intuitive methods for training and predicting. Even better, it stores our enviornment as a model.env member, so all we need for typical workflows is a model instance.

Let's train -- The following takes a fair bit of time to run:

from stable_baselines import A2C
import time
start = time.time()

# Train a multilayer perceptron in the lunar lander.
model = A2C('MlpPolicy', "LunarLander-v2").learn(900000)

# Mine finished in about 10 minutes.
print(f"Finished in {time.time()-start}s")

An actor and a critic walk into a bar.

While that runs, let's talk about the Actor-Critic paradigm. The problem Actor-Critic solves involves none other than our reward function. In many cases the rewards only happen at the end of an "episode" AKA a "game." Consider a reward function for tic-tac-toe:

Return 0 during play
Return 0 if game ends in a tie
Return -1.0 if game ends in a loss for the agent
Return 1.0 if game ends in a win for the agent

As you can see, our agent is taking several steps each episode which recieve no feedback at all from the reward function, slowing training and making it difficult to give fine-grained feedback to our agent about specific actions. To solve this, Actor-Critic actually implements two models:

A Critic model that tries to predict reward values that a given action will earn, given an observation about the environment. The table of predicted reward values for action-observation pairs is sometimes called a "Q table."
An Actor model that takes observations from the environment, but not rewards. It learns a strategy for playing in the environment, AKA a "policy." Instead of optimizing on the environment's rewards directly, this model optimizes on feedback from the critic model. When we do predictions, just the Actor model needs to be called.

We can easily imagine our Actor model in the lunar lander module, while the Critic sits in the command center observing the lander's actions.

Each time an action is taken the Critic consults a table of actions, angles, speeds, and so on. Seeing whether the action will contribute to a successful landing, the Critic radios over to the Actor with feedback about the action, and the Actor learns to be a better pilot from the feedback. Over the course of thousands of landings, the Critic updates the table frequently based on crashes, successes, and surprises. Some days, the Actor flies with the radio turned off, but it doesn't learn anything.

Record a GIF of the landing

By now your model might be done training. Per the docs, here's how we record a gif of our model in action:

import imageio
import numpy as np

# Number of images to capture
n_images = 1200

images = []

# init a new episode
obs = model.env.reset()
# init the img var with the starting state of the env
img = model.env.render(mode='rgb_array')

for i in range(n_images):
  # At each step, append an image to list
  images.append(img)

  # Advance a step and render a new image
  action, _ = model.predict(obs)
  obs, _, _ ,_ = model.env.step(action)
  img = model.env.render(mode='rgb_array')

imageio.mimwrite('./lander.gif',
                [np.array(img) for i, img in enumerate(images) if i%2 == 0],
                fps=29)

If you're successful then lander.gif should look something like this:

Displaying gifs in Jupyter is a little counterintuitive, here's how I do it:

from pathlib import Path
gifPath = Path("lander.gif")
# Display GIF in Jupyter, CoLab, IPython
with open(gifPath,'rb') as f:
    display.Image(data=f.read(), format='png')

Wrapping Up...

In this episode you:

Learned about Actor-Critic
Learned about Simple Baselines
Saw how to setup giffing the environments in a notebook
Landed on the moon!

Way to go!

In future episodes we'll take a closer look at how RL can create value for users, businesses, and robot overlords. Stay tuned and thanks for reading.