DEV Community: ITHAKA

Choose Boring Releases

Dane Hillard — Thu, 26 Sep 2024 16:52:04 +0000

Academic research increasingly relies on diverse content types, including gray literature and primary source materials, alongside traditional peer-reviewed works. ITHAKA supports this evolving research landscape through initiatives that foster cross-content connections on JSTOR, our digital library that supports research, teaching, and learning. These include infrastructure services that enable institutions to make their digital archives and special collections discoverable on the platform, and a years-long effort to integrate Artstor — a vast collection of images and multimedia for educational use — onto the JSTOR platform.

Albrecht Dürer (German, Nuremberg 1471–1528 Nuremberg). “Alberti Dvreri Pictoris et Architecti Praestantissimi De Vrbibvs…” 1535. Illustrated book, 78 pp.; H: 13 3/4 in. (35 cm). The Metropolitan Museum of Art. https://jstor.org/stable/community.34718827. Just one of the millions of items now available on JSTOR.

On August 1, 2024, the Artstor migration to the JSTOR platform culminated in the need to smoothly guide users from the Artstor Image Workspace (AIW) at library.artstor.org to appropriate counterparts on www.jstor.org. The amount of work this required was daunting; the AIW platform was as large and complex as JSTOR, and covered many different product areas with ownership spread across several different teams. To achieve this transition without drawing focus away from other objectives, we had to adopt a strategic approach.

The challenge

Our two main constituents for this transition were our users — primarily art history researchers and faculty — and search engine web crawlers. Like JSTOR users, many Artstor users start their searches outside of our platforms and need to be able to find things no matter what platform changes we make.

The first piece of the challenge was technical: Client-side routing was used extensively on the AIW platform. That routing wasn’t built with web crawlers in mind, so many modern web crawlers couldn’t handle it very well. Plus, these client-side routes were generally invisible to our servers because they used hash-based routing rather than the history API. Some pages did support server-side rendering so web crawlers could index them, which gave us an additional scenario to support.

The second challenge was human: Domain knowledge about the details of all this routing and the user value of various pages was spread throughout the organization, and the people with the domain knowledge didn’t always have the technical knowledge to deal with the redirects.

Finally, there was the sheer scope of the work to be done. AIW itself was a huge platform, but on top of that we also had to redirect supporting areas like the marketing and support sites. Very early on, we had to set some parameters to rein in the scope:

We wouldn’t guarantee or aim for 100% of traffic to be redirected. Some pages and requests simply couldn’t be redirected anywhere reasonable, and redirecting to generic informational pages can lead to search engine ranking penalties. More importantly for our user focus, it would be disorienting to click a bookmark or a search engine result and get redirected to something irrelevant.
We would create a central, extensible service designed for loose coupling of the URL patterns to match and the behavior when such a match occurred. This would reduce unnecessary coupling and dependencies between teams as well as the context those teams had to hold onto during the migration project.
We would enable teams to implement redirects for the broadest-used page types (such as individual items), but would equally enable implementing redirects for the long tail of irregular page types. This would create a familiar implementation pattern for teams that they wouldn’t have to abandon for more exotic use cases.
We would enable visibility into the behavior of the system as early as possible, so that we could observe the trend over time. This fostered an experimentation mindset: We could develop hypotheses about how the traffic would be handled, implement a change, and quickly confirm or disconfirm the desired effect after deploying the new change. This also gave us opportunities to leverage parts of our deployment platform we hadn’t exercised much previously, adding some greenfield work to an otherwise brownfield kind of activity.

So instead of perfection, we defined our scope as removing as much risk as possible and building in as much confidence as we could ahead of our launch date. Our goal was to coast through the finish line instead of scrambling.

The approach

So how did we do that? A few key tactics showed up for us and made things possible throughout this project.

Practice product-minded engineering

We approached the project using product-minded engineering principles, which leverage user-focused agile techniques and strong cross-disciplinary and interdisciplinary skills to identify problems worth solving. By considering both user needs and architectural needs together, we came away with a more globally maximum solution without oversimplifying or overengineering.

Use traffic shadowing

Traffic shadowing allows inbound requests to be sent to two or more destinations, each able to take some action. Although one system is still responsible for sending a response back to the user, other systems can also make observations about or take action on requests. Our Capstan platform made this easy to try, and it worked very well. We continued serving AIW pages to users while also sending requests to our new redirect service, so we could see how it would behave and perform.

Measure early and often

We built metrics into the redirect service so that as we started implementing and deploying redirect behaviors we could assess the total traffic volume and understand what portion of that traffic was being handled by a known redirect. That was a huge confidence builder, as we could actually demonstrate not only that the service should redirect the implemented URLs once turned on, but that it was in fact doing so — with the shadow traffic.

Use feature flags

Where we couldn’t implement traffic shadowing for some supporting pages, we used feature flagging. With feature flags, we can ship two different behaviors and then allow staff and automated tests to toggle those behaviors on or off before exposing that new behavior to users. Even under normal circumstances we have many teams developing across many areas of the platform, and a feature flagging strategy brings speed and safety by ensuring we don’t have to perform a “big bang” release at the very end of a project.

Think in transition architectures

In effect, all of the redirect architecture we built to support this migration was a transition architecture. But we also built some smaller transition systems to move the control of traffic from external vendors into our own platform, making it easier to flip a switch on release day instead of having to handle several moving parts in a row.

Easing the transition architecture burden made a big difference on several of our support sites. For example, the DNS records for these sites were initially pointed directly at third-party vendors; before we could control the traffic, the traffic would have to come to us. If we waited until release day for that to happen, the DNS change could take longer than expected to propagate, or it could result in unforeseen consequences, or the logic we implemented for it could be wrong. Instead, we designed these constraints away. We worked to bring the DNS under our control earlier, and although it initially continued to point to the third-party vendors, we could now make immediate changes to its behavior.

Create a loosely coupled design

Because we had to collect and implement so much widely scattered domain knowledge into these redirects, we had to build a system that made it easy to contribute and hard to cross wires or step on other people’s toes. The system we created allowed teams to come in with nothing more than a URL pattern to match on AIW and a destination URL to send the traffic to. If necessary, they could also contact downstream services for data to help them decide where to redirect that traffic.

Basically, we decided to meet teams where they are, with their domain knowledge, so they didn’t have to think too much about the technical aspects of how redirection works. This also allowed the team working on the redirect service to do that work without having to worry too much about the domain knowledge. This separated responsibility in a nice way and ensured better outcomes for us and our users.

Have regular, valuable trade-off discussions

Finally, we engaged throughout this whole project in high-quality trade-off discussions. We talked about the level of effort a particular change required compared to the expected user impact. We talked about specific pages and how to handle them. And because we had such a rich cross-section of technical and domain knowledge in those discussions, we made better decisions about how to simplify things or adjust to reduce our overall risk and costs.

This is really a reprise of product-minded engineering, but this highlights one of its most impactful outcomes — giving us context about what’s possible, what gives us the best ROI, and how we might deliver better outcomes for us and users.

The result

We worked hard leading up to our launch date, August 1. But release day itself was one of the most boring of all time. We did about 30 minutes of real work to flip on some of those feature flags and deploy a couple of applications, all of which we were highly confident in because of our previous metrics and testing. In that short time we started redirecting over 95% of all AIW traffic to known locations on JSTOR and its supporting sites. The remaining requests were for things we had intentionally decided not to handle.

We only shipped one bug that we know of, which we spotted quickly. Because of our system design we were able to revert immediately to the old system while we worked on a short-term fix, and within a couple of weeks put a long-term fix in place so we could sunset the old system.

Meanwhile, we’re listening for user experience feedback in case anyone is confused or notices broken redirects, and we’re watching how our search engine crawling and ranking responds. As Google and Bing and others recrawl our content, they’ll start sending people directly to JSTOR. NISO recommends leaving these redirects in place for at least a year, so we’ve committed to that. Next year we’ll evaluate whether this architecture has run its course and act accordingly.

The takeaways

While the technical tools and architecture we used were obviously important, adopting a loosely coupled, extensible design was key to our success. By bringing solutions to teams that allowed them to focus on their domain knowledge we played to everyone’s strengths. It also ensured teams weren’t colliding with each other too often. This approach helped us focus and synthesize everyone’s strengths. It may not be possible in all cases, but for this project it was one of our best and most productive decisions.

By thinking in transition architectures and building key visibility into the system at the outset, we were able to create significant confidence in our progress and readiness for release day, making the finale just a blip on the radar. This raised an unexpected consequence — it would have been easy to say, “Look, we did the thing,” and part ways on release day, because it was so uneventful. Choose boring releases, but be sure to keep the afterparty exciting.

Interested in learning more about working at ITHAKA? Contact recruiting to learn more about ITHAKA tech jobs.

Our Open Source Software in the Wild

Dane Hillard — Fri, 21 Jun 2019 17:20:42 +0000

ITHAKA has a mission to make knowledge widely available, accessible, and sustainable. This mission runs parallel to the open source approach to software development. In making software, systems, and methodologies public, organizations enable others to build on their successes as well as critically examine these artifacts for bugs or security flaws.

The collaborative nature that open source strives toward acts as a tide that raises all ships; the community can produce improvements that benefit everyone. These software changes are tracked over time and the historical record that arises also acts as an audit trail for reproducibility or as a catalyst for someone hoping to learn how something is done.

Our front-end developers that work on JSTOR consume data from a myriad of back-end systems, from those we manage in-house for authentication and search to third-party APIs for fetching data from Wikipedia. As the list of services we interact with grew over the last few years, we found it difficult to manage the services, let alone all the endpoints we communicate with in each of them. We built a Python project, ithaka/apiron, in order to ease this burden. Sometime soon after that, we decided that apiron might be useful enough to others to warrant publication and got to work on all the legal and marketing threads involved.

Since open sourcing apiron, it’s seen moderate but intriguing interest from the Python community. Jeff Triplett, a Director of the Python Software Foundation, was the first third-party contributor to the apiron project. Jeff lives in Lawrence, KS, the birthplace of the Django web framework that we make heavy use of on JSTOR. He organizes the DjangoCon US conference as the president and co-founder of the Django Events Foundation North America.

Another third-party contributor, who we know only by the cryptic handle teffalump, used apiron to build a Python package for communicating with servers managing DICOM, a standard for working with medical imaging records. Excitingly, their work went on to be used by folks in the Brown University Behavioral and Neuroimaging team.

One beautiful consequence of the freedom of open source software is that you never quite know where it will end up. We built apiron to improve how we talk to a search service, and already it’s spread to use cases we couldn’t have imagined at the outset if we’d tried. Exploring these use cases as opportunities to learn will ultimately help us improve our software and ourselves.

As ITHAKA publishes software that generates reach and impact, those we engage with may develop an interest in working with us more directly, perhaps as collaborators or even as employees It’s also a new way to pursue our mission to expand access to knowledge (in all its forms).

If you’re curious about the steps involved in open sourcing projects, my experience with it, how to effectively manage an open source project, or are thinking about using apiron, please leave a comment or reach out to me on Twitter at @easyaspython!

Blazing fast Python

Dane Hillard — Wed, 19 Dec 2018 15:01:01 +0000

Profiling Python applications using Pyflame

This post originally appeared on Build Smarter.

Perhaps you’ve faced the fortunate challenge of scaling up a Python application to accommodate a steadily increasing user base. Though most cloud hosting providers make it easier than ever to throw more hardware at a problem, there comes a point when the cost outweighs convenience.

Around the time scaling horizontally starts looking less attractive, developers turn to performance tuning to make applications more efficient. In the Python community there are a number of tools to help in this arena; from the built-in timeit module to profiling tools like cProfile, there are quick ways to test the difference between a particular line of code and any of its alternatives.

Although profiling tools help you see important information about which calls in your application are time consuming, it’s difficult to exercise an application during local development the same way your users exercise it in real life. The solution to bridging this gap? Profile in production!

Pyflame

Pyflame is a profiling tool for Python applications that makes use of the ptrace(2) system call on Linux. ptrace allows processes to observe (and in some use cases, manipulate) another process’ memory. Pyflame ultimately uses ptrace to aggregate statistics about a running Python process’ stack in a format that’s helpful for visualizing where that process spends the majority of its time. Since Pyflame runs as a separate process and inspects existing stack data, its overhead is small compared to solutions that run as part of the process itself. This also means you can be sure the profile analysis you end up with faithfully represents your application’s behavior and isn’t greatly skewed by the profiling work.

After running Pyflame and a bit of post-processing you’ll be able to generate visualizations like this:

The y-axis is the call stack; the x-axis represents the proportion of CPU time spent in a particular call

But how?

As ptrace is specific to Linux, Pyflame can only target applications running in a Linux environment. Pyflame may also need some help finding your Python; it defaults to trying to link against Python 2.7 if it isn’t sure what Python to use. Given these caveats, installation takes just a few steps that will produce a pyflame executable:

$ git clone [https://github.com/uber/pyflame](https://github.com/uber/pyflame)
$ cd pyflame/
$ ./autogen.sh
$ ./configure
$ make

The pyflame executable can be found in the src directory after the build.

Now that you’ve built the tool, it should be ready to analyze your Python application. To do so, start by snagging the process ID (PID) of the application you’re interested in. If you’re running a Django application, for instance, this would probably be the PID of one of the WSGI workers running the Django app. After you’ve got a PID handy, you’ll run pyflame with options for:

How long to sample the process, in seconds
How often to sample the stack of the process, in seconds
Removing the idle time of the application from the data to simplify the resulting visualization
The name of the output file
The PID of the process

This will look something like

$ ./src/pyflame -s 300 -r 0.01 -x -o my-app-20181212.samp -p 9876

Pyflame will sample the process with PID 9876 every ten milliseconds for five minutes, producing the my-app-20181212.samp data file. Trying to read this file yourself won’t make a lot of sense, but fortunately it’s in just the right format for FlameGraph.

FlameGraph

FlameGraph is an interactive visualization tool for exploring CPU, memory, and other trace data. Since you’ve got your trace data, you can run it through FlameGraph in a couple of additional steps:

$ git clone [https://github.com/brendangregg/FlameGraph](https://github.com/brendangregg/FlameGraph)
$ cd FlameGraph/
$ cat /path/to/my-app-20181212.samp | ./flamegraph.pl > my-app.svg

Open up the my-app.svg file in a browser and you’ll see something resembling the screenshot from earlier. You can hover over any segment to see what call it is and how much of the CPU time it represents. Clicking on a segment zooms into it so that you only see it and its children. Most usefully, you can perform a regex-capable search using the “Search” button in the top right. Searching for a particular pattern will highlight any matches:

Pink segments match the search query

By exploring a flame graph, you can more easily find slow and common calls in your application that aren’t always obvious from poring over loads of raw text. And by identifying these calls, you can start to understand how you might effectively pare things down into a leaner application.

The pink segments above represent instances of a particular method call within our application; this call isn’t particularly expensive, but occurs many times on nearly every request. We realized that while we had code that seemed like it was caching these calls, the TTL was only set to one second. By updating the TTL to ten seconds, we were able to reduce how often this call occurred, which was confirmed in the “after” graph:

Less pink means less overall CPU time spent on matching calls

So what?

The call we were examining now consumes less of the total CPU time, but how does this translate to the real world? We’ve seen a noticeable drop in actual CPU load per request, shown by the increased gap between the two lines below:

Requests/minute in yellow, CPU load in green (the big dip at 10:40 is an artifact between deployments)

This drop in CPU load translated to increased throughput and a significant drop in response time:

Before this change, our application was largely CPU bound. We tune Apache to throttle connections in tandem with this behavior so that instances aren’t overwhelmed. After this change, there was enough headroom to reduce the application’s footprint by two server instances. We’re now connection bound and can look into opening that throttling up a bit, which could allow us to shed another instance or two. Across all our applications, especially those requiring significant computing resources, these kinds of optimizations can lead to thousands in savings annually.

Pyflame is not just helpful for identifying opportunities, but also helpful for confirming hypotheses. Optimizing calls that appear frequently/loudly in the flame graph can result in tangible benefits, which can ultimately reduce cost. I recommend adding this tool to your arsenal for performance testing and monitoring.

To see more about flame graphs, Pyflame, and profiling tools, check out these resources:

“Visualizing Performance with Flame Graphs” by Brendan Gregg, USENIX ATC ’17
rbspy, a sampling profiler for Ruby (by Julia Evans)
“Build Impossible Programs” by Julia Evans, Deconstruct 2018
“Flamegraph that! Self-service profiling tool for Node and Python services” by Ruth Grace Wong, PyCon Canada 2017

Open sourcing apiron: A Python package for declarative RESTful API interaction

Dane Hillard — Mon, 01 Oct 2018 17:22:32 +0000

This post originally appeared on Build Smarter.

At ITHAKA our web teams write applications that each interact with a large handful of services—sometimes as many as ten. Each of those services provide multiple endpoints, each with their own set of path variables and query parameters.

Gathering data from multiple services has become a ubiquitous task for web application developers. The complexity can grow quickly: calling an API endpoint with multiple parameter sets, calling multiple API endpoints, calling multiple endpoints in multiple APIs. While the business logic can get hairy, the code to interact with those APIs doesn’t have to.

We created a module some time ago for low-level HTTP interactions, and use it throughout our code base. For a good while, though, the actual details of each service call—the service name, endpoint path, query parameters—were scattered throughout the code. This inevitably led to duplication as well as a bug or two when we made an update in one place and forgot about the other.

To reduce the pains from this, we eventually took stock of these scattered configurations and centralized them in one registry module. This module essentially contains a giant dictionary of all the services we interact with:

service_endpoints = {
    'CONTENT_SERVICE': {
        'SERVICE': 'content-service',
        'METADATA_ENDPOINT': '/content/{id}',
        'CITATION_ENDPOINT': '/citation/{citation_type}/{id}',
    },
    'SEARCH_SERVICE': {
        'SERVICE': 'search',
        'SEARCH_ENDPOINT': '/search',
        'EXCERPTS_ENDPOINT': '/excerpt?contentId={content_id}',
    },
    ...
}

Each service has a 'SERVICE' key containing the name of the service used to discover hosts, and some number of '*_ENDPOINT' keys that describe an endpoint and its parameters. Calling these services looks like this:

from http import make_get_request_with_timeout
from services.registry import service_endpoints

CONTENT_SERVICE = service_endpoints.get('CONTENT_SERVICE', {})
METADATA_ENDPOINT = CONTENT_SERVICE.get('METADATA_ENDPOINT', '')

# determine content_id...

metadata = make_get_request_with_timeout(
    service_name=CONTENT_SERVICE.get('SERVICE'),
    endpoint=METADATA_ENDPOINT.format(id=content_id),
    headers={'Accept': 'application/json'},
    request_timeout=5,
)

As you can see, there are a variety of shapes to these endpoints. This solved the issue of duplication across the codebase, but we still faced a couple of problems with this approach:

Strings as endpoint descriptors don’t result in structured data. This is pretty difficult to introspect or validate.
Even with fully-formattable strings, sometimes a call needed to exclude a parameter all together, or add a new one. This had to be done ad-hoc after the fact.
Our HTTP module still had a laundry list of methods, each with slightly different behavior and unclear names like make_get_request_fast (how fast?). Many of these methods called the same underlying methods with different default parameters, and the stack got pretty deep sometimes. Choosing the right method for a call was hard.

In order to address the high variability of behaviors and lack of structured data of this problem, we built a new paradigm for HTTP interactions that provided a declarative interface for configuring services. We wanted a few things out of it:

Code describes how a service interaction looks, not the details of how to make the underlying HTTP call happen.
The endpoint descriptors are structured and support introspection.
Default behaviors can be declared in the service configuration, but can also be easily overridden dynamically at call time.

With these desires in mind, we came up with apiron. With apiron the same definition from above looks more like this:

from services import IthakaDiscoverableService
from apiron.endpoint import Endpoint

class ContentService(IthakaDiscoverableService):
    service_name = 'content-service'

    metadata = Endpoint(path='/content/{id}')
    citation = Endpoint(path='/citation/{citation_type}/{id}')

And the code to call the service looks more like this:

from apiron.client import ServiceCaller, Timeout
from services import ContentService

CONTENT_SERVICE = ContentService()

# determine content_id...

metadata = ServiceCaller.call(
    service=CONTENT_SERVICE,
    endpoint=CONTENT_SERVICE.metadata,
    path_kwargs={'content_id': content_id},
    headers={'Accept': 'application/json'},
    timeout_spec=Timeout(read_timeout=5),
)

We can now define what ContentService looks like and easily refer back to that class whenever we need to understand its shape. Service discovery is now a plugin system. Endpoints can be introspected and have their parameters validated and enforced.

With apiron we’ve been able to replace many of our existing service calls quickly and with little pain. The code has become clearer and with the cognitive load out of the way we can begin focusing on other gains like streaming responses and data compression. It’s been nice for us, and we’d like to make it nice for you too.

You can install apiron from PyPI with pip (or your favorite package manager):

$ pip install apiron

There are a few other helpful tools in the package, so give it a try today!

ithaka / apiron

🍳 apiron is a Python package that helps you cook a tasty client for RESTful APIs. Just don't wash it with SOAP.

apiron

apiron helps you cook a tasty client for RESTful APIs. Just don't wash it with SOAP.

Gathering data from multiple services has become a ubiquitous task for web application developers The complexity can grow quickly calling an API endpoint with multiple parameter sets calling multiple API endpoints, calling multiple endpoints in multiple APIs. While the business logic can get hairy, the code to interact with those APIs doesn't have to.

apiron provides declarative, structured configuration of services and endpoints with a unified interface for interacting with them.

Defining a service

A service definition requires a domain and one or more endpoints with which to interact:

from apiron import JsonEndpoint, Service

class GitHub(Service):
    domain = 'https://api.github.com'
    user = JsonEndpoint(path='/users/{username}')
    repo = JsonEndpoint(path='/repos/{org}/{repo}')

Interacting with a service

Once your service…

View on GitHub