Data Science is dying out

Gabriel Pickard — Wed, 26 Feb 2020 17:11:59 +0000

As a Data Scientist you usually do one of two things:

You comb through a pile of numbers derived from a SQL database, S3 or an HDFS cluster, in order to answer questions like "what keeps customers from converting?" and make predictions like "How much is a given user worth?".
You build Machine Learning models that do actual ongoing work in deployed applications.

For both of these task groups there exists a different job title with similar job description:

If you replace the SQL with Excel spreadsheets, then Analysts have been and still are answering the same kind of questions.
Software Engineers have been doing Machine Learning for a long time. Long enough that there now is such a thing as a Machine Learning Engineer.

Data Scientists distinguish themselves from Analysts by knowing a bit more about programming. They distinguish themselves from Software Engineers by knowing less about programming. They differ from both in that Data Scientists usually are expected to have gone to grad school and know a thing or two more about math.

The "Data Scientist" job title was a way for industry to hire academics to do the kind of computational modeling that PhD candidates usually do in academia.
Also, it was a way for managers to feel cool, hip and with it.

But it's dying.

Why you say? Well, hear me out:

Analysts can and do learn how to write SQL queries. Often (not always) the math involved in answering business questions can actually be kept relatively simple, you often don't need a PhD to do it. Doesn't hurt though, I guess.
Machine learning is getting commodified. You no longer need a PhD and write your own backpropagation algorithm in order to do Deep Learning. (It's sad, I know.) All you need is GPUs and money to burn. -- Well, and feature engineering, you do need feature engineering from time to time. And some experience in the field and domain knowledge. But all those can be acquired by any bright individual. Also, if your models are to do real work, they usually need to be integrated into real code. Here it comes in handy to know a thing or two about Software Engineering. Hence the rise in popularity of the ML Engineer.

Now I know for a fact that once you give a group of people in the professional class a title and a decent salary, they'll make sure their field will continue to be needed, come hell or high water. So, in fact: I lied.

Data Science is going nowhere, it's just that if you happen to not have Data Scientists on hand at your organization, don't fret, you still can get stuff done.

But if you decide you do desperately need someone to make sweet, sweet Science to your Data, you might be in luck: I'm a Sciencer of Data myself, possibly accepting clients. -- Or maybe I'm an ML Engineer, or something like that... Just give me a ring.

Developer Experience: fundamentally harder than normal UX

Gabriel Pickard — Sun, 23 Feb 2020 20:56:13 +0000

Many (maybe most) of us Software Engineers are deeply involved in projects where UI and UX are very important concerns (or at least they should be). Throughout the short history of our profession, we have become more and more focused on understanding and serving the people who will actually be subjected to the stuff we build: users. In many cases, this focus on UI/UX isn't because we particularly enjoy worrying about such matters -- no, it's simply because we have to if we want to build successful products!

There has been tremendous growth in the field of UX/UI. Without a doubt, today's applications are much more user-friendly than those from 30 years ago. Back then, users were given features, interface be damned. It was only over time that customers came to appreciate, expect, demand and reward those applications that actually cared to consider their needs.
Consumer demand drove UX as a discipline.

This process has been fast in some areas, slow in others. Nowhere has it been slower than in the realm of programming tools. We coders still put up with horrid UX/UI when programming.

Why DX lags behind

In my view the User Interface of programming encompasses a lot, from the type system of the programming language that you use, its error messages, to the editor you're writing code in, the websites you go to in order to get help, all the way to the cloud hosting systems you deploy on. Developer Interface / Developer Experience is comprised of all of this.

Counter to any UI/UX philosophy, as programmers we find ourselves maintaining vast background knowledge about the structure and dynamics of our programs, with nary a visual cue to help us.
It's often so hard to figure out what exactly went wrong when something goes wrong.
A lot of tools are truly ugly!

When was the last time you heard of a programming language discussed in terms of discoverability, succinctness, relevance, let alone beauty?

I believe there are two reasons for the discrepancy between general UX and DX:

Coding tools were around before UI/UX was a thing. We've simply gotten used to them: Dealing with the idiosyncracies of bash, vi, or the JavaScript type system have become a part of the professional hazing process. They may be suboptimal, but they're ours! This is compounded by the fact that the history of technology is so path dependent. In so many cases it is much easier and more profitable to build on something existing rather than reinvent the wheel. -- Even if the existing wheel is full of arbitrary decisions and in fact impediments to what you are trying to do. Just look at the long reign of the x86 architecture, or consider the fact that basically all Operating Systems these days follow the Unix architecture. The fact that JavaScript is what it is. -- I don't mean to say that these technologies are without merit, however they also have considerable flaws and it has been more efficient to work with and around than to replace them entirely. Certainly every mature technology will have to deal with technical debt in the inner workings of its guts. I have no bones with that. However, if the gory details are in the guts, then we coders are the GI surgeons having to deal with them. -- And we need sharper scalpels, better imaging and protective gear. In short: better tools.
Fundamentally, programming is a Turing Complete business. You can view a UI as a form of (visual) formal language. For most GUIs the language complexity, the different states that the application interface can find itself in, would largely correspond to regular languages (some may be context-free, but that's pushing it). The visual languages are "flat", predictable, without feedback loops or long distance dependencies. It is in such an environment that UI/UX principles have been able to thrive. But what did our Intro to Theoretical CS course teach us in college? You can't assume that statements about regular languages will still hold for Linearly Bounded Automata, let alone Turing Machines. Fundamental principles in conventional UI/UX design no longer apply when it comes to programming. DI/DX is a very, very special case UI/UX.

So, what I believe has happened is that in many cases we programmers have tilted at windmills of trying to improve our tooling -- the eternal yak shave -- only to be thwarted by the complexity of our own work, inevitably giving up and going with the most rudimentary system capable of doing the job somehow, even if it sucks.

Toward better DX

So, what's the alternative? Here are my suggestions for a world with better Developer Experience:

Watch what coders actually do -- this is one principle from UX that applies readily. Library developers rarely know what error messages users most commonly get stuck on; How about user studies? We track every click that people make on our website and agonize over button sizes, but know precious little about how common which call paths in our framework are. The elephant in the room here of course is privacy.
Address the many components that make up programming in concert: Editor, shell, repl, language, version control host, cloud host and framework. Generally these pieces are not very aware of each other. There is some movement on this front: Microsoft is gearing up to tightly integrate VSCode, GitHub and Azure. This portends market domination. Competitors take note!
Search -- It's our dirty little secret how much programming nowadays depends on Google. Arguable it is the most important item in our toolbelt. However, search engines nowadays do a uniquely poor job supporting developers. Most of recent Natural Language Understanding developments have made search less precise and less useful for coders, while hardly any new features have been added with our needs in mind. I find this topic fascinating and I'll be writing more about it-- coming up!
Standardize fundamental questions -- How do I run this thing? Where does the code start? Is my system configured correctly? Project Readmes are hardly ever up to date. Why do we treat this as a moral failing instead of a usability issue?
Tests are a usability dead end -- I know this may be contentious, but I believe test suites are another realm of excessive moralizing en lieu of better tools and better processes. Too often they function as a security blanket that simply encases the parts of the code that are unit testable, while leaving the vulnerable, untestable bits fluttering in the wind. Approaches such as generative testing seem more promising.
Intelligently hide details -- Writing code is about control, however understanding code is not. Most environments throw up their hands and overwhelm you with the entire pile of everything the program does, in minute detail. Others take a different tack and try to hide the bitter realities behind magic -- impenetrable and beautiful until you inevitably do have to care about what's behind the curtain. Why do we have so few systems that would allow us to zoom in and out? Get you a coding environment that can do both.
Coding is a social process -- The success of GitHub is a testament to this. I'm not sure, but maybe we can do even better along these lines.
Programming Languages are User Interfaces -- The most fundamental unit of Developer Experience is the programming language. The Elm language is one of few examples where this reality was considered explicitly in the design process.
Programming is about empowering -- dumbing things down is not enough. I believe minimalism is a cheap approach to UX in general, but it definitely doesn't work for DX -- it fundamentally misses the point of what programming is about: expressive power. I believe this is why many "graphical"/beginner programming languages have failed.
Engage with Theoretical Computer Science -- In a way we are hunting for UI widgets (or other kinds of artifacts) that can faithfully represent the full computational complexity of algorithms. I have been actively working on this front and it's a long term project that you hopefully will be hearing more about. Representing computational complexity is also the one area where Bret Victor's excellent work may have fallen short of what we ultimately need.

DX is worth it

Interest in this topic is growing. It's definitely worth looking into. The software industry is so big and expensive that even small improvements can have considerable impact. Furthermore, Software Engineers actually do respond to better UX -- and they wield considerable purchasing power. Consider the following example:

I once worked at a small company, which for regulatory reasons couldn't use the normal cloud-hosted GitHub. Our engineering department strong-armed the rest of the company to purchase on-premise GitHub Enterprise at a cost equivalent to hiring one or two additional engineers, just so we could use Pull-Requests! We recognized the difference it made in our productivity and we demanded it -- only the best of tools would do.

This article first appeared at: https://www.gabrielpickard.com/posts/developer-experience-fundamentally-harder-than-normal-ux/

DEV Community: Gabriel Pickard

Data Science is dying out

Developer Experience: fundamentally harder than normal UX

Why DX lags behind

Toward better DX

DX is worth it