DEV Community: Renato Cordeiro Ferreira

Is it time for Data Science / Engineering patterns?

Renato Cordeiro Ferreira — Sat, 05 Jan 2019 00:06:33 +0000

Most software engineers have already read Design Patterns, the seminal book by Erich Gamma, Richard Helm, Ralph Johnson and John Vlissides (the Gang of Four a.k.a. GoF) that made most programmers of object-oriented languages talk in terms of singletons, iterators, strategies, etc.

For those who don't know the concept, here it's the definition from Wikipedia:

In software engineering, a software design pattern is a general, reusable solution to a commonly occurring problem within a given context in software design. It is not a finished design that can be transformed directly into source or machine code. It is a description or template for how to solve a problem that can be used in many different situations. Design patterns are formalized best practices that the programmer can use to solve common problems when designing an application or system.

I remember reading this book cover-to-cover in two days (a not so common approach since this book is a kind of catalogue) in 2014. I had some experience with object-oriented programming (OOP) from my undergraduate courses. However, I still had some difficulties to design new systems with this paradigm. Suddenly, someone gave me a guide to make new projects. The idea really appealed to me!

It's a common mistake for new GoF's book readers to think that design patterns are "the hammer that will pound all nails". I made this mistake. I tried to plan everything in terms of patterns. But there are no silver bullets. I overshadowed their advantages and pitfalls. They give flexibility in one hand but increase complexity on the other hand. This is one of the biggest criticisms to the concept of design patterns (this and, for GoF patterns, OOP language limitations).

My current master's and former monograph's advisor made his PhD with Ralph Johnson at University of Illinois Urbana-Champaign (UIUC). We've been working together in the refactoring of a machine learning framework created by our research group, ToPS. The framework uses patterns extensively to the point we identified and documented a new design pattern -- the Secretary pattern -- which we presented in the 11th SugarLoaf-PLoP, the Latin American conference of Pattern Languages of Programs.

Thanks to this project and the relationship with my advisor, I think I realized how and why patterns are valuable (at least to me): to standardize and increase the descriptive power of developers' language. As I said in the beginning of this text, GoF's book made most programmers of object-oriented languages talk in terms of singletons, iterators, strategies, etc. This spares time while it makes design / architecture discussions more concise and accurate.

All this context brings me to the reason for this post. Today Andrew Ng, the famous Stanford professor who co-founded Coursera and made one of the most popular machine learning online courses of all time, posted this thread on Twitter:

This post is my (long) response to him.

GoF gave birth to a movement of pattern writers who document individual patterns and whole pattern languages for things such as domain-driven design, enterprise architectures, continuous delivery, microservices, game programming, and many others.

But where are the data science and data engineering patterns?

As an enthusiast of this way of documenting best practices, I believe that we should search for these patterns and document them. My first realization of this was when I went to SugarLoaf Plop in 2016 and saw patterns for many areas, but none related to these. As Andrew Ng said (emphasis mine):

I'm also seeing many AI teams use new processes that haven't been formalized or named yet, ranging from how we write product requirement docs to how we version data and ML pipelines.

As I said above, I think patterns are most valuable to standardize and increase the descriptive power of developers' language.

This is an exciting time for developing these ideas!

So let's create them!

It's important to notice I talked about data science and data engineering instead of AI or ML as Andrew Ng said. This was a thought word-choice. I usually describe machine learning as one of the artificial intelligence techniques (statistical learning) available for data scientists when they're trying to reason about their data. However, data science is more broad and include other tasks (such as data mining or visualization) where patterns also could be useful. Meanwhile, dealing with lots of data has its own challenges, thus (big) data engineering can also benefit from this approach.

Thanks for reading! What do you think? Would you read data science / engineering patterns? Do you believe this is a good way to document best practices for the area?

In my PhD (which I hope will start soon), I want to explore the relationship between software engineering and data science / data engineering. I think that exploring patterns could be an interesting direction to take, hence my interest for Andrew Ng's tweet.

If you liked this discussion, take a look in my other post where I wrote about software engineering for the first time. There I introduced my idea of R.A.D.I.C.A.L systems, which can also use AI/ML for Intelligent data transformation.

Building for Scale? Make a R.A.D.I.C.A.L. System!

Renato Cordeiro Ferreira — Wed, 26 Dec 2018 23:56:44 +0000

The year is ending, 2019 is almost here and soon the 2010s will come to an end. This decade was marked by services from companies like Netflix, Google and Facebook thriving and becoming part of our daily lives.

For us in the tech field, these companies are examples. They build and operate systems that handle billions of users, 24/7, all around the world. Just think about it. That's a scale hard to grasp and many would doubt we could have so many people connected just a decade ago.

In this post I want to introduce a new concept came up with a couple days ago: R.A.D.I.C.A.L. systems, that is, systems that are Reactive, Automated, Distributed, Intelligent, Complex, Agile, and Lean.

I invented this acronym as a way to assemble these techniques that big companies are using to create successful systems that scale. I've been reading lots of books and watching lots of talks about these subjects and I realized there was no term to describe systems that mix all of them.

Most of the concepts here are not new, but they're becoming more and more popular thanks to the challenge of making systems -- R.A.D.I.C.A.L. systems -- capable of serving a huge number of people.

I'll start backwards by introducing each letter of the acronym in the reverse order. I believe this is the most constructive way to introduce the concepts. Whenever possible, I'll put links for those interested in reading more about the subjects I'm talking about.

This is my first public post about software engineering, so I hope you enjoy!

1. `L` stands for Lean Inception

Projecting systems from the scratch is hard, mainly when we don't know the business domain that we want to tackle. Lean Startup by Eric Ries taught us how to build, measure and learn to create a Minimum Viable Product (MVP) in an environment of uncertainty. On the other hand, Design Thinking brought to our attention the importance of research to have insights about problems in order to ideate and generate prototypes to get a specific solution. How can we mix these mindsets to create a new project?

Lean Inception by Paulo Caroli is a methodology that joins some techniques from Lean Startup and Design Thinking to start a new product. It does that by aligning the ideas of all stakeholders, product owners and developers involved in the project. The result (by the end of the one week session) is a set of MVP Canvas that describe the main goals, advantages and drawbacks of each iteration of development. They are accompanied by user stories that can be used by developers during the construction of the system.

A R.A.D.I.C.A.L system should use the Lean Inception to choose which and when features should be built.

2. `A` stands for Agile Development

The Agile Manifesto is 17 years old and being agile is no longer hipster but rather mainstream. It's hard to find a developer who does not agree with the core values of agility, where individuals and interactions, working software, customer collaboration and responding to change are valued respectively more than processes and tools, comprehensive documentation, contract negotiation and following a plan.

The flexibility of Agile made it the default mindset when companies start developing -- whichever techniques they choose to use, prepackaged in methodologies such as Extreme Programming (more focused on the developer-side), Scrum (more focused on the management side), Kanban (focused on the organization of tasks), and many others.

A R.A.D.I.C.A.L system should use the Agile Development to optimize how and where features should be built.

3. `C` stands for Complex Event Processing

Millions to billions of users produce many, many (did I say many?) data, all the time, and we need a way to handle it. In an era of Internet of Things where more and more devices are connected together, we need a way to handle all the signals they produce when interacting with humans and other machines.

Complex Event Processing is a theory that advances many (big) data processing techniques to establish a well-defined method to deal with infinite data sets, generating ways to query, transform and mix different streams. It gives a framework to abstract dataflows, processes and exceptions that are happening inside a system.

A R.A.D.I.C.A.L system should use Complex Event Processing techniques to track and analyze the data inside it.

4. `I` stands for Intelligent Data Transformation

Artificial Intelligence and, in particular, Machine Learning are trending in the last couple years since humans can no longer grasp the amount of data we are producing. Pattern Recognition, Computer Vision, Natural Language Processing are some of the areas that AI/ML are making the most impact.

The result is a new generation of systems that make things that we could only dream of a decade ago, such as voice assistants, autonomous cars and reusable rockets. As the ecosystem matures, using algorithms and models from these areas to find out valuable information to your product is no longer a task only for PhDs.

A R.A.D.I.C.A.L system should use Intelligent Data Transformation techniques to learn and generate insights from the data inside it.

5. `D` stands for Distributed Environments

Some say that Cloud Computing is making to computation what the first power plants made to electricity. We no longer need to lose time maintaining our own servers to run and scale a system for new users. The cloud, however, brought many new challenges for developers. We no longer have the control of when a machine will be available, neither where exactly our systems are running.

Distributed systems are as old as the first networks. It is a cheap solution to increase processing power and the cloud made it much cheaper. However, the CAP theorem made the words availability and consistency known by every developer that deals with these systems. This mix of theoretical and practical challenges pushed the creation of a new generation of tools that encapsulate the best patterns to manage these environments.

A R.A.D.I.C.A.L system should use Distributed Environments to be scalable in a simple and unrestrained manner.

6. `A` stands for Automated Infrastructure

If servers are out of our hands (thanks to the cloud), automation is very important to guarantee that we can run our services anywhere (particularly with the least vendor lock-in as possible). Infrastructure as Code was the first step to enable developers to take part of operators responsibility, leading to the much broader Cloud-Native movement (which includes technologies for virtualization, orchestration, observability and many other buzzwords) and the Serverless movement (which tries to free developers to think in nothing but their business logic).

At the same time, DevOps is shining as a mindset to close the gap between development (creating new things) and operations (maintaining existing things), allowing the full ownership of products by those who created them. This heavily relies on and allows Continuous Delivery of new features, where they can be packaged, tested and deployed with shorter cycles.

A R.A.D.I.C.A.L system should use Automated Infrastructure to be deployable in a safe and reproducible manner.

7. `R` stands for Reactive Microservices Architecture

Complex business cases may demand large systems that span many business subdomains. Domain-Driven Design by Eric Evans brought the concept of a ubiquitous language that should be used to describe these systems and bounded contexts that divide them. These are the building blocks for a Microservices architecture, that explores their semi-independence to create a system composed by many independent services which can be developed by standalone teams.

But building microservices is not a silver bullet for modern systems since there is a set of prerequisites in order to adopt them instead of a monolith. Anyway, the architecture still can become a big ball of mud. The Reactive Manifesto indicates how creating responsive, resilient, elastic and message-driven services may help to strength systems for scale.

A R.A.D.I.C.A.L system should use Reactive Microservices Architecture to encode complex business models and domain-specific knowledge.

That's it! What do you think about R.A.D.I.C.A.L systems? I'm sure they are not easy to make, and that grasping every concept requires lots of learning -- but it's been a very nice journey.

I want to thank my friends Fernando, Rafael and Victor for the valuable feedback. This post is just scratching the surface, but I want to highlight that there are systems -- R.A.D.I.C.A.L. systems -- that mix all these concepts.

If you want to discuss more about these ideas, please reach me at Twitter or LinkedIn.

DEV Community: Renato Cordeiro Ferreira

Is it time for Data Science / Engineering patterns?

Building for Scale? Make a R.A.D.I.C.A.L. System!

1. L stands for Lean Inception

2. A stands for Agile Development

3. C stands for Complex Event Processing

4. I stands for Intelligent Data Transformation

5. D stands for Distributed Environments

6. A stands for Automated Infrastructure