DEV Community: Yuelin Wen

The issue with Monorepos

Yuelin Wen — Fri, 03 Apr 2020 20:49:50 +0000

A monorepo (also called a monolithic repository) is an arrangement where a single version control system (VCS) repository is used for all the code and projects in an organization. The alternative to monorepo is having multiple repositories for different projects or even functionalities. In that case, the system will be called a polyrepo. In this article, we will define the idea of a monorepo and identify some of its benefits and some famous companies using it. Finally, we will focus on whether a monorepo is suitable for everyone.

By the end of this piece, you’ll notice that even though the monorepo system is used by many tech giants such as Google, Facebook, and Uber, it still has its challenges. I found the observation by Google's spokesman interesting in this regard. She acknowledges that using monorepo isn't for everyone. But how does she support that argument? I’ll be focusing on some of her reasons by looking at the advantages and disadvantages of the system.

Why Some Tech Giants Employ a Monorepo setup

The fact that some of the biggest tech giants of our time are using a monorepo system implies that it has some benefits. What could be some of the great attributes of this setup which attract them to it? Let's look at the leading three.

Extensive Code Sharing and Collaboration Across Teams

The main advantage of a monorepo is that all code exists in a single repository. Therefore, team members across an organization work in the same place. This results in a lowered sense of ownership and blurred boundaries. Consequently, there is an increased likelihood of code reuse, less code duplication, and more teams working on shared infrastructure.

The unclear boundaries which such a system creates make changes made by other developers feel less invasive, a situation that fosters more interaction among developers and teams. As a result of these advantages, more useful libraries build-up and more projects get to share the same dependencies.

Simplified Dependency Management With a Single Version of Truth

Since a monorepo system places all the code in the same place, everything is built and updated at the same time. There is no need to reach out to other repositories and constantly check for potential breaks. For instance, technical debt generates in polyrepo when changes occur, but dependency updates don't immediately follow. In a monorepo, changes in a dependency immediately lead to an update of the dependent code and no technical debt is accrued, assuming best practices are in place.

In a monorepo, library updates are also made easier when files are not hosted in different repositories. Moreover, because everything is updated constantly, there will be only one version for all the dependencies, avoiding the diamond dependency problem. The diamond dependency problem occurs when two projects depend on one dependency. However, the challenge comes about when the projects depend on different versions of the dependency. The diamond dependency makes release without breakage very difficult. In certain instances, it can make the building of a project impossible.

Atomic Changes/Large-scale Refactoring

When developers have one repository, they search through one database to make atomic changes across the system. On the same basis, code refactoring is made easier since developers do not have to locate different repositories. The layout of the codebase will be organized in a single tree so that developers can easily trace dependencies. For instance, Google gives each source file a unique identification by assigning the file path as their namespace.

If the Tech Giants are Using a Monorepo Setup, Shouldn’t We All?

So, if the monerepo system is employed by some of the biggest tech giants in the world, shouldn’t we be falling over each other to use the same system? Aimee Lucide, a mobile engineer on Uber’s Driver Signups team, seems to have provided the answer. She gave a speech about how Uber started from a monorepo and then switched to a polyrepo. However, Uber would later switch back to monorepo when the business grew huge and successful.

In her presentation, Lucide explained that the reason the monorepo became the rideshare company’s ultimate choice is that their system comes with an Android app studio, it has a single source of truth, and it makes code sharing easier. However, she also admits that her team quickly encountered some serious problems with the monorepo when their codebase scaled. As Uber’s mobile codebase got bigger, Aimee explains, her team started to experience an integrated development environment (IDE) lag. Eventually, developers could no longer scroll without the code freezing up. She explained that every time they tried to build, git slowed down. The reason was that the whole exact clone would have to pull out.

When the codebase is huge, it consumes a huge portion of the developer’s time when they try to work on a project; even the tiniest one. When everyone is working on a single repository, a broken master sometimes occurs. Broken master happens when one person accidentally breaks the code. Consequently, others get unexpected errors that they can’t figure out.

So why did Uber switch back to a monorepo while they still didn't have the answer to the problems encountered on the system? Uber got big enough to throw its resources at the problem. In Lucide’s own words: “As your company starts to get bigger, you can actually invest resources to mitigate some of the original downsides.”

From Lucide’s answer, it is clear that the monorepo challenge persists. What’s different is that Uber can now throw money at them. So, the real question for anyone who wants to use a monorepo is simple: Are you willing to invest as much as Google, Facebook, Twitter, and Uber? For many of us the answer is equally simple: No.

What’s Good For Google is Not Good For All

In the same way buying out Chelsea Market may not be as much a smart move to you as it could be to Google, the benefits of a monorepo to Google may just not be advantageous to you. As noted above, the tech giants are able to use the monerepo system because they have the resources to throw at the challenges. Hence, one could say the reason is not based on the fact that the system has more benefits for everyone.

Let us take a closer look at the weaknesses of some of the supposed benefits that the tech giants like Google see in the monorepo system. This will make you realize why a monorepo may not be for everyone.

Extensive Code Sharing and Collaboration Across Teams

Because a single repository hosts all the projects, code reuse must be more frequent, and code duplication must be lower right? This is not always the case. Why?

Even in a medium-sized monorepo codebase, much like the case for Uber, mentioned above, it doesn’t really make sense for a developer to grab the whole extract clone of the entire repository. Not only does it take a long time, but it also creates a lot of unnecessary trouble in either searching or editing. Therefore, there are typically two methods required for any monorepo that hopes to scale.

Virtual File System (VFS) that Enables Developers to Access Only a Portion of the Code Locally

According to Rachel Potvin, engineering manager at Google, Google has about 2 billion lines of code power. She was speaking at a Silicon Valley conference. This is for all the Google internet services such as Google Search, Google Maps, Google Docs, Google+ and so on. Despite years of experiments, there was still not an optimal open-source version control system that supported their code in one repository.

To solve the challenge, Google built its own version control system called Piper. This system came with features that help developers to browse and change codes in this vast pool of lines. One feature mentioned by Potvin in her Silicon Valley conference presentation which is used by developers to access Piper is CitC. CitC allows developers to browse through the repository without having to clone or sync. While only being able to modify files stored in the developer’s machine, developers can browse and edit files freely across the whole repository.

Sophisticated Source Code Indexing and Tooling for Code Searching and Discovery

Because of the complexity of a monorepo version control system, a tool to help perform a search across the entire repository is crucial. Google invested in building a custom plug-in for the Eclipse IDE to make handling billions of lines of code possible. The Code-indexing of Google also supports static analysis cross-referencing. These are the type of investments required when codebases scale in a monorepo.

Even with tools that satisfy the requirements, developers will only get to access small portions of the code when working on a project. How different is it to view a portion of the tree using a Virtual File System when compared to looking at different repositories? Tools for easily iterating over many different repositories and collecting the results are readily available. The scaling of the codebase, to a large extent, makes working with a single VCS much similar to working with multiple repositories because of the tool requirements. Collaboration and code sharing has, in fact, more to do with engineering culture than ways to store files.

Simplified Dependency Management/One Version of Truth

It is unreasonable to believe that since a monorepo has all code in a single repository and everything is built at the same time, dependency management is unnecessary. At scale, huge efforts are required to help update on a large scale and run automated tests to ensure code health in place of the usual dependency management in polyrepo.

Because it is easy to create dependencies, developers put less effort into establishing well-thought-out APIs. Abandoned projects can also be under the radar and continue to be maintained and updated, leading to an unnecessary loss in productivity. How do the giants deal with this challenge?

Google invested in creating new tools beyond standard tools and human efforts to deal with codebase complexity, underused/unnecessary dependencies, and code discovery. On top of that, more investment is put into making tools and indexing. This creates better search and browsing.

Moreover, not all the projects are deployed at the same time, previously deployed software can be redeployed if needed. This means that in a monorepo, multiple versions of code exist, build artifacts will have to be carefully tracked, much like what happens in polyrepo, where different versions of dependencies need to be reconciled and managed.

Atomic Change/Large Scale Refactoring

A monorepo system may be counter-intuitive. Editing or searching across the whole codebase at the local machine is a much more difficult task than you might expect.

While implementing a very sophisticated VFS, as mentioned above, makes using monorepo or polyrepo the same, making changes to a fundamental library API and having it reviewed by affected teams can be impossible because of the merge conflict in the case for a monorepo. When it happens, developers, much like they do in a polyrepo, will do one of two things: working around the API issue or deprecate the current API and replace it with a new one.

To make refactoring across large amounts of codes, automated tooling for refactoring will need to be in place to make the whole process possible.

More Issues to Look out for in a Monorepo

Anyone who wants to use a monorepo system will need to be conscious of other issues with the system. I focus on some of the leading ones below.

Tight Coupling

Because of the nature of monorepos, certain issues can be experienced. These include unclear boundaries, less sense of ownership, and collaboration. Consequently, projects in a monorepo are prone to depend on each other, creating tight coupling. This leads to brittle software. Brittle software happens when any small fixes can imply alternation in many other parts of the codebase making maintenance impossible without fracturing the whole system.

A polyrepo, on the other hand, can mitigate this kind of issue by offering defined boundaries and responsibilities to each team.

Broken Master

Broken master is also a result of the unclear boundaries in a monorepo. All failing tests on the master are considered to be a broken master. The broken master can seriously reduce productivity if left unsolved. Therefore, before any monorepo projects start, ways or tools to eliminate broken master are required.

The broken master does a lot less harm in a polyrepo because a break in one or two repositories doesn’t have huge impacts on others.

Is it Good for Everyone?

The answer to our question regarding whether a monorepo is good for everyone is simple: No.

No one can question the fact that a monorepo system has its advantages. That is the reason why big tech companies line up to use it. However, there are some who believe the amount of effort put into simply keeping projects in a monorepo VCS running is impractical for smaller organizations. I will leave you with some questions if you may want to switch to the monorepo system. How much is the sense of collaboration worth to your business? Is your company willing to invest more to get the benefit of the system? After all, like Rachel Potvin said in her Silicon Valley conference presentation, monorepo is not for everyone. Therefore, corporations have to look at their specific contexts, think hard, and select the system that best meets their unique needs.

24 influential books programmers should read (2020 updated)

Yuelin Wen — Thu, 02 Apr 2020 22:31:01 +0000

The fast-paced world of programming demands that developers remain up-to-date. In fact, getting ahead of the curve makes a programmer stand out in his professional field. Don’t let yourself fall behind. Innovate so you can offer your company the kind of programming tailored for their needs. Here is a curated list of books that can support any software engineer.

Part 1: Books for Writing Code

Code Craft: The Practice of Writing Excellent Code (Pete Goodliffe)

All programmers know how to write code that works. But only the top coders and programmers can create outstanding code - clean software code that is comprehensible and lasting. Code Craft can help you join their ranks. Move beyond just writing whatever works and write outstanding, award-winning programming. This book teaches the best code presentation style, problem solving techniques, variable naming, error handling, and security.

Code Craft takes readers beyond their own problems and forces them to look at reality in order to evaluate the best way to move forward. The real world requires a level of skill in teamwork, documentation, communication, and process. The advice given is accessible to all programmers and is based on practical experience. Questions and answers at the end of each chapter allow readers to reinforce the ideas presented to them.

Solid Code (Developer Best Practices) / (Donis Marshall)

Efficiency is key in any code developing session. Solid Code looks to help programmers create robust, bug-free code. Two Microsoft .NET experts teach valuable lessons with examples based on real world problems that programmers have to face every day. Learn to navigate expenses and teamwork on a regular basis. Improve your designs, prototypes, debugging, and testing through the myriad of tips offered in this programming book. Start to create incredible, world-changing products and code.

Solid Code teaches you to:

Use metaprogramming to reduce code complexity, while increasing flexibility and maintainability.
Treat performance as a feature - and manage it throughout the development life cycle.
Apply best practices for application scalability.
Employ preventative security measures to ward off malicious attacks.
Practice defensive programming to catch bugs before run time.
Incorporate automated builds, code analysis, and testing into the daily engineering process.
Implement better source-control management and check-in procedures.
Establish a quality-driven, milestone-based project rhythm—and improve your results!

The Clean Coder: A Code of Conduct for Professional Programmers (Robert C. Martin)

In The Clean Coder: A Code of Conduct for Professional Programmers, legendary software expert Robert C. Martin advises starting programmers to code with discipline and technique, instead of simply following directions. He crammed each page with enormous value: learn how to estimate, code, refactor, and test with ease.

But Martin does so much more than just teach rudimentary technique. He also explains the importance of attitude. The Clean Coder focuses on approaching software development with honor, self-respect, and pride.

Coders will learn to:

Work well and work clean.
Communicate and estimate faithfully.
Face difficult decisions with clarity and honesty.
Understand that deep knowledge comes with a responsibility to act.

Readers will also learn:

What it means to behave as a true software craftsman.
How to deal with conflict, tight schedules, and unreasonable managers.
How to get into the flow of coding, and get past writer’s block.
How to handle unrelenting pressure and avoid burnout.
How to combine enduring attitudes with new development paradigms.
How to manage your time, and avoid blind alleys, marshes, bogs, and swamps.
How to foster environments where programmers and teams can thrive.
When to say “No”–and how to say it.
When to say “Yes”–and what yes really means.

Debugging the Development Process: Practical Strategies for Staying Focused, Hitting Ship Dates, and Building Solid Teams (Steve Maguire)

Steve Maguire tells the story of how his Microsoft software teams learned how to develop the best software without wasting a dime or a day. Some of his methods could be considered controversial, but they worked for Maguire. He does not hold back about what succeeded and what failed for his team at Microsoft.

Maguire also details how to take the energy that one programmer might have for his job and channel it to the whole team to guarantee effective, enjoyable work.

Other lessons include:

Why you might want to kick your star programmer off your team.
How to avoid corporate snares and overblown corporate processes.
How to make tiny changes that produce major results.
How to deliver on schedule without overwork.
How to pull twice the value out of everything you do.
How to get your team on a creative roll.
How to raise the average programmer level at your company.

Refactoring: Improving the Design of Existing Code (Martin Fowler)

As the application of object-oriented programming has become commonplace, a new problem confronts the software development community.

The market has been flooded with poorly designed software that have been developed by programmers who are not as experienced. This means the applications are not only inefficient, but overall a waste of money and intellectual resources. Only now developers are realizing how difficult these systems are to work with. Books for software engineering have not addressed this problem as much.

Expert-level programmers have found better ways to solve these problems with complex yet brilliant methods meant to overwrite the performance of the program but uphold its structural integrity. These practices have not been passed down to programmers who follow in their footsteps. This book seeks to change that.

In Refactoring: Improving the Design of Existing Software, Martin Fowler breaks the silence. The programming book demystifies these master practices and demonstrates how software practitioners can realize the significant benefits of this new process.

Code Complete: A Practical Handbook of Software Construction, Second Edition (Steve McConnell)

This is widely considered as one of the best practical guides to programming, effectively demonstrating the art and science of software development. The author, Steve McConnell, integrates many techniques and must-follow principles and turn them into actionable guidance. He pulls them from research, academia, and daily business practice.

This guidance is suitable for readers with various experience levels, different backgrounds, or different project sizes. Readers can learn to develop the highest quality clean software code.

These are the main topics in this book:

Design for minimum complexity and maximum creativity.
Reap the benefits of collaborative development.
Apply defensive programming techniques to reduce and flush out errors.
Exploit opportunities to refactor—or evolve—code, and do it safely.
Use construction practices that are right-weight for your project.
Debug problems quickly and effectively.
Resolve critical construction issues early and correctly.
Build quality into the beginning, middle, and end of your project.

Clean Code: A Handbook of Agile Software Craftsmanship (Robert C. Martin)

Clean Code is divided into three parts. The first teaches the necessary elements of clean software code: principles, patterns, and practices.

The second part is a series of case studies detailing the problems and solutions in varying order. Each case study is an exercise in cleaning up code: you will learn to transform a code base that has some problems into one that is sound and efficient.

The third part is a study in thought: a contemplation of heuristics that were a part of the case studies but require more understanding. This programming book is a must for any developer, software engineer, project manager, team lead, or systems analyst with an interest in producing better code.

Readers will come away from this book understanding:

How to tell the difference between good and bad code.
How to write good code and how to transform bad code into good code.
How to create good names, good functions, good objects, and good classes.
How to format code for maximum readability.
How to implement complete error handling without obscuring code logic.
How to unit test and practice test-driven development.

Code Simplicity: The Fundamentals Of Software (Max Kanat-Alexander)

Good software development results in simple code. But because the value of simple code was not recognized until recently, there is almost no simple code in the digital world. Programmers were encouraged to flood the market with their incomprehensible code. This book seeks to change that. It is a guide meant to help programmers understand what is needed to make good code and details universal principles that can be used for any programming language or project.

Skill level and experience don’t matter here. You’ll learn how to build and adhere to a plan for software development. You will also become better at managing your time in determining where your programming will go and what path it will take.

Learn what differentiates great programmers from poor programmers.
Understand the ultimate purpose of software and the goals of good software design.
Determine the value of your decisions now and in the future.
Examine real-world examples that demonstrate how a system changes over time.
Learn to allow for the most change in the environment with the least change in the software.
Make easier changes in the future by keeping your code simpler now.
Understand why to write tests and what tests to write.

Part 2: Books for Software Engineering

Rapid Development: Taming Wild Software Schedules (Steve McConnell)

Everyone in the technical world is looking to gain control over their development schedules. In RAPID DEVELOPMENT, author Steve McConnell addresses that concern head-on with overall strategies, specific best practices, and valuable tips to help shrink and control development schedules, as well as keep projects moving. Inside this programming book, you’ll find:

A rapid-development strategy that can be applied to any project and the best practices to make that strategy work.
Candid discussions of great and not-so-great rapid-development practices - estimation, prototyping, forced overtime, motivation, teamwork, rapid-development languages, risk management, and many others.
A list of classic mistakes to avoid for rapid-development projects, including creeping requirements, shortchanged quality, and silver-bullet syndrome.
Case studies that vividly illustrate what can go wrong, what can go right, and how to tell which direction your project is going.

Rapid Development is a real-world guide to more efficient application development.

Software Project Survival Guide (Developer Best Practices) / (Steve McConnell)

The author of the classics CODE COMPLETE and RAPID DEVELOPMENT has more wisdom to impart to his readers. Steve McConnell draws on solid research and a career's worth of experience to map the surest path to your goal - what he calls, "one specific approach to software development that works pretty well most of the time for most projects."

Nineteen succinct chapters teach how to attack and overcome the development process. Readers will learn to plan, design and manage their programming, as well as guarantee quality, testing, and archiving. Simple, elegant code is not out of reach for all programmers in the field.

Agile Software Development: The Cooperative Game (Alistair Cockburn)

Alistair Cockburn begins by updating his powerful model of software development as a “cooperative game of invention and communication.” Among the new ideas, he introduces:

harnessing competition without damaging collaboration.
learning lessons from lean manufacturing.
balancing strategies for communication.

Cockburn also explains how the cooperative game is played in business and on engineering projects, not just software development.

Next, he systematically illuminates the agile model, shows how it has evolved and answers the questions developers and project managers ask most often, including:

Where does agile development fit in our organization?
How do we blend agile ideas with other ideas?
How do we extend agile ideas more broadly?

Cockburn takes on crucial misconceptions that cause agile projects to fail. You’ll also find a thoughtful discussion of the controversial relationship between agile methods and user experience design.

This edition contains important new contributions on these and other topics:

Agile and CMMI.
Introducing agile from the top down.
Revisiting “custom contracts.”
Creating change with “stickers.”

Getting Real: The Smarter, Faster, Easier Way to Build a Successful Web Application (Jason Fried and others)

Through the example of 37signals, readers will learn business, design, programming, and marketing principles that apply to their programming. This book packs a powerful punch with simple, straightforward insights, contrarian points of view, and an outside-the-box view of software design. But don’t be fooled: this is not simply a tutorial. It’s a programming book of ideas.

Digital web developers and programmers of all kinds can find the answers they seek in this book. 37signals had wild success in five apps: Basecamp, Campfire, Backpack, Writeboard, and Ta-da Last, as well as their open-source web application framework, Ruby on Rails. All this was built in just two years with no outside funding, no debt, and only seven people (distributed across seven time zones).

This process was successful for over 500,000 people around the world who all sought to use these applications. Access the knowledge needed to gain success in the pages of this programming book.

Perfect Software: And Other Illusions about Testing (Jerry Weinberg)

Perfect Software and Other Illusions About Testing seeks to destroy negative ideas about testing and testers. Jerry Weinberg, the author of The Psychology of Computer Programming, dismisses the terrible notions that surround that step in software development. Testing isn’t a stigma, it’s necessary.

Be entertained and informed by Weinberg’s expert story-telling while learning about what is possible in the world of software testing. Fallacies and rumors are swept aside and readers learn to steer clear of common pitfalls.

Meanwhile, test strategy development that works for any project is taught here.

Topics include:

Why Not Just Test Everything?
Information Immunity.
What Makes a Test Good?
Major Fallacies About Testing
Determining Significance.
Testing Without Machinery.
and much more.

Coder to Developer: Tools and Strategies for Delivering Your Software (Mike Gunderloy)

Coder to Developer helps you excel at the many non-coding tasks entailed, from start to finish, in just about any successful development project. Furthermore, it equips you with the mindset and self-assurance required to pull it all together, so that you see every piece of your work as part of a coherent process.

Inside, you'll find plenty of technical guidance on such topics as:

Choosing and using a source code control system.
Code generation tools--when and why.
Preventing bugs with unit testing.
Tracking, fixing, and learning from bugs.
Application activity logging.
Streamlining and systematizing the build process.
Traditional installations and alternative approaches.

To pull all of this together, the author has provided the source code for Download Tracker, a tool for organizing your collection of downloaded code, that's used for examples throughout this book. The code is provided in various states of completion which allows readers to dig deep into the actual process of building software. But you'll also develop "softer" skills, in areas such as team management, open-source collaboration, user and developer documentation, and intellectual property protection.

If you want to become someone who can deliver not just good code but also a good product, this programming book is the place to start. This book is essential for building successful software projects.

Software Estimation: Demystifying the Black Art (Developer Best Practices) / (Steve McConnell)

In his highly anticipated programming book, acclaimed author Steve McConnell unravels the mystery to successful software estimation. Software Estimation distills academic information and real-world experience into a practical guide for working software professionals.

Instead of arcane treatises and rigid modeling techniques, this guide highlights a proven set of procedures, understandable formulas, and heuristics that individuals and development teams can apply to their projects to help achieve estimation proficiency.

Discover how to:

Estimate schedule and cost—or estimate the functionality that can be delivered within a given time frame.
Avoid common software estimation mistakes.
Learn estimation techniques for you, your team, and your organization.
Estimate specific project activities - including development, management, and defect correction.
Apply estimation approaches to any type of project—small or large, agile or traditional.
Navigate the shark-infested political waters that surround project estimates.

Software Engineering: A Practitioner's Approach (Roger S. Pressman)

The programming book is organized in five parts: Process, Modeling, Quality Management, Project Management, and Advanced Topics. The chapter count will remain at 32, unchanged from the sixth edition. However, eight new chapters have been developed and another six chapters have undergone major or moderate revisions. The remaining chapters have undergone minor edits/updates.

Facts and Fallacies of Software Engineering (Agile Software Development) / (Robert L. Glass)

While reading Facts and Fallacies of Software Engineering, you may experience moments of “Oh, yes, I had forgotten that,” alongside some “Is that really true?”

The author of this book doesn’t shy away from controversy. In fact, each of the facts and fallacies is accompanied by a discussion of whatever controversy envelops it. You may find yourself agreeing with a lot of the facts and fallacies, yet emotionally disturbed by a few of them! Whether you agree or disagree, you will learn why the author has been called “the premier curmudgeon of software practice.”

These facts and fallacies are fundamental to the software building field.

Part 3: Books for Mentality

The Intentional Stance (Daniel C. Dennett)

How are we able to understand and anticipate each other in everyday life, in our daily interactions? Through the use of such concepts as belief, desire, intention, and expectation asserts Daniel Dennett. The author spent twenty years developing this full-scale presentation of a theory of intentionality.

According to Dennett, humans adopt a stance,a predictive strategy of interpretation that presupposes the rationality of the people - or other entities - we hope to understand and predict.

These principles of radical interpretation have far-reaching implications for the metaphysical and scientific status of the processes referred to by the everyday terms of folk psychology and their corresponding terms in cognitive science. While Dennett's philosophical stance has been steadfast over the years, his views have grown richer and more refined.

The Intentional Stance brings together both previously published and original material: four of the book's ten chapters - (Chapters One, Eight, Nine, and Ten) - appear here for the first time the remaining six were published earlier in the 1980s but were not easily accessible. The older chapters are followed by reflections - essays reconsidering and extending the claims of the earlier work.

These reflections and the new chapters represent the vanguard of Dennett's thought. They reveal fresh lines of inquiry into fundamental issues in psychology, artificial intelligence, and evolutionary theory, as well as traditional issues in the philosophy of mind. It is one of the most crucial books for software engineering.

The Back of the Napkin: Solving Problems and Selling Ideas with Pictures (Dan Roam)

When Southwest Airlines co-founder Herb Kelleher brainstormed a solution to beat the traditional hub-and-spoke airlines, he grabbed a bar napkin and a pen. He drew three dots to represent Dallas, Houston, and San Antonio and three arrows to show direct flights. The image made it easy to sell Southwest Airlines to investors and customers. It also solved his problem.

Used properly, a simple drawing on a humble napkin is more powerful than Excel or PowerPoint. It helps crystallize ideas, force people to think outside the box, and communicate ideas in a direct fashion. In this book, Dan Roam argues that everyone is born with a talent for visual thinking, even those who swear they can’t draw.

This book shows readers how to clarify a problem or sell an idea by visually breaking it down. It promotes a simple set of visual tools that take advantage of everyone’s innate ability to look, see, imagine, and show. These tools are supported by twenty years of visual problem solving, combined with the recent discoveries of vision science.

THE BACK OF THE NAPKIN proves that thinking with pictures can help anyone discover and develop new ideas, solve problems in unexpected ways, and dramatically improve their ability to share their insights. This book will help readers to literally see the world in a new way.

The Timeless Way of Building (Christopher Alexander)

Christopher Alexander believes that the current theory of architecture in our world is bankrupt.

More and more people are aware that something is deeply wrong. But these ideas seem so powerful that critics are uncomfortable, even afraid, to openly say how they feel. Disliking what is happening might make them look foolish.

Now, at last, there is a coherent theory which describes in modern terms an architecture as ancient as human society itself.

The Timeless Way of Building is the introductory volume in the Center for Environmental Structure series. Christopher Alexander presents a new theory of architecture, building, and planning. This theory has at its core that the age-old process by which the people of a society have always pulled the order of their world from their own being.

Alexander writes, "There is one timeless way of building. It is thousands of years old, and the same today as it has always been. The great traditional buildings of the past, the villages and tents, and temples in which man feels at home have always been made by people who were very close to the center of this way. And as you will see, this way will lead anyone who looks for it to buildings which are themselves as ancient in their form as the trees and hills, and as our faces are."

Hackers and Painters: Big Ideas from the Computer Age (Paul Graham)

"The computer world is like an intellectual Wild West, in which you can shoot anyone you wish with your ideas if you're willing to risk the consequences. " --from Hackers & Painters: Big Ideas from the Computer Age, by Paul Graham

We live in the computer age. Our world, our society, is built and perfected by computer programmers and software designers. These people call themselves hackers. Who are these people, what motivates them, and why should you care?

Consider these facts: Everything around us is digitized. Typewriters don’t exist. They have been replaced by a computer. Your phone functions like a computer. So has your camera. Your television is a glorified computer. Your car was not only designed on computers but has more processing power in it than a room-sized mainframe did in 1970. Letters, encyclopedias, newspapers, and even your local store are being replaced by digital products. The world that existed fifty years ago looks very different from the world that exists now.

Hackers & Painters: Big Ideas from the Computer Age gathers historical evidence to draw a meta picture of the phenomenon of digital progress. In an easy-to-read style, Paul Graham explores the “intellectual Wild West.”

This book teaches programming best practices, including the importance and necessity for order and symmetry in software design, how to become profitable, the separation between heresy and free speech, the history of programming language and its current peak today, the open-source movement, digital design, and internet startups. It ranks highly among books for software engineering.

The Productive Programmer (Theory in Practice (O'Reilly)) / (Neal Ford)

In a software developer career, a proven method to produce the code in a more efficient manner is key. The Productive Programmer teaches methods that will tweak your current system for developing software. It is simple to read and even simpler to improve your work.

Master developer Neal Ford discusses how to work smarter, using all of the tools and resources you have to create the best code. He also shows how to stay motivated, get the most out of your tools, and avoid doing the same task over and over again. Become more valuable to your team!

You'll learn to:

Write the test before you write clean software code.
Manage the lifecycle of your tools.
Build only what you need now, not what you might need later.
Apply ancient philosophies to software development.
Question authority, rather than blindly adhere to standards.
Make hard things easier and impossible things possible through meta-programming.
Be sure all code within a method is at the same level of abstraction.
Pick the right editor and assemble the best tools for the job.

The Effective Engineer: How to Leverage Your Efforts In Software Engineering to Make a Disproportionate and Meaningful Impact (Edmond Lau)

Effective engineers have a certain system that they employ in order to get work done in a fast, efficient manner. Wouldn’t it be nice if you had the same system? Make the attempt to avoid trial and error and learn how to be a wanted commodity, not just another employee.

Edmond Lau illustrates his points with interviews of top engineers at the best tech companies. He tries to get them to answer his question: How do engineers become the best in their field? What tips should every engineer know in order to advance himself.

These stories and anecdotes were collected from engineering VPs, directors, managers, and experienced leaders at modern software companies. Some of these companies include Google, Facebook, Twitter and Linkedin. Other up and coming companies include Dropbox, Square, Box, Airbnb, and Etsy. A handful of startups, like Reddit, Stripe, Instagram, and Lyft, were also included.

From their stories, there are some lessons that readers can follow to reach success.

This book teaches readers to

Prioritize the right projects and tasks to increase your impact.
Earn more leeway from your peers and managers on your projects.
Spend less time maintaining and fixing software and more time building and shipping new features.
Produce more accurate software estimates.
Validate your ideas cheaply to reduce wasted work.
Navigate organizational and people-related bottlenecks.
Find the appropriate level of code reviews, testing, abstraction, and technical debt to balance speed and quality.
Shorten your debugging workflow to increase your iteration speed.
Use metrics to quantify your impact and consistently make progress.

The Pragmatic Programmer: From Journeyman to Master (Andrew Hunt)

Learn how to grow and develop your career by adding responsibilities and techniques to your portfolio. If you can write flexible and adaptable code, you can work anywhere. This book is a series of lessons that stand alone and allow you to pick it up from time to time to learn something new. The author uses examples to illustrate his points. You’ll learn the best practices for software development while sharpening your mental tools to use to help you rise in the field.

The Pragmatic Programmer teaches readers to:

learn how to Fight software rot.
Avoid the trap of duplicating knowledge.
Write a flexible, dynamic, and adaptable code.
Avoid programming by coincidence.
Bullet-proof your code with contracts, assertions, and exceptions.
Capture real requirements.
Test ruthlessly and effectively.
Delight your users.
Build teams of pragmatic programmers.
Make your developments more precise with automation.

Agile Shortfalls and What They Mean for Developers

Yuelin Wen — Wed, 18 Mar 2020 23:38:59 +0000

What is the best software development methodology to use? This question is the topic of hot debate during the project implementation stage. However, what you choose depends on many specific factors. To organize the flow of a software development project, there are several methodologies to use.

Two of the most widely used software development methodologies are Waterfall and Agile. While there are many devoted Waterfall fans out there, Agile has gained its own dedicated following. But Agile doesn’t work for every software project, so keep reading to find out why.

How Agile Was Born

Agile is newer than the traditional Waterfall method. Unlike Waterfall, Agile is a type of Rapid Application Development or RAD often paired with Scrum. Several years prior to 2001, people were looking for new ways to replace this traditionally-established, unresponsive framework. Several approaches of a new methodology had been invented in the 12 to 18 months before 2001, when Agile debuted. They included the following:

Extreme Programming
Crystal Methodologies
Scrum
Adaptive Software Development
Feature-Driven Development
Dynamic Systems Development Methodology

The hope was to add “agility” to the development process by breaking it up into smaller sections. The goal was to help get new products to the market quickly while gathering consumer feedback during development. Also, it made it easier to try out new features and functionalities before completing the final product.

On February 13, 2001, at The Lodge at Snowbird Ski Resort in the Wasatch Mountains of Utah, a group of 17 people proposed the principles of Agile software development. Agile methodology, instead of specific ways of doing things, is a set of rules to follow for a flexible, efficient, people-centric software development process.

Reactive Vs. Predictive: A Comparison of Agile and Waterfall Methodologies

First, Let’s talk about the Waterfall method. The normal routine of a Waterfall project has conventionally five stages. It usually goes from planning, building, testing, reviewing and, finally, into deploying. What is important in a Waterfall project is that the team cannot proceed until the current stage is over.

Also, like a Waterfall, the order is usually from planning to deploying, with no skipping of any steps in the middle. There is no way for anyone to start coding until planning is over, then no one is able to start testing until programming is fully complete. Moreover, if things don’t work out, the current status gets bounced back to the previous stage.

For example, if there is a problem about a certain function that is impossible to build, the whole team goes back to the planning stage where everyone has to stop everything and return to the conference room.

Most of the steps in Waterfall project management can take a substantial amount of time. Also, the discovery of any problems at each step can send the project back to base 1. If there are a variety of problems, the project must be repeated over and over until it runs out of money.

Worse yet, by the time the project is finally complete, the end product can be a total misunderstanding of the product owner’s needs. Or, even worse it is rendered useless or obsolete because of changes in technology.

The Waterfall method represents a plan-driven approach. That means it attempts to map out the whole detailed process of the software development in the planning phase. The goal is to make software development more predictable and efficient. Therefore, when changes happen, the Waterfall method won’t be able to react to them without starting the project all over again.

In this increasingly fast-paced world, the Waterfall method is becoming less and less effective in building software that meets the high demand. The Agile methodology, with its more reactive approach instead of predictive, is set out to thrive on the changes necessary to meet the needs in this ever-evolving world.

Scrum Vs. Agile Vs. Agile Scrum

We often hear people talking about “Agile” and “Scrum” as if they were two different things. However, we can also hear the term “Agile Scrum” used in many places. So, are they the same or are they different?

Agile is a framework that aims to foster individual interaction in the software development process and create customer collaboration. Also, it can quickly respond to changes and generate working software. There are 12 principles in the Agile Manifesto that explain in greater details the goals, the mindset and the requirements to practice agile software development.

According to the Agile Manifesto, Agile is just a framework without any specification on how to proceed with software development. However, under the umbrella of the Agile philosophy of software development, Scrum is one of the most common approaches to practice the Agile methodology.

Scrum was developed between the ‘80s and ‘90s with a focus on the management process of software development. In Scrum, the project is divided into manageable chunks, known as Sprints. Sprints are estimated to take under 30 days to be developed into the ship-ready stage. There is also a daily Scrum and burndown chart to ensure communication and monitor progress.

Agile Scrum Software Development: Common Roles, Steps and Procedures

The two specific roles in Agile Scrum software development are as follows:

Product Owner: While this is obviously the person who owns the product, it also represents the user and customer of the end product. They are also the person who sets the direction and identifies the right features to release.
Scrum Master or Project Manager: The Scrum master sets up the procedure, arranges meetings and makes sure the development process goes smoothly.

The rest of the team usually consists of executives, developers, testers and customers.

Understanding the Agile Scrum Software Development Process

This is the Agile Scrum software development process in just five steps:

Creating the User Backlog: The user backlog is a collection of features,known as user stories in Scrum because the features are written from the end user’s perspective, that will make the software great.
Identifying Which User Stories to Release: The product owners choose which user stories in the user backlog will be included in the product release.
Prioritizing User Stories and Estimate the Workload: The chosen user stories are prioritized and given an estimated workload.
Developing Sprints: User stories are combined to form a sprint. Sprints are short duration milestones that normally take under 30 days to achieve. The end product of each sprint must be a ship-ready deliverable. In each sprint, there will be four stages: plan stage, build stage, test stage and review stage.
Monitoring progress. The Scrum master will have to monitor the progress to see if the team is on the right track. That way, they’ll know if they will finish the project on time. Other ways to keep the team on track and in the know include:

Burndown Chart: A burndown Chart created at the beginning of the project lists the remaining workload. Workloads on the burndown chart should head toward zero, although some fluctuations can occur in between each day.
Daily Scrum: Daily stand up meetings for team members to report their progress, identify problems, and communicate with each other.

The Disadvantages of the Agile Methodology

Agile methodology came about because there were many things that the Waterfall methodology couldn't do. However, that doesn’t make Agile the best at everything. Here are some of the disadvantages to this methodology:

More Dedication and Time Investment

While focusing more on people and the interactions between people, Agile needs clients to commit more time and energy into the project. Clients can’t just show up once at a specific time to provide their plan and then wait for the project to be complete.

Because the project development becomes an ongoing process for the client as well, they will have to commit to giving up a portion of their time each month, sometimes each week or even each day, to the project.

Too Much Focus on the Details

Since what makes Agile efficient is short planning and quick reaction to change, sometimes the project may end up with a ton of features without a clear connection between them. Moreover, some functions are more suited for seeing the big picture in the future, which makes them shine less in the user review phase. This results in a shortsighted project. Also, the bigger projects may be too large for chopping into smaller, more manageable milestones.

A New Concept to Some Clients

Before Agile, there was already a solid-structured software development process that people had been using for years. Agile is relatively new, so it may take more effort from the client side, as well. So, it shouldn’t come as a surprise when some clients need some more help in understanding their role in an Agile development project.

Less Predictability and Documentation

Compared to the Waterfall method, which is more plan-oriented or document-oriented, Agile tends to react when a problem presents itself, rather than predicting the problem. This can leave projects with the Agile approach with much less documentation before the project even starts.

Some people argue that the key part of documentation in Agile software development is its source code. This property of Agile software development can be much harder for new members to understand. Furthermore, the team can sometimes fall off track because of insufficient details for direction before the project begins.

Tough to Maintain a High Level of Collaboration

Because Agile separates a project into blocks and assigns tasks to different individuals at the same time,collaboration can be difficult. Frequent updates and daily Scrums are crucial to the entire team.

While some still turn to the traditional Waterfall approach, Agile is a viable alternative, thanks to the flexibility at every stage of the project. The Agile Scrum development process works well, but it doesn’t work for every single project. Huge, complicated ones may be too much to break up into manageable Sprints.

However, the fixed, “all or nothing” approach of Waterfall can make it too cumbersome and time-consuming for a successful project finish. The decision on which to use often is dependent on the size of the project, as well as how much involvement the product owner wants to commit.

Agile and Waterfall: A Quick Comparison Study

Here is a quick comparison of Agile and Waterfall software development methodologies:

Waterfall works best for projects where the requirements are well-defined and have no expected changes. Agile works better when there is a bigger chance of frequent project adjustments.
Waterfall is simple to manage yet rigid because of its sequential approach. Agile is more flexible, yet not as defined since it allows changes at any stage. To mitigate that, teams should create a clear roadmap for delivering the finished product to avoid uncertainty during each phase of the project.
In Agile, the project description and each specific step of the project can change often. In Waterfall, they are definite from the start. The requirement of each step must be fulfilled before moving on to the next.
With Waterfall, the product owner only has to be involved in the beginning. But, with Agile, they must approve each phase of the project before the development team can move on to the next.

Conclusion

Will you use Agile, Agile Scrum or Waterfall for your next software development project? While it may depend on your team’s preferences, it also depends on the specifics of the project, such as the size and scope. Be sure to consider all factors in order to make the best choice.

16 Amazing Python libraries you should be using now (2020 updated)

Yuelin Wen — Tue, 14 Jan 2020 15:23:15 +0000

In this article we will get familiar with several amazing Python libraries being used by the best software teams. With the exception of HTTPX (which is in beta), the libraries listed below are being actively developed & maintained and are backed by a strong community.

HTTPX

HTTPX was developed by Tom Christine, who is a software engineer specialized in API design and development.

The async paradigm is increasingly becoming more common in high-performance modern applications, but when you are using the async paradigm, the Requests library cannot do its job very well.

Therefore, HTTPX comes to solve this problem. HTTPX is an asynchronous HTTP client based on the well-established usability of Requests and gives you HTTP/2 and HTTP/1.1 support. It gives you an API compatible with the Requests library as much as possible, and it has a tight control on the timeouts. HTTPX can also call directly into a Python web application using the ASGI protocol and it is fully type annotated. The library is also equipped with all the standard features of Requests, such as International Domains and URLs, Keep-Alive & Connection Pooling, and Sessions with Cookie Persistence, etc.

Moreover, you can use either asyncio or trio for HTTPX and apply this in high-performance async web frameworks. As a result HTTPX is able to handle a large numbers of requests.

Arrow

As many Python developers know, one way to work with date & time objects is by using the uncompleted date, time, and timezone functions in the Python's standard library and some other low-level modules. However, they don't have high performance and good usability; for example, they have too many modules and types, which is hard for users to remember and distinguish. The conversions between timezones and timestamps are also long-winded.

Luckily, a sensible and human-friendly Python Library called Arrow can help users create, manipulate, format, and convert dates. It mainly aims to reduce your imports and code for dealing with dates and times.

Arrow supports Python 2.7, 3.5, 3.6, 3.7 and 3.8, and broadly supports ISO 8601. Of course, it can do timezone conversion, and the timestamp is also a property in it.

Some features of Arrow can give you a clue of its capability. Arrow has a drop-in replacement for DateTime, and is Timezone-aware by default. Users will get easy-to-use creation options for many general input scenarios. The shift method is beneficial for relative offsets, such as offsets for weeks. It's possible to automatically format and parse strings, which saves a lot of time. One more attractive feature is that Arrow can create periods, ranges, floors, and ceilings for time frames, ranging from microseconds to years.

Python Fire

Python Fire can automatically generate CLIs for any projects. The library makes the process of creating CLIs super simple. You only need to write the functionality at the command line as a function, module, or class, and once you call Fire, the CLI that you desire is ready for you.

You probably want to test early on during the process of writing a Python library. Without Python Fire, you have to write a primary test method to run the functionality you desire to achieve. Although you can do the test in this way, you need to change the primary method whenever you have new features you want to test, updating the primary method again and again, which is time-consuming and annoying. Using Fire you don't need to change your primary test method continuously when you test your command line.

It is usually not super quick to understand a function by looking at its code, particularly when the function was written by someone else. A better way is to call Fire on the module. This feature allows you to easily inspect all module functions/methods.

Moreover, Fire can let you transit directly between Bash and Python so that you can use the unix tools at the same time.

Starlette

Starlette is a lightweight ASGI framework or toolkit for building high-performance asyncio services.

This production-ready library has many features, including support for WebSocket and GraphQL. Starlette can do in-process background tasks, CORS, GZip, Static Files and Streaming responses. All of these features have extensive test coverage and a code base that is 100% type annotated with zero hard dependencies.

Starlette is meant to be used as a complete framework, or as an ASGI toolkit, providing users with the flexibility to apply any of the components independently. Moreover, reusable components can be shared between any ASGI framework, creating an ecosystem of shared middleware and mountable applications.

Mypy

Mypy is an optimal static type checker for Python 3 and Python 2.7, and it is similar to a static analyzer, or a lint-like tool. By adding some typing annotations when you write your program, Mypy can help you type check your code and look for general bugs. Those annotations you leave in your code guide Mypy to do its job without interfering with your application because the annotations are viewed as comments that have no effects in the execution of your code.

Mypy gives developers the flexibility to decide the workflow. The purpose of Mypy is to combine the advantages of dynamic typing and static typing in the applications. Therefore, users can use dynamic typing as a backup when static typing doesn't work, such as for legacy code. When you run Mypy in your program like a linter, the errors will be reported in a compiler-style format. Mypy provides programmers with a robust and consistent check for a project and its dependencies.

Another advantage of Mypy is its learning curve, which is minimal. Most new users of Mypy should be able to annotate the code correctly for the first time. Moreover, the Mypy cheat sheet is a perfect start. One more advantage of Mypy is that It has a much lower false-positive rate compared to most static analyzers.

FastAPI

FastAPI is a high-performance web framework for API development. FastAPI is based on standard Python type hints for Python 3.6+.

FastAPI comes with many interesting features, perhaps the most important one is speed since it is one of the fastest Python frameworks available. The speed of coding is also 200% faster compared to other frameworks.

And if that wasn't enough, FastAPI can help to keep a low bug rate and can even reduce human related errors by almost 40%. The framework is easy to learn and has an interactive documentation.

FastAPI is based on open standards, for instance OpenAPI. It also comes with declarations of path operations, parameters, body requests, and security. Features for automatic client code generation in different languages are also available.

Immutables

Immutables is an immutable mapping type for Python. A Hash Array Mapped Trie (HAMT) used in Clojure, Scala, Haskell, and other functional languages.

Immutable mappings based on HAMT have O(log N) performance for both set() and get() operations, which is essentially O(1) for relatively small mappings.

Expiring Dict

Expiring Dict is a very handy Python caching library. It provides a dictionary with ordering and auto-expiring values for caching purposes. Dictionary elements have a TTL (max age) and max length, which are checked on each access.

VCR.py

Have you ever used Ruby's VCR library? It records your test suite's HTTP interactions and replays them during future test runs for fast, deterministic and accurate tests.

VCR.py is similar to Ruby's VCR library. It makes the tests with HTTP requests more straightforward and quicker. If your code is in a VCR.py context manager or decorated function, when you run it VCR.py will record all the HTTP interactions during your tests. Then VCR.py will serialize and write the HTTP interactions to a cassette, which is a flat file. When you execute the recorded code it will replay the serialized requests and responses from the cassette file.

This process has many benefits because the requests will not generate any HTTP traffic. Therefore, VCR.py can do its job offline, help to generate deterministic tests as well as significantly increase their execution speed.

When you make changes to your tests all you need to do is to delete the cassette file. Then when you re-run the code, VCR.py will record again the HTTP interactions and generate a new cassette.

Transformers

Transformers (former pytorch-transformers and pytorch-pretrained-bert) provides Natural Language Understanding (NLU) and Natural Language Generation (NLG) with more than 32 prepared models in over 100 languages and deep interoperability between TensorFlow 2.0 and PyTorch. Those architectures generally are BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet, and others.

Transformers is an easy-to-use library with no barriers to get familiar with. It is even more powerful and concise than Keras, with the added benefit that it has extraordinary high performance when dealing with NLU and NLG projects.

The goal of Transformers is to become the state-of-the-art NLP for everyone. No matter if you are a deep learning researcher, hands-on practitioner, or AI/ML/NLP teacher and educator, you can get a lot of support from Transformers. The library is very efficient as it reduces the compute costs. For example, you can share your trained models with your colleagues so that they do not need to spend extra efforts retraining models. It comes with more than 30 preset models, and over 100 languages which can save a lot of time on computing and thus help to reduce production costs.

Lastly, Transformers can use the most suitable framework for each part of your applications over time. Specifically, Transformers can use only three lines of code to train a state-of-the-art system for you. The deep interoperability between TensorFlow 2.0 and PyTorch models is an added benefit.

Modin

If you are a data scientist you are probably familiar with the Pandas library. Modin can scale your Pandas workflows by changing only one line of code. Sounds great, right?

Before proving more details on Modin we first need to introduce Ray, which is the key to how Modin works. Ray is a fast and accessible framework for building and running distributed applications. Ray is packaged with Tune, RLlib, and Distributed Training.

The relationship between Modin and Ray is that Modin uses Ray to speed up the Pandas notebook, scripts, and libraries in an effortless way. A weakness of other distributed DataFrame libraries is the regular conflicts among Pandas code. Modin can solve this problem by integrating the Pandas code seamlessly.

Using Modin, you do not need to know details about the cores of your system and the distribution of the data. You do not need to abandon the Pandas notebooks you previously used to take advantage of the enormous acceleration provided by Modin.

Moreover, the modin.pandas is an extremely lightweight parallel DataFrame. After you install Modin, there is no need to stop using the previous Pandas API because Modin can do transparent distribution of the data and computation.

Sometimes, when facing different data sizes it may be difficult to scale 1KB DataFrames to 1TB, there is often a significant overhead. With Modin, you can directly get DataFrames at both 1KB and 1TB.

Dash

If you are building web apps with complex or large datasets, Dash might be for you. Dash is very suitable for visualizing data and it also provides apps with customized user interfaces in pure Python.

Dash can even help you to build user interfaces with Python code in a short amount of time. Dash replaces many of the tools and technologies used for building an interactive web application and it does its job in a simple way.

Detectron 2

Detectron is an object detection platform. It is one of the open-source projects adopted by the Facebook AI Research group.

Detectron 2 is the second generation of this library with many performance enhancements. The library is flexible and extensible and makes training on GPU servers a very quick process. It also comes with state-of-the-art object detection algorithms, allowing developers to do advanced research without the whole complete dataset.

Detectron 2 was rewritten from Scratch in PyTorch, which is a great tool for deep learning. The vast and active community behind PyTorch is an added benefit for Detectron2 users.

With Detectron2, users can insert their customized code into the object detection system as they see fit. Under this situation, hundreds of lines of code can successfully develop a new research project, and the core Detectron2 and the brand-new research achievement can be divided clearly. Moreover, Detectron2 also supports semantic segmentation and panoptic segmentation.

The reason why Detectron2 is faster than the original version is due to moving the entire training pipeline to GPU. Distributing the training to different GPU servers makes the scaling process of large data sets much easier.

Streamlit

Streamlit is a library that can provide you with the fastest way to build custom Machine Learning tools.

The library embraces Python scripting resulting in clean code and fast prototyping. Each change in the Python code directly reruns the code from top to bottom. The cache primitive in Streamlit is consistent, immutable-by-default so users can reuse the information effortlessly.

Imbalanced-learn

Imbalanced-learn offers many re-sampling techniques for machine learning projects. This python library is widely used in datasets to show a robust between-class imbalance.

Imbalanced-learn is compatible with scikit-learn. Scikit-learn is a simple and efficient tool for predictive data analysis. It is built on NumPy, SciPy,and matplotlib. It is open source and very reusable. The library is available for Python 3.6+.

PyTorch

PyTorch offers rapid prototyping for dynamic neural networks and strong GPU support.

The compositions of PyTorch are torch, torch.autograd, torch.jit, torch.nn, torch.multiprocessing, and torch.utils. Component torch is a NumPy like Tensor library, which has the strong GPU support. Torch.autograd is a tape-based automatic differentiation library that is accessible to all various Tensor operations in torch. Torch.jit can work as a compilation stack to create serializable and ideal models. Troch.nn is aggregated with autograd as a neural network aiming for maximum flexibility. Torch.multiprocessing is useful for data loading and Hogwild training. Lastly, torch.utils has a data loader and other utility functions for convenience.