DEV Community: NeuroSYS

Minimum Viable Product – the concept, examples, and the rationale

NeuroSYS — Tue, 11 May 2021 10:25:09 +0000

Make your product stand… up!

Unlikely as it might seem, episode one of the hit TV show “Marvelous Mrs. Maisel” and Joaquin Phoenix’s “Joker” share a similar scene. In both, characters walk out on stage as wannabe stand-uppers and deliver lackluster performances only to get booed and eviscerated by the audience. And while one can’t help but feel sorry for them, there is a lot to be learned from their failures.

But contrary to appearances, these failures consist not in the poor quality of the performances, but rather in the inability to learn and improve based on received feedback. In other words, the characters in those scenes lack grit and thick skin. They give up at the first sign of trouble. But “going out there” is definitely a step in the right direction.

In fact, many of the best stand-uppers in business today developed their most successful material in exactly the same way – testing it out through trial and error in small clubs. But instead of quitting if things got ugly, they fed off their audiences’ reactions and honed their work until everything was right and could be reproduced over and over again.

This pragmatic and efficient approach, which might alternately be called “the way of the recursive stand-upper” (or “boomerang-do” ), has also found its way to software development. And for good reason.

Why? Consider the following: if you plan to put a lot of time and effort in creating a product or service, you need to know that it will find customers. Assuming you have the resources, you could go the old-school way: pay for focus group research, big data analysis, or other costly methods of ensuring the return on your investment. But most of the time, neither fledgling stand-uppers nor beginner developers, for that matter, have these kinds of resources, and yet some of them manage to hit it off. Is it just pure luck? Not really. In many cases, it is the result of a very specific strategy.

Enter “Minimum Viable Product”

As fond of fancy names as software development is, it coined the term “Minimum Viable Product”, which is another way of calling that first draft of a stand-up performance you deliver to a small group of listeners, paying keen attention to their response. In other words, it is the smallest possible version of the final product you can show to its intended users so they can test it and give you ideas on how to make it better. It is not the same as an outline, a pitch, or a summary. It is the real deal. It must be fully functional in its own, minimal way.

Examples

Examples of minimum viable products abound in the history of software development.

Amazon started off merely as an online catalog of books that visitors could order with home delivery. There weren’t any other departments, there weren’t even any warehouses. If someone ordered a book, Amazon had to buy it from a bookstore first and only then sent it to the customer.

Facebook was initially a service where you could set up a rudimentary profile page and connect it with other users.

When it launched, Foursquare was a simple website where you could check-in at a given place. That was it.

Notice, however, what all of these MVPs have in common. Each has at least one major feature that works. This is what makes them different from mere ideas. They represent the smallest possible realization of an idea, and with that, a promise of something greater, which piques people’s interest and whets their appetite for more.

And there was more. Each of the MVPs I mentioned grew with time and evolved to better suit its users’ needs, ending up as the mammoths we know today. But while it was possible largely because of the MVP approach, it is important to understand that a Minimum Viable Product is not a guarantee of success. Actually, one might argue that one of its main roles is to protect against a likely failure.

The reasoning behind MVP

The vast majority of all business endeavors end up unsuccessful. When odds are against you, it is best to bet low, and Minimum Viable Product allows you to do just that – it gracefully manages the inevitable and gut-wrenching risk factor of every new business venture, because there isn’t much to lose with it but quite a lot to gain.

However, that alone would not be enough to make all the fuss about MVP, would it? Luckily, there’s more. MVP also lets you perfect your product or service in real-time. Just as the aspiring stand-upper in front of an audience could take mental notes on which of their jokes work and which don’t, software developers can observe their users’ reactions to their product. These reactions could come in a variety of forms. They can be as obvious as comments, emails, or messages, but also something much less direct, like the sheer number of people that sign-up for your product or download a beta version of your app. If that number exceeds your expectations, you’re probably on the right track.

But even if your users don’t care enough to write, nothing is stopping you from reaching out to them! What’s more, not only can you contact them individually, which would be time-consuming, but you can also create a Facebook group or a forum. You could even run a crowd-funding campaign to put your users in one place, encourage them to share their insights and, quite possibly… some of their incomes.

Get ready for an adventure

As you can see, MVP lets you embark on an exciting journey of discovery that will either take you to the top or prevent from falling down a precipice. So don’t be shy! Buckle up and take the first step to success.

One Step Ahead Towards Fully Automated AI-powered Microbiological Analysis

NeuroSYS — Tue, 27 Apr 2021 13:07:08 +0000

We’re happy to announce that, with support from the National Center for Research and Development, NeuroSYS has launched a project on automating the classification and counting of bacterial colonies for microbiological analysis.

Project’s goals

During the next 2 years, the dedicated team will be working on developing algorithms that use deep learning to classify bacteria and identify the number of colonies based on the image of bacterial cultures grown on a Petri dish.

Such algorithms will form the foundation of the final outcome of this project – an AI-powered software for automated analysis of bacterial cultures in microbiological laboratories.

The final product is intended for different variants of using: i) in standard trim (for bacterial analysis based on ready-made images, ii) connected to a simple image acquisition set consisting from a camera mounted on a tripod (these sets for laboratory use are available on the market, such as the ones produced by Carl Zeiss), iii) in conjunction with existing advanced systems for automating microbiological laboratory environment (i.e. MicroTechniX).

Real-world applications

Time and accuracy are of the essence for microbiological analysis performed in industrial labs – where microbiology testing is a legally required part of a production process, and its automatisation reduces operational costs.

As of today, microbiological analysis is semi-automated. Some of its stages, such as sample preparation and labelling, incubator, image acquisition can be performed with the help of robotic devices. At the same time, the results’ evaluation is mainly done manually and requires tedious repetitive work performed by a team of highly educated employees.

The software that we’re working on will help speed up the process, reduce associated costs, and avoid potential human errors when the analysis is performed manually.

The areas of application include but not limited to healthcare, pharmaceutical, food, cosmetics, and veterinary industries.

How does it work?

The solution will be based on the algorithms of deep learning. In recent years they have dominated the majority of image classification and object detection challenges achieving impressive results when applied to visual data. More precisely, a combination of Convolutional Neural Networks (CNN) with other types of neural networks and classical algorithms will be used to detect and classify different species of bacteria.`

In cooperation with microbiology experts, we are preparing a database that will contain carefully described photos of Petri dishes with specific species of bacteria. “Data is of great importance for the learning process of neural networks. That’s why we plan to generate around 20 000 training images for the project.” – explains Tomasz Golan, R&D Director of the project.

Additionally, spectral analysis and three-dimensional vertex mapping (laser scanning) will be performed in order to obtain more information about the samples.

“We believe that deep learning is the technology that will lie a foundation for a future automated microbiological analysis and power up the capabilities of industrial laboratories, ultimately speeding the delivery and enhancing the accuracy of testing results.” – shares his prognosis with us Tomasz Kowalczyk, CEO at NeuroSYS.

Tech stack

Python, Pytorch, TensorFlow, OpenCV

MAGDA – our open-source solution for spaghetti code

NeuroSYS — Wed, 14 Apr 2021 12:49:52 +0000

Introduction

We would like to introduce you to our latest open-source library: MAGDA. The name is an abbreviation for “Modular Asynchronous Graphs with Directed and Acyclic edges”, which fully describes the idea behind it. The library enables building modular data pipelines with asynchronous processing in e.g. machine learning and data science projects. It is dedicated for Python projects and is available on the NeuroSYS GitHub, as well as on the PyPI repository. It aids our R&D teams not only by introducing some abstraction (classes and functions) but also by imposing an architectural pattern onto the project.

Example of coffee brewing pipeline with MAGDA

How does it get rid of spaghetti code?

As described above, MAGDA is composed of a few features:

1. Modular – code should be divided into small logical blocks (modules) with explicit input and output. The module could be a simple filter, database connector or a wrapper on a huge deep learning model. Just remember: one module – one role.
2. Asynchronous – the library is based on asyncio and ray, which allows it to run modules simultaneously. This gives us a simple optimization out of the box.
3. Graphs – modules are joined together into one connected pipeline/stream. During the design stage, we can think of modules as graph nodes and focus solely on their role and how they connect with each other.
4. Directed – the modules’ dependencies (and graph’s connections) are asymmetric. Since the graph always “flows” in the same direction, we can easily determine the ancestors and predecessors of a module. Therefore, we can clearly point out where the pipeline begins and ends.
5. Acyclic – each module is always processed just once during every run. This means that there is no path in the graph (modules’ dependencies) which starts and ends at the same module.

By combining all of these features, MAGDA creates a concrete project template, where each part of the project is enclosed into a module with a specific input and output. Each module’s behavior can also be modified by providing custom, module-specific parameters.

Application flow is created by joining modules into a pipeline, where each part of the pipeline can be replaced by another module with a corresponding interface. Finally, the whole pipeline can be easily written to and automatically loaded from a single YAML file.

When correctly applied, you obtain a project with clearly defined boundaries and interfaces. When modifying a module, you rely only on information provided by the accepted interfaces and parameters regardless of the rest of the system – similar to the “inversion of control” design pattern.

Use-cases

The library can be used in every Python project, which can be described as an instruction with a set of well-defined steps. Our R&D team is making use of MAGDA in various services: from small solutions with only a few modules to a complete Question-Answering pipeline. The most valuable is the easiness of replacing any part of the pipeline without concerning about the rest of the system. Creating a modular application is especially important when performing reliable and repeatable experiments, where only certain parts or parameters are modified. Apart from that, you can also gain from asynchronous processing of several subparts at the same time.

Summary

Since MAGDA is our brand new project (current version: 0.1), some features might still be missing. Feel free to create an issue, share a feature request, or post a question and contribute!

NeuroSYS GitHub: https://github.com/NeuroSYS-pl/magda
Documentation: https://github.com/NeuroSYS-pl/magda/wiki
PyPI Repository: https://pypi.org/project/magda/

Dream teams need trust and transparency (5 ways to make it work)

NeuroSYS — Tue, 09 Mar 2021 12:55:44 +0000

Trust as well as open and honest communication is key to any project’s success. That is a simple idea which is hard to disagree with. But when someone starts talking about transparency in projects, it’s often considered to be such a vague substance far from exact sciences, that the topic doesn’t get due attention.

“Project transparency: definition

By transparency in project management, we understand the culture where all the information is visible and easily accessible, and communication among the stakeholders is open and honest.”

At NeuroSYS we think it’s important not only to talk about transparency in project management but strive to achieve it. And to make this conversation more practical, we want to share some specific methods and tools we use for making our projects more transparent.

1. Solid client onboarding

As a rule of thumb, the best projects come out as a result of a trustworthy partnership with a client. That’s why we take all the measures to help the client become a part of the team from day zero. And ensuring project transparency here is a key.

We secure smooth cooperation by building and strengthening the bond within the team from the very beginning. Our onboarding is engaging, honest and personal (we always try to meet in person and introduce the whole dedicated team offline whenever possible; if not – cameras during our conversations are always on).

We don’t impose our rules during onboarding. Instead, we offer to set up the way of cooperation, that would be convenient for both sides.

The onboarding aims to make it clear: what is expected from every party and how they can participate; how to track the progress of the project; what unforeseen circumstances there can be and how to deal with them; etc. None of the questions or possible scenarios should stay undiscussed after onboarding.

Last but not least, it’s also crucial to define during onboarding who will be the main person responsible for communication and making decisions from the client’s side. Having communication flow defined and under control will ensure that nothing gets lost or stays out of attention.

2. Enhancing software transparency

Let’s be realists: no matter how attached you are to the software projects you’re working on, one day you’ll most probably transfer it to someone else. And the least what you want to happen is your code becoming a black box or mysterious inscriptions of some ancient tribe in the hands of a newly joined IT team or a team member.

To make sure that the software we create is easy to take over, we accompany it with clear and informative documentation, as well as generous wiki descriptions.

We practice the habit of taking care of the documentation for many years now and as a result, the code we create can be easily taken over at any moment.

3. Being always true and loyal to agile

Transparency is one of the main pillars of agility, and agile frameworks are packed with procedures and habits that will help you improve it.

Following agile ceremonies and making sure that all the stakeholders attend them regularly is a great way to make a project glass-clear. Sprint planning makes it explicit what is expected from the team, when and why. Daily meetings help to keep everyone up-to-date and react fast if something goes wrong. And retros can be a great source of lessons learnt for the whole organisation, not only the team involved.

4. Using tools for improving transparency in a project

There are plenty of tools and instruments on the market that can help you visualize information better, make communication more efficient, and thus improve the transparency of your project.

But in our humble opinion, it’s not so important what exact tool (or set of tools) you use. What matters is that people don’t get lost among too many tools and that they feel comfortable using them. We also appreciate the possibility to integrate tools between each other and apply customizations for creating one centralized flow of information.

Here are some of the tools we are using:

YouTrack by JetBrains: it’s an issue tracker for developers with a broad range of functionalities, allowing teams to visualise the agile board, organize user stories, do the planning, time tracking, reporting, and track bugs. Our clients get full access to all the information in YouTrack, so it easier for them to follow the status of the project and be involved.

Mattermost / Slack

These are the tools that great for communication overall. Our teams use dedicated channels there which are also integrated with the YouTrack, so they can get notifications when something happens that requires their attention (f.e.: a pull request is merged, or pull requests waiting for review, etc.).

Confluence

Confluence is another team collaboration tool that allows to upload, manage, share the project related documentation and work on it together. We like it for its wiki section and the whole documentary format, that helps us to keep the most important information about a project in one place.

A more detailed overview of IT project management tools you can find here.

5. Transparent culture

Trust and transparency depend on the culture inside the company, or how people feel about sharing information or asking. And we believe there are also specific things you can do to improve it:

Flat organisational structure
Between our CEO and a junior, there are just a few steps in the management hierarchy. This means that everyone feels comfortable to approach the CEO or other higher-ranked colleagues regarding any matter, This makes communication less entangled. Also, it helps people of any position to influence the company decisions more easily.

No shaming for making mistakes
People tend to hide some information if they believe they can be punished or criticised for it. That’s why it’s important to reassure them in the opposite. Here agility helps again. Thanks to short sprints, it’s impossible to make a mistake that is hard to fix. People aren’t afraid to make mistakes and are more eager to communicate openly.

Adversarial Attacks For Fooling Deep Neural Networks

NeuroSYS — Thu, 25 Feb 2021 09:16:42 +0000

Even though deep learning performance advanced greatly over recent years, its vulnerability remains a cause for concern. Learn how neural networks can be fooled in this post and stay tuned for more tips on improving your neural networks’ safety .

In 2014 Szegedy et al. [1] discovered that many machine learning models, including achieving state-of-the-art performance neural networks , are highly vulnerable to carefully crafted adversarial attacks, i.e. attacks based on minor modifications of the input. In the next years, many algorithms have been proposed to generate samples that can fool neural networks, so-called “adversarial examples”. In today’s post, we’ll discuss 3 different methods of doing so.

Fast gradient sign method

Goodfellow et al. [2] proposed to add a carefully crafted small noise vector in order to fool the neural network. Below you can see an example. We start with the image correctly classified as an African elephant. After adding an imperceptible noise, the neural network recognized an image as a sea snake.

The illustration of the fast gradient sign method to fool a neural network. After adding imperceptibly small noise to the image presenting an African elephant the neural network recognizes it as a sea snake with 81% confidence (Photo by Andrew Rice on Unsplash).

Jacobian-based Saliency Map Attack

Papernot et. al [3] introduced a method called Jacobian-based Saliency Map Attack (JSMA), which tries to alter as few pixels as possible. To achieve this it uses a saliency map, which shows an impact each pixel has on the classification result. A large value means, that changing this pixel will have a significant impact on the outcome of the classification. The JSMA algorithm picks the most important pixel based on a saliency map and changes it to increase the likelihood of the attack target class. The process is repeated until the network is fooled or the maximal number of modified pixels is reached (in that case the attack was unsuccessful). Here is an example:

The illustration of the JSMA algorithm. On the left side, the original image correctly classified as jellyfish is presented. In the middle, you can see the image after the attack, recognized as a sea lion. On the right side, there is an amplified difference between the two images. 146 pixels were changed in order to perform the attack, which constitutes less than 1% of all pixels (Photo by Patrick Brinksma on Unsplash).

One pixel attack

The last method that will be described in this article requires to only change one pixel in the input image in order to fool the neural network [4]. It is based on a differential evolution algorithm. It works as follows: multiple random changes are made to the image and it is checked how these changes influence the class confidence of the network. The goal is to decrease the confidence of the network in the appropriate class. Worst scenarios (little or no change in confidence) are ignored and the search is continued around most promising candidates. As a result, the confidence in the proper class will be very low and a different class will take over.

The illustration of the one pixel attack method introduced in [4]. On the left side, the original image is presented, correctly recognized as an African elephant. On the right side, the image with one pixel changed can be seen, which results in the wrong classification result (Photo by AJ Robbie on Unsplash).

To sum up

It has been proven that neural networks are vulnerable to adversarial attacks. This calls into question their applicability in real-world scenarios, especially in systems where assuring safety is crucial. Thus it is very important to be able to defend against such attacks . In our next post, we will present some methods to achieve this.

Literature

[1] Szegedy, Christian, et al. “Intriguing properties of neural networks.” arXiv preprint arXiv:1312.6199 (2013)

[2] Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. “Explaining and harnessing adversarial examples.” arXiv preprint arXiv:1412.6572 (2014)

[3] Papernot, Nicolas, et al. “The limitations of deep learning in adversarial settings.” 2016 IEEE European symposium on security and privacy (EuroS&P). IEEE, 2016

[4] Su, Jiawei, Danilo Vasconcellos Vargas, and Kouichi Sakurai. “One pixel attack for fooling deep neural networks.” IEEE Transactions on Evolutionary Computation 23.5 (2019): 828-841

5 Powerful Tips To Make Your Life Science IT Project Succeed

NeuroSYS — Fri, 19 Feb 2021 12:58:07 +0000

Life Science companies are rapidly developing thanks to new technological advances becoming available at record speed. Almost every stage of any product development cycle can be optimized with the help of IT or automation today.

This opens a wide pool of possibilities on one hand, but raises a number of challenges from technological and IT management point of view on the other.

Software development in Life Science must adhere to high-quality standards, constantly evolving compliance and regulations (whereas applied), rapid requirements changes, as well as rigid testing and validation practices. At the same time, it is expected that a solution is delivered as soon as possible (since competitors are also up-and-doing), that it’s stable and flexible despite all the complexity hiding behind.

Navigating these hurdles requires a special set of skills and experience from an IT team involved in a project since even minor development mistakes can lead to major harm for a business (i.e. intellectual property leakage, reputational damage, regulatory penalties etc.) and put at risk health and lives of its customers.

After years of working on complex Life Science IT projects (such as lab automation software or AI-powered solution for automated microbiological analysis) we can underline five key factors that need to be taken into account while building software for Life Science projects:

Deep understanding of business

This is a ‘must-have’ for any project we’re doing, but LifeScience has its peculiarities. Here the IT provider needs to get into scientific details, biological or medical data and existing workflows related to the project in order to understand what exactly the client wants to achieve, their context, limitations, environment etc.

Developers cannot create great software for laboratories working in isolation and having no clue how and where their code will be used. That’s why we always insist on visiting the client’s operations, observe how things work in real life, what tools and technologies are being used, accurately gather requirements to suggest the most optimal IT solution.

Moreover, the IT provider shall be familiar with all the standards of the industry and how to work with them effectively. Risk assessment (FMEA analysis), impact assessment, installation & operational qualification protocol, validation plan, functional traceability matrix – there are just a few examples of the documentation we are dealing with while working on Life Science projects.

Software transparency

It’s not enough to understand, we also strive for our solution to be easily understood by creating quality software with clear functionality and logic.

Trust is essential in Life Science. Unless people understand the principles behind Life Science software, they are unlikely to use it (or allow using it): No doctor would agree using analysing software for X-ray images without understanding how it works. No patient would sign up for personal medical treatment generated through an unknown algorithm. And no authority would allow selling medicine without a clear explanation of its discovery path, effectiveness and safety.

To avoid these situations we also prepare detailed technical documentation, precise delivery process and QA procedures description to be followed and provide proofs of compliance.

IT systems flexibility

Life Science and Pharma products are subject to rapid changes due to discoveries occurring regularly and constantly evolving regulations around them. That’s why creating flexible IT systems that are easy to alter and scale is a priority and one of the main challenges here.

This need can be addressed through:

implementing a hybrid project management system where developers follow agile methodologies (to ensure continuous delivery and flexibility of the process) while high-level management (risk, budget, documentation, etc.) is done in PRINCE or another customary for a particular company way (to avoid disruptions in processes that are already complex enough );
an early assessment of the possibility of adapting modular architecture or microservices (to makes changes technically possible);
thorough documenting and validation.

Quality Assurance for Life Science projects

Quality Assurance is, with no doubts, of the greatest importance for Life Science projects. Not only because it’s often a regulatory requirement and necessary for mitigating the risks, but also because its effective implementation can give a competitive advantage and speed up the production cycle.

IT providers must pay special attention to establishing and following functional requirements, quality standards, validation documentation, and automated validation scripts when building software for Life Science projects. They also ought to provide evidence of these procedures to be followed.

Openness to new tech and R&D pilots

Life Science, where data and accuracy is a key, benefits greatly from adopting new technologies. Artificial intelligence (AI), machine learning (ML) and deep learning (DL) already help Life Science companies to reduce the need for human control over the processes, scale operations and optimize the work behind complex biological tests to a few mouse clicks.

Being open to adopting new tech is not simply an advantage. Companies that miss this chance – trail far behind.

Defense strategies against adversarial attacks

NeuroSYS — Tue, 16 Feb 2021 11:54:31 +0000

In our previous article, we showed that neural networks are vulnerable to carefully crafted adversarial attacks, i.e. attacks based on minor modifications of the input.

While those perturbations are often invisible to a human observer or perceived as noise that does not influence the recognition, they can fool even the most successful state-of-the-art neural network based systems. Thus, it is of great importance to be able to defend against such attacks, especially in safety-critical environments.

Multiple techniques have been proposed over the last years to achieve this, including but not limited to: adversarial training, image preprocessing or transformation and feature denoising.

The most popular defense strategy is adversarial training. In this method, images are adversarially attacked with a chosen approach (for example fast gradient sign method described in our previous post) and perturbed examples are introduced to the model during the training process so that it learns to classify them correctly [1,2]. Another group of methods involves image preprocessing. Since perturbations are often perceived as noise to a human observer, several approaches based on transforming or preprocessing input images before they are fed to the neural network have been introduced [3,4].

Additionally, Xie et al. [5] proposed feature denoising as a method of defense against adversarial attacks based on the observation that perturbations in images cause noise in the feature maps.

In today’s article, we will present two state-of-the-art methods of defense against adversarial attacks.

Adversarial training

Adversarial training was first introduced by Szegedy et al. [1] and it is currently the most popular technique of defense against adversarial attacks. This method involves training a network on a mixture of adversarial and clean examples. Goodfellow et al. [6] followed them and trained a network using adversarial loss Ladv(x):

where L(x) is the loss function computed on the original image and L(xperturbed) is the loss function computed on an adversarially perturbed image (using e.g. the fast gradient sign method described in our last post) and ɑ is a hyperparameter assigning weights to the components of the adversarial loss function.

Multiple improvements have been proposed to the regular adversarial training. For example, Harini et al. [6] added an adversarial logit pairing component to the loss function, which encourages the logits (outputs from the network before applying softmax normalization function) obtained from two images (original and adversarial) to be similar. The intuition behind this idea is that both clean and perturbed versions of the same example should be assigned to the same class so big differences between their logits are penalized.

Feature denoising

Xie et al. [5] recently proposed another method of defense against adversarial attacks, namely, feature denoising. Authors showed that while adversarial perturbations are often barely perceptible in the pixel space, they cause a significant noise in the feature maps. As the image is passed through the network, the perturbation increases and not-existing activations of the feature maps arise, which leads to wrong predictions.

Below feature maps of original and perturbed images are presented:

Feature map calculated for an original image (top) and adversarially perturbed version (bottom). On the adversarial image, activations are present in regions not relevant to the prediction. Adapted from Ref. [5].

Based on this observation, the authors developed a new method of defense against adversarial attacks, namely, they added multiple blocks for feature denoising to the neural network and trained it end-to-end on adversarial examples.

The authors utilized ResNet [7] with 4 denoising blocks. Below adversarial images are presented, together with the feature maps before (left) and after (right) applying denoising:

Adversarial images are presented, together with the feature maps before (left) and after (right) applying denoising. Adapted from Ref. [5].
As can be seen, noise is substantially reduced as a result of denoising operation. Adding feature denoising during adversarial training improved the accuracy of classification on adversarially attacked images. The method won first place in the Competition of Adversarial Attacks and Defences (CAAD) 2018 [8].

To sum up

Neural networks’ vulnerability to adversarial attacks casts doubt on their usability in the real-world environment, especially in safety-critical systems, like for example autonomous cars and medicine. Thus finding an effective defense strategy is of great importance.

However, there are a few challenges that we have to face when applying the defense method. For example, weak performance on adversarial examples produced with attack approaches that haven’t been used during adversarial training is a common issue. Additionally, drop in classification accuracy on unperturbed images may happen.

Literature

[1] Szegedy, Christian, et al. “Intriguing properties of neural networks.” arXiv preprint arXiv:1312.6199 (2013)

[2] Madry, Aleksander, et al. “Towards deep learning models resistant to adversarial attacks.” arXiv preprint arXiv:1706.06083 (2017)

[3] Xie, Cihang, et al. “Mitigating adversarial effects through randomization.” arXiv preprint arXiv:1711.01991 (2017)

[4] Das, Nilaksh, et al. “Keeping the bad guys out: Protecting and vaccinating deep learning with jpeg compression.” arXiv preprint arXiv:1705.02900 (2017)

[5] Xie, Cihang, et al. “Feature denoising for improving adversarial robustness.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019

[6] Kannan, Harini, Alexey Kurakin, and Ian Goodfellow. “Adversarial logit pairing.” arXiv preprint arXiv:1803.06373 (2018)

[7] He, Kaiming, et al. “Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016

[8] http://hof.geekpwn.org/caad/en/index.html

Project co-financed from European Union funds under the European Regional Development Funds as part of the Smart Growth Operational Programme.
Project implemented as part of the National Centre for Research and Development: Fast Track.

How to start a machine learning project with an external AI company – a practical guide

NeuroSYS — Thu, 11 Feb 2021 09:04:04 +0000

Today’s topic has resulted directly from discussions with our clients and discovering their concerns. I won’t beat around the bush – machine learning projects are characterized by a high risk. Unlike software development, which is difficult but still way easier to plan, there are many uncertainties involved. Starting an ML project pretty often you don’t know if your problem can be solved by technology at all because no one has ever solved it. Thus, it belongs to the domain of research and starts with carrying out a feasibility study. By the way, if any AI company can promise you to solve a niche, untypical problem beforehand, I’d recommend being extremely cautious. Because it sounds too good to be true. There are just too many questions, predictions, and hypotheses that need to be supported by evidence.

Beyond a doubt, the game is worth the candle. If a machine learning project is successful – you can boost your business immensely, gain a great advantage over your competition, and create new, unique products. In this article, I want to show you how we launch and manage machine learning projects in order to reduce risk and costs, while still being able to deliver expected results in a planned way.

What is machine learning

Machine learning is a subfield of artificial intelligence. Its purpose is to develop algorithms so that computers learn and can improve automatically over time. They are programmed to find patterns and eventually make data-based predictions and decisions. The keyword here is ‘automatically’, which means without any further human intervention. Different types of machine learning algorithms are utilized in plenty of industries and applications, such as medical diagnosis, insurance, finance, telecommunication, and production.

Specifics of machine learning projects

Machine learning projects aim at significant changes in an organization – improving production processes, optimizing supply chains, and making better business decisions. They can be also a basis for new, groundbreaking products or unique features to be added to the existing ones. No matter your aim is to improve manufacturing efficiency by 10% or predict how many baubles you need to order this year to meet market demand, there are no ready-made solutions available.

Encompass unpredictability

Thus, the machine learning projects, along with other artificial intelligence projects, are accompanied by relatively high risk. Just like every innovation, research projects mean we need to confirm our hypotheses first. We must prove that what we want to achieve is in fact achievable. For this reason, it is practically impossible to determine the time and budget needed for the completion of most machine learning projects just at the very beginning. This generates obvious problems such as how to get round your board of directors to agree to the costs and with the unpredictability of the outcome. It doesn’t apply to typical projects that take advantage of machine learning, such as recommender systems, which make use of ready-made tools and libraries. But such projects I’d in fact call IT projects.

Minimize the risk

The key here is to minimize the risk and costs and to make the process as safe as possible. Especially that apart from the very project there’s also the question of choosing the right supplier, which always adds to the risk if you are to cooperate for the first time. We’ve already expanded upon the topic in our previous blog post on 5 common fears about working with an external provider, where we reflect on general issues concerning cooperation with an outsourcing company. Having told you how risky the process is, let’s discuss how to make your machine learning project less uncertain.

Stages of our machine learning development process

Companies don’t want to create long-lasting bonds with their providers before they get to know them well – and that’s completely understandable. We facilitate that by having created the grounds for ending cooperation quickly if something goes wrong and a client isn’t satisfied with the outcome. You don’t have to sign a multi-annual contract at the very beginning of the cooperation and you can test us beforehand. This way we also help you to verify quite quickly, even after one or two sprints, if your project has chances to be successful.

So, here’s the process explained, which has been developed over the years on projects realized not only for our clients but also internally. I believe this way of managing R&D projects helps to minimize the risk for both sides.

Phase 1: analyze the problem

You come to us with a problem. What it seems to you is that you can solve it or just make things in a better way – faster, more effectively – with e.g. neural networks or deep learning.

Since every case is different when it comes to machine learning, we need to learn as much as we can about your organization and its challenges. To achieve that, after signing an NDA, we interview you and conduct a workshop session (which is discussed further in this article). Knowing your problem, existing solutions and equipment, the area that is to be automated, and having sample data we can propose an initial idea for a solution or a solving approach. This stage of the project is entirely free and requires pieces of information from you (possibly sample data) and 2-3 hours of your time.

Phase 2: create a feasibility study

Already knowing the problem, we can investigate it further on our own. In phase two we devote time to verify our initial assumptions and theories that had been developed in stage one. We conduct a feasibility study to assess if particular algorithms and solutions will work out. In the end, we try to propose a complete, long-term solution. This phase takes up to a week.

Phase 3: divide into subprojects and start with the first one

Machine learning projects can last from a few weeks to several months and it’s difficult to give the right estimation with minimal information. To be able to effectively manage such a project, we try to divide it into subprojects that have clear timing, deliverables, and estimation. It’s important to define the first 1-2 sprints during which we can prove the core hypothesis of the project. Thus, you know clearly if the path we’ve taken is correct and will solve your problem.

Without a long-term commitment from your side, you can validate basic assumptions for the ML project, such as how many training samples are needed, how effective the system/functions will be, and whether the results you’re expecting are likely to be achieved. Moreover, this method allows you to get to know us quickly – if we keep our promises, how responsive we are, and if you like working with us.

From our experience, the key part of each ML project is data – not only its quantity but also the quality. The initial sprints allow us to realistically evaluate what challenges we will face working with client’s data – and this is particularly important for estimating the project. These can be simple things like mistakes and lack of standardization of data to problems with gathering real-live datasets that could be used for the project.

This way we minimize the risk ‘to the minimum’ – for both sides. You don’t have to sign long-term contracts with an external provider and we don’t have to estimate projects that are highly unpredictable and thus risky. Of course, we won’t deliver the full production solution within the first sprint but it will be significantly cheaper to verify if the very idea is executable at all – and how long it might take to build a complete system.

Examples of machine learning projects

Machine learning can be taken advantage of by companies from a variety of industries. Some of the greater examples are microbiology, nanoelectronics, transportation (self driving cars), mining, healthcare, financial services, and manufacturing.

For example, we’ve developed deep learning algorithms that count and classify microorganisms grown on cell-culture dishes. To achieve high-precision we used methods such as deep neural networks, image processing, and object detection. You’ll read more about this object counting project here.

Yet another example but from a different industry is our AI and AR platform for industrial processes facilitation. It allows managers to draw benefits from augmented reality training by adopting it straight away. The full case study of nsFlow is right here.

Summary

It is possible to develop machine learning solutions without the high risk and immense expenses. You don’t need a multi-million budget to develop it and finally achieve what was desired – doing it step by step. The machine learning development process described above will help you achieve your goals with more peace of mind.