DEV Community: Squash.io

Smoke Testing Best Practices: How to Catch Critical Issues Early

Evandro Miquelito — Wed, 03 May 2023 18:00:35 +0000

Intro

In today's fast-paced software development world, it's essential to catch critical issues before they make their way into production. Smoke testing plays a vital role in this process, as it provides a quick, high-level assessment of the system's stability after a new build or release.

By verifying that the most crucial functionalities are working as expected, smoke tests help ensure that the software is ready for further, more in-depth testing. In this blog post, we will discuss the best practices for smoke testing, guiding you through the process of designing and executing effective tests to identify critical issues early in the development lifecycle.

I. Understanding Smoke Testing

A. Definition and purpose

Smoke testing, sometimes referred to as build verification testing or confidence testing, is a preliminary testing process performed on a new build or release of a software application. The primary purpose of smoke testing is to ensure that the application's critical functionalities are working as expected, and the build is stable enough to proceed with further testing. Smoke tests are typically a small set of high-level test cases that provide a quick assessment of the system's overall health.

B. Importance in the software development lifecycle

Smoke testing plays a crucial role in the software development lifecycle (SDLC) as it helps identify critical issues early on, saving time and resources in the long run. When a new build is released, it's essential to verify that the core features are functioning correctly before investing time and effort in more comprehensive testing. By catching critical issues early, smoke testing reduces the risk of discovering major problems later in the development process, when fixing them can be more costly and time-consuming. Furthermore, smoke testing helps ensure that the software is ready for subsequent stages of testing, such as integration, system, and acceptance testing.

C. Differences between smoke and sanity testing

Although the terms smoke testing and sanity testing are often used interchangeably, there are key differences between the two. While both types of testing aim to determine if the application is stable enough for further testing, their focus and execution differ.

Smoke testing is a broader form of testing, typically conducted when a new build is released to verify that the critical functionalities are working correctly. It is performed early in the SDLC and helps identify major issues before other testing phases begin. Smoke tests are usually pre-defined and can be automated.

On the other hand, sanity testing is a narrower form of testing that focuses on specific components or features that have been modified or added in a recent build. It is conducted later in the SDLC, often after regression testing, to confirm that the changes made have not adversely affected the system's functionality. Sanity tests are generally more ad-hoc and may not be as easily automated as smoke tests.

In summary, while both smoke and sanity testing serve essential purposes in the SDLC, their focus, execution, and timing differ. Understanding these differences can help you better plan and execute your testing processes, ultimately improving the quality of your software.

II. Identifying Critical Functionalities

A. Analyzing the application

The first step in creating an effective smoke testing process is to identify the critical functionalities of your application. This involves thoroughly analyzing your application and understanding its primary purpose, core features, and user workflows. Gaining a deep understanding of your application's architecture, dependencies, and user interactions will help you pinpoint the areas where issues could have the most significant impact on the overall system stability and user experience.

B. Prioritizing key features

Once you have a clear understanding of your application, it's time to prioritize the key features that should be included in your smoke tests. These features are typically the ones that are most critical to the application's functionality, have a high level of complexity, or are frequently used by the end-users. The goal is to focus on the areas of the application that are most likely to cause problems if they fail.

To prioritize the key features, you can start by creating a list of all the essential functionalities and then rank them based on their importance, complexity, and usage. This prioritization will help you focus your smoke testing efforts on the areas that matter the most, ensuring that you catch critical issues early in the development process.

C. Involving stakeholders

Involving stakeholders in the process of identifying critical functionalities is crucial for ensuring that your smoke tests align with the application's requirements and user expectations. Stakeholders, such as product managers, business analysts, or end-users, can provide valuable insights into the features that are most important to them and the ones that could cause the most significant impact if they fail.

Collaborating with stakeholders during the smoke test planning phase can help you create a more comprehensive and effective smoke testing process. By incorporating their feedback and understanding their priorities, you can ensure that your smoke tests cover the areas that matter the most to your users and stakeholders, ultimately leading to a more stable and reliable application.

III. Designing Effective Smoke Test Cases

A. Creating comprehensive test scenarios

To create effective smoke test cases, you need to develop comprehensive test scenarios that cover the critical functionalities identified in the previous step. These scenarios should represent the most common user workflows and interactions with the application, ensuring that the key features are tested from a user's perspective.

When creating test scenarios, consider the different ways users may interact with the application and the expected outcomes. For example, if you have an e-commerce application, some critical functionalities might include user registration, login, product search, adding items to the cart, and completing a purchase. The test scenarios should cover these workflows in detail, ensuring that the application behaves as expected.

Here's an example of a simple test scenario for user registration:

Test Scenario: User Registration

1. Navigate to the registration page.
2. Fill in the required fields with valid data.
3. Click the "Register" button.
4. Verify that a confirmation message appears.
5. Verify that the user is redirected to the dashboard.

B. Focusing on positive test cases

Smoke testing primarily focuses on positive test cases, which are tests that verify that the application behaves correctly under expected conditions. The goal is to confirm that the critical functionalities work as intended, rather than attempting to find all possible edge cases and errors.

For instance, in the user registration example mentioned earlier, a positive test case would involve providing valid input data and ensuring that the registration process is successful.

def test_user_registration_success():
    # Setup: Navigate to the registration page and enter valid data.
    navigate_to_registration_page()
    enter_valid_registration_data()

    # Action: Click the "Register" button.
    click_register_button()

    # Assert: Verify that a confirmation message appears and the user is redirected to the dashboard.
    assert is_confirmation_message_displayed()
    assert is_redirected_to_dashboard()

C. Ensuring test cases are easy to understand and maintain

It's essential to ensure that your smoke test cases are easy to understand and maintain. This involves writing clear and concise test case descriptions, using descriptive function and variable names, and following the best practices for writing clean and maintainable code.

One way to make your test cases more maintainable is by using the Arrange-Act-Assert (AAA) pattern. This pattern involves organizing your test code into three distinct sections: setting up the test data and preconditions (Arrange), executing the action being tested (Act), and verifying that the expected outcome has occurred (Assert).

Here's an example of a smoke test case using the AAA pattern:

def test_adding_item_to_cart():
    # Arrange: Navigate to the product page and ensure the cart is empty.
    navigate_to_product_page()
    clear_shopping_cart()

    # Act: Add a product to the cart.
    add_product_to_cart()

    # Assert: Verify that the product is successfully added to the cart.
    assert is_product_in_cart()

By following these best practices, you can design effective smoke test cases that provide a quick and accurate assessment of your application's stability, helping you catch critical issues early in the development process.

IV. Automating Smoke Tests

A. Benefits of automation

Automating smoke tests can provide several benefits to your software development process:

1. Speed: Automated tests can be executed much faster than manual tests, which allows you to quickly assess the stability of a new build and proceed with further testing or development.

2. Consistency: Automated tests follow the same steps each time they are executed, ensuring that the results are consistent and reliable.

3. Reusability: Once created, automated test scripts can be easily reused for future builds, reducing the time and effort needed for manual smoke testing.

4. Continuous Integration: Automated smoke tests can be integrated into your continuous integration pipeline, ensuring that critical issues are caught early and automatically during the development process.

B. Choosing the right automation tools

When choosing a smoke testing automation tool, consider the following factors:

1. Compatibility: The tool should be compatible with your application's technology stack and your development environment.

2. Ease of use: The tool should be easy to learn and use, with a user-friendly interface and clear documentation.

3. Flexibility: The tool should be flexible enough to handle various test scenarios and adapt to changes in the application's requirements.

4. Reporting: The tool should provide detailed and easy-to-understand test reports, making it simple to identify and fix issues.

Some popular automation tools for smoke testing include Selenium, TestNG, JUnit, and Pytest. Each of these tools has its advantages and limitations, so it's essential to evaluate them based on your specific needs and requirements.

C. Integrating automation into your development process

To effectively integrate smoke test automation into your development process, follow these steps:

1. Create a smoke test suite: Develop a suite of automated smoke tests based on the test scenarios and cases designed in the previous steps. Ensure that the tests cover the critical functionalities of your application.

Using the Pytest framework, you can create a smoke test suite like this:

# test_smoke.py

def test_user_registration():
    # Your test code for user registration

def test_login():
    # Your test code for user login

def test_search_product():
    # Your test code for searching a product

def test_add_to_cart():
    # Your test code for adding an item to the cart

def test_checkout():
    # Your test code for completing a purchase

2. Set up a test environment: Configure a test environment that closely mirrors your production environment. This will help ensure that your automated tests accurately reflect real-world conditions.

3. Implement version control: Use a version control system, such as Git, to manage your smoke test scripts and keep track of changes.

To add your test suite to a Git repository, execute the following commands:

$ git init
$ git add test_smoke.py
$ git commit -m "Add smoke test suite"
$ git remote add origin https://github.com/yourusername/yourrepository.git
$ git push -u origin master

4. Schedule test execution: Configure your smoke tests to run automatically whenever a new build is released or at regular intervals, depending on your development process.

In this example, we will use Jenkins to schedule and run the smoke tests:

Install the necessary plugins for your project (e.g., Python, Pytest, Git).
Create a new Jenkins job and configure the Git repository containing your smoke test suite.
Add a build step to execute the Pytest command: pytest test_smoke.py
Schedule the build to run whenever a new build is released or at regular intervals using the "Build Triggers" section.

5. Monitor and analyze test results: Regularly review the test results to identify and fix any issues that arise. Ensure that the relevant stakeholders are informed of the test results and any critical issues that need attention.

After executing your smoke tests in Jenkins, you can view the test results on the build page. The Pytest plugin provides a detailed report with pass/fail status and any error messages or stack traces.

By automating your smoke tests and integrating them into your development process, you can quickly and reliably catch critical issues early on, reducing the risk of major problems making their way into production.

V. Integrating Smoke Testing into the Development Pipeline

A. Identifying the appropriate stage for smoke tests

Smoke tests should be executed early in the development pipeline, typically right after a new build is created and before any further in-depth testing. The primary goal of smoke testing is to quickly identify critical issues that could render the application unusable or unstable. By executing smoke tests early in the pipeline, you can catch these issues before they reach later stages, saving time and effort.

B. Coordinating with the development team

To ensure that smoke testing is effectively integrated into the development pipeline, it's crucial to coordinate with the development team. This involves:

Communicating the purpose and importance of smoke tests to developers, so they understand their role in maintaining the stability of the application.
Collaborating with developers to identify the critical functionalities that should be included in the smoke tests, as well as any changes to these functionalities as the application evolves.
Encouraging developers to execute smoke tests locally before committing their code to the version control system. This can help catch issues early and reduce the number of broken builds.

For example, developers can run smoke tests locally using the Pytest framework:

$ pytest test_smoke.py

C. Implementing continuous integration and deployment

Integrating smoke tests into your continuous integration (CI) and continuous deployment (CD) pipeline can help ensure that critical issues are caught early and automatically during the development process. Here's how you can implement this integration using a CI/CD tool like Jenkins:

Create a new Jenkins job dedicated to smoke testing and configure it to trigger automatically whenever new code is pushed to the version control system.
Add a build step in the Jenkins job to check out the latest version of your code from the version control system (e.g., Git) and execute the smoke tests using the appropriate test runner (e.g., Pytest).

$ git pull origin master
$ pytest test_smoke.py

Configure the Jenkins job to notify the development team of the test results, either by email or through a messaging platform like Slack.
Integrate smoke tests with your deployment process. If the smoke tests pass, proceed with the deployment of the new build to the staging or production environment. If the tests fail, halt the deployment process and notify the development team to fix the issues.

By integrating smoke testing into your development pipeline, you can quickly identify and address critical issues, ensuring a more stable and reliable application throughout the development process.

VI. Tracking and Reporting Smoke Test Results

A. Establishing a clear reporting process

To effectively track and report smoke test results, you should establish a clear and consistent reporting process. This involves:

Defining the format and content of the test reports, including details such as the test cases executed, pass/fail status, error messages, and any relevant screenshots or logs.
Choosing a centralized location for storing test reports, such as a shared drive or a dedicated test management tool.
Setting up automated notifications to inform relevant stakeholders of the test results, either through email or a messaging platform like Slack.

For example, you can use the Pytest framework to generate an XML report of your smoke test results:

$ pytest test_smoke.py --junitxml=smoke_test_report.xml

B. Monitoring test results over time

It's essential to monitor smoke test results over time to identify trends and patterns that could indicate potential issues in the application. By regularly reviewing test results, you can proactively address any emerging problems before they become critical.

Some key metrics to track include:

Test pass/fail rates: Monitor the percentage of tests passing and failing in each smoke test run. An increase in the failure rate could indicate issues with the application or the test suite.
Test execution time: Track the time it takes to execute the smoke tests. A significant increase in execution time could indicate performance issues or inefficient test cases.
Test coverage: Keep track of the number of critical functionalities covered by the smoke tests. As the application evolves, it's essential to ensure that new critical features are included in the smoke tests.

C. Communicating results to stakeholders

Effective communication of smoke test results is crucial for ensuring that relevant stakeholders are aware of the application's stability and any critical issues that need to be addressed. To communicate test results:

Share test reports with stakeholders, either through email or a shared drive. Ensure that the reports are clear, concise, and easy to understand.
Present test results in team meetings, discussing any issues that were encountered and the steps taken to resolve them.
Establish a process for escalating critical issues to the appropriate team members, ensuring that they are addressed promptly and efficiently.

For example, you can use a messaging platform like Slack to automatically notify stakeholders of the smoke test results:

# Assuming you have a slack bot set up, you can use a script like this:

# Assuming you have a slack bot set up, you can use a script like this:

import os
from slack_sdk import WebClient
from slack_sdk.errors import SlackApiError

client = WebClient(token=os.environ['SLACK_API_TOKEN'])

try:
    response = client.chat_postMessage(
        channel='#smoke-test-results',
        text='Smoke Test Results: 10 Passed, 1 Failed'
    )
except SlackApiError as e:
    print(f"Error posting message: {e}")

By establishing a clear reporting process, monitoring test results over time, and effectively communicating results to stakeholders, you can ensure that your smoke tests provide valuable insights into the stability and reliability of your application.

VII. Continuous Improvement of Smoke Tests

A. Regularly reviewing and updating test cases

To ensure that your smoke tests remain effective and relevant, it's crucial to regularly review and update the test cases. This involves:

Analyzing test results and identifying any recurring issues or patterns that could indicate problems with the test cases themselves.
Reviewing the test cases for clarity and simplicity, ensuring that they are easy to understand, maintain, and update.
Updating test cases as needed to address any changes in the application, such as new features or modified functionalities.

For example, if a new critical feature has been added to your application, you should create a new smoke test case for it:

# test_smoke.py

# Existing test cases...

def test_new_critical_feature():
    # Your test code for the new critical feature

B. Adapting to changes in the application

As your application evolves, it's essential to adapt your smoke tests accordingly. This involves:

Identifying new critical functionalities that should be included in the smoke tests, either through discussions with the development team or by analyzing changes to the application's requirements.
Modifying existing test cases as needed to accommodate changes in the application, such as updates to user interfaces, APIs, or data structures.
Removing or de-prioritizing test cases that no longer cover critical functionalities, ensuring that the smoke tests remain focused on the most important aspects of the application.

C. Incorporating feedback from the development team and stakeholders

To continuously improve your smoke tests, it's important to incorporate feedback from the development team and other stakeholders. This involves:

Regularly discussing the smoke test results with the development team and stakeholders, seeking their input on any issues encountered and potential improvements to the test cases.
Encouraging developers to participate in the creation and maintenance of smoke test cases, fostering a sense of ownership and responsibility for the quality of the application.
Implementing improvements based on feedback, such as modifying test cases, updating the reporting process, or refining the smoke testing schedule.

For example, if the development team suggests that the current smoke tests don't adequately cover a specific functionality, you can work with them to create a new test case that addresses the concern:

# test_smoke.py

# Existing test cases...

def test_updated_functionality():
    # Your test code for the updated functionality, based on feedback from the development team

By regularly reviewing and updating your smoke tests, adapting to changes in the application, and incorporating feedback from the development team and stakeholders, you can ensure that your smoke tests remain effective and continue to provide valuable insights into the stability and reliability of your application.

VIII. Case Study: Successful Smoke Testing in Practice

A. The challenges faced

A medium-sized e-commerce company faced several challenges in their software development process. They experienced frequent critical issues in production, leading to downtime and lost revenue. The development team struggled to identify and fix issues before they reached production, as they lacked a consistent and effective testing process. This situation led the company to explore smoke testing as a potential solution to improve their application stability.

B. Implementing smoke testing best practices

To address these challenges, the company decided to implement smoke testing best practices, which included the following steps:

Identifying critical functionalities: The development team worked with product managers and other stakeholders to identify the key features of their e-commerce platform that were critical to the user experience, such as user registration, login, product search, adding items to the cart, and checkout.
Designing effective smoke test cases: The team created comprehensive test scenarios and focused on positive test cases to ensure that the critical functionalities were working as expected. They also made sure that the test cases were easy to understand and maintain.

# test_smoke.py

def test_user_registration():
    # Test code for user registration

def test_login():
    # Test code for user login

def test_search_product():
    # Test code for searching a product

def test_add_to_cart():
    # Test code for adding an item to the cart

def test_checkout():
    # Test code for completing a purchase

Automating smoke tests: The company chose to automate their smoke tests using the Pytest framework and integrated the tests into their continuous integration (CI) pipeline. This allowed them to run the smoke tests automatically whenever new code was pushed to the version control system.
Integrating smoke testing into the development pipeline: The company identified the appropriate stage for smoke tests in their pipeline and coordinated with the development team to ensure the tests were executed consistently. They also implemented continuous integration and deployment using tools like Jenkins to automate the entire process.

C. Results and lessons learned

After implementing smoke testing best practices, the company experienced several positive outcomes:

Improved application stability: By catching critical issues early in the development process, the company significantly reduced the number of issues reaching production, leading to a more stable and reliable user experience.
Faster issue resolution: The development team was able to quickly identify and fix issues before they reached later stages of the development pipeline, saving time and effort.
Greater collaboration: The process of identifying critical functionalities and designing test cases encouraged collaboration between the development team, product managers, and other stakeholders, fostering a shared sense of responsibility for the quality of the application.
Continuous improvement: The company's commitment to regularly reviewing and updating their smoke tests, adapting to changes in the application, and incorporating feedback from the development team and stakeholders ensured that their smoke tests remained effective and provided valuable insights into the application's stability.

Through the successful implementation of smoke testing best practices, the company was able to address their challenges and significantly improve their software development process. This case study demonstrates the value of smoke testing in identifying and resolving critical issues early in the development process, ultimately leading to a more stable and reliable application.

Conclusion

Implementing smoke testing best practices is key to ensuring the stability and reliability of your software. By focusing on critical functionalities, designing test cases that cover key scenarios, automating the process, and integrating smoke tests into your development pipeline, you can catch issues early and prevent them from escalating into more significant problems.

By doing so, you'll save time, reduce costs, and ultimately deliver a higher-quality product to your users. So, embrace these best practices and watch your software development process become more efficient and robust, giving you the confidence you need for every release.

10 Proven Steps to Double the Speed of Your Django App

Evandro Miquelito — Wed, 26 Apr 2023 12:00:00 +0000

Introduction

In this article, we will share 10 proven steps that can help you double the speed of your Django app. Whether you're dealing with slow page load times, high response times, or performance bottlenecks, these steps will provide you with practical tips and techniques to optimize your Django app and deliver a better user experience.

1. Optimize database queries

Review your database queries and make sure they are efficient. Use Django's query optimization techniques such as select_related, prefetch_related, and defer/only to minimize the number of database queries and reduce database load.

Here is a detailed guide on how to optimize db queries in Django.

2. Use caching

Implement caching using Django's built-in caching framework or external caching tools like Memcached or Redis. Caching frequently accessed data can significantly reduce database queries and speed up your app.

Examples:

1. Using Django's built-in caching framework:

# settings.py

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11211',  # Replace with your Memcached server's address
    }
}

# views.py

from django.core.cache import cache
from .models import MyModel

def get_data():
    # Query the database to get data
    data = MyModel.objects.all()
    return data

def get_data_with_cache():
    # Try to get data from cache
    data = cache.get('my_data')
    if not data:
        # If data is not available in cache, fetch from database and store in cache
        data = get_data()
        cache.set('my_data', data)
    return data

2. Using Memcached as an external caching tool:

# settings.py

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11211',  # Replace with your Memcached server's address
    }
}

# views.py

from django.core.cache import cache
from .models import MyModel

def get_data():
    # Query the database to get data
    data = MyModel.objects.all()
    return data

def get_data_with_cache():
    # Try to get data from cache
    data = cache.get('my_data')
    if not data:
        # If data is not available in cache, fetch from database and store in cache
        data = get_data()
        cache.set('my_data', data)
    return data

3. Using Redis as an external caching tool:

# settings.py

CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': 'redis://127.0.0.1:6379/1',  # Replace with your Redis server's address
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
        }
    }
}

# views.py

from django.core.cache import cache
from .models import MyModel

def get_data():
    # Query the database to get data
    data = MyModel.objects.all()
    return data

def get_data_with_cache():
    # Try to get data from cache
    data = cache.get('my_data')
    if not data:
        # If data is not available in cache, fetch from database and store in cache
        data = get_data()
        cache.set('my_data', data)
    return data

Note: The above code examples assume that you have already installed and configured the caching tools (Memcached or Redis) on your server, and you have the appropriate caching backend installed in your Django project. Make sure to replace the cache backend settings (such as LOCATION) with the correct address of your caching server.

3. Optimize view functions

Review your view functions and optimize them for performance. Avoid unnecessary calculations, database queries, or data processing in your views. Use Django's class-based views for efficient code organization and performance.

Bonus: use asynchronous handlers when it's possible.

Examples:

1. Using `select_related()` to reduce database queries:

from django.shortcuts import render
from .models import MyModel

def my_view(request):
    # Fetch data from database with related objects
    data = MyModel.objects.all().select_related('related_model')

    # Perform some calculations
    processed_data = [item.some_field * 2 for item in data]

    # Filter data based on a condition
    filtered_data = [item for item in processed_data if item > 10]

    # Render the response
    return render(request, 'my_template.html', {'data': filtered_data})

2. Utilizing Django's built-in caching framework:

from django.shortcuts import render
from django.core.cache import cache
from .models import MyModel

def my_view(request):
    # Try to get data from cache
    data = cache.get('my_data')
    if data is None:
        # If not available in cache, fetch from database
        data = MyModel.objects.all()

        # Perform some calculations
        processed_data = [item.some_field * 2 for item in data]

        # Filter data based on a condition
        filtered_data = [item for item in processed_data if item > 10]

        # Store data in cache for future use
        cache.set('my_data', filtered_data)

    # Render the response
    return render(request, 'my_template.html', {'data': data})

3. Using Django's `Prefetch` to optimize related object queries:

from django.shortcuts import render
from django.db.models import Prefetch
from .models import MyModel

def my_view(request):
    # Fetch data from database with related objects using Prefetch
    data = MyModel.objects.all().prefetch_related(Prefetch('related_model'))

    # Perform some calculations
    processed_data = [item.some_field * 2 for item in data]

    # Filter data based on a condition
    filtered_data = [item for item in processed_data if item > 10]

    # Render the response
    return render(request, 'my_template.html', {'data': filtered_data})

Using select_related(), Django's caching framework, and Prefetch, can further optimize the view functions by reducing database queries, utilizing caching, and optimizing related object queries, respectively, leading to improved performance in Django applications.

4. Optimize templates

Review your templates and minimize the use of heavy computations or complex logic in the templates. Use Django's template caching, template inheritance, and template tags for optimized rendering.

Examples:

1. Utilizing Django's template caching:

# my_view.py

from django.shortcuts import render
from django.core.cache import cache
from .models import MyModel

def my_view(request):
    # Try to get data from cache
    data = cache.get('my_data')
    if data is None:
        # If not available in cache, fetch from database
        data = MyModel.objects.all()

        # Perform some calculations
        processed_data = [item.some_field * 2 for item in data]

        # Filter data based on a condition
        filtered_data = [item for item in processed_data if item > 10]

        # Store data in cache for future use
        cache.set('my_data', filtered_data)

    # Render the response with cached data
    return render(request, 'my_template.html', {'data': data})

<!-- my_template.html -->

{% extends 'base_template.html' %}

{% block content %}
    <!-- Render the cached data in the template -->
    {% for item in data %}
        <p>{{ item }}</p>
    {% endfor %}
{% endblock %}

2. Utilizing Django's template inheritance:

<!-- base_template.html -->

<!DOCTYPE html>
<html>
<head>
    <title>My App</title>
</head>
<body>
    <header>
        <!-- Common header content -->
    </header>
    <main>
        <!-- Render the content from child templates -->
        {% block content %}{% endblock %}
    </main>
    <footer>
        <!-- Common footer content -->
    </footer>
</body>
</html>

<!-- my_template.html -->

{% extends 'base_template.html' %}

{% block content %}
    <!-- Render the content specific to this template -->
    <h1>My Template</h1>
    <!-- Include template tags for optimized rendering -->
    {% load myapp_tags %}
    <p>Processed Data: {% my_template_tag data %}</p>
{% endblock %}

3. Creating custom template tags for complex logic:

# myapp_tags.py

from django import template

register = template.Library()

@register.filter
def my_template_tag(data):
    # Perform complex logic on data
    processed_data = [item.some_field * 2 for item in data]

    # Filter data based on a condition
    filtered_data = [item for item in processed_data if item > 10]

    # Return the processed data as a string
    return ', '.join(map(str, filtered_data))

<!-- my_template.html -->

{% extends 'base_template.html' %}

{% block content %}
    <!-- Render the content specific to this template -->
    <h1>My Template</h1>
    <!-- Include the custom template tag for optimized rendering -->
    <p>Processed Data: {{ data|my_template_tag }}</p>
{% endblock %}

5. Enable Gzip compression

Enable Gzip compression for HTTP responses using Django's middleware or web server configuration. Gzip compression reduces the size of data transferred over the network, improving app performance.

Examples:

1. Enabling Gzip compression using Django middleware:

# middleware.py

import gzip
from django.middleware.common import CommonMiddleware
from django.utils.decorators import gzip_page

class GzipMiddleware(CommonMiddleware):
    """
    Middleware class to enable Gzip compression for HTTP responses.
    """

    def __init__(self, get_response=None):
        super().__init__(get_response)
        self.get_response = get_response

    @gzip_page
    def __call__(self, request):
        # Handle Gzip compression for HTTP responses
        response = self.get_response(request)

        # Set response headers to indicate Gzip compression
        response['Content-Encoding'] = 'gzip'
        response['Vary'] = 'Accept-Encoding'

        return response

Note: The gzip_page decorator is used from Django's django.utils.decorators module to compress the response content using Gzip.

2. Enabling Gzip compression using web server configuration (e.g., Nginx):

# nginx.conf

http {
    gzip on;
    gzip_types text/html text/css application/javascript;

    # Other nginx configuration settings
}

In this example, Gzip compression is enabled in the Nginx web server configuration by setting gzip on; and specifying the file types to be compressed using the gzip_types directive.

6. Use a Content Delivery Network (CDN)

Utilize a CDN to cache and serve static files, such as CSS, JavaScript, and images, from geographically distributed servers. This can reduce server load and improve page load times.

Examples:

1. Utilizing a CDN with Django:

# settings.py

# Set the URL of your CDN
CDN_URL = 'https://cdn.example.com/'

# Configure the STATIC_URL to point to the CDN URL
STATIC_URL = CDN_URL + 'static/'

<!-- template.html -->

<!-- Use the CDN URL for serving static files -->
<link rel="stylesheet" href="{{ STATIC_URL }}css/styles.css">
<script src="{{ STATIC_URL }}js/scripts.js"></script>
<img src="{{ STATIC_URL }}images/image.jpg" alt="Image">

2. Utilizing a CDN with a web server (e.g., Nginx):

# nginx.conf

http {
    # Configure Nginx to proxy requests for static files to the CDN
    location /static/ {
        proxy_pass https://cdn.example.com/static/;
    }

    # Other Nginx configuration settings
}

<!-- template.html -->

<!-- Use the Nginx proxy location for serving static files -->
<link rel="stylesheet" href="/static/css/styles.css">
<script src="/static/js/scripts.js"></script>
<img src="/static/images/image.jpg" alt="Image">

7. Optimize database connection management

Use connection pooling to efficiently manage database connections and reuse existing connections instead of creating new ones for every request. This can reduce overhead and improve database query performance.

8. Use asynchronous tasks

Offload time-consuming tasks to asynchronous tasks using Django's asynchronous task frameworks like Celery or Django Channels. This can free up server resources and improve app performance.

Examples:

1. Offloading tasks to Celery:

# tasks.py

from celery import shared_task

@shared_task
def process_data(data):
    # Perform time-consuming task here
    # ...

# views.py

from .tasks import process_data

def my_view(request):
    # Offload task to Celery
    process_data.delay(data)

    # Continue with view logic
    # ...

2. Offloading tasks to Django Channels:

# consumers.py

import asyncio
from channels.generic.websocket import AsyncWebsocketConsumer

class MyConsumer(AsyncWebsocketConsumer):
    async def connect(self):
        await self.accept()

    async def receive(self, text_data):
        # Offload task to Django Channels
        await self.channel_layer.async_send("my_channel", {
            "type": "process_data",
            "data": text_data,
        })

    async def process_data(self, event):
        # Perform time-consuming task here
        # ...

# views.py

from channels.layers import get_channel_layer
from asgiref.sync import async_to_sync

def my_view(request):
    # Offload task to Django Channels
    channel_layer = get_channel_layer()
    async_to_sync(channel_layer.send)("my_channel", {
        "type": "process_data",
        "data": data,
    })

    # Continue with view logic
    # ...

9. Optimize server configuration

Review and optimize your server configuration, including web server settings, database settings, and caching settings, to fine-tune performance.

10. Monitor and analyze app performance

Regularly monitor and analyze the performance of your Django app using performance monitoring tools, profiling, and logging. Identify and optimize bottlenecks to continually improve app performance.

Remember, performance optimization is an ongoing process, and results may vary depending on the specific requirements and characteristics of your Django app. It's important to thoroughly test and benchmark your app after implementing any optimizations to ensure they are effective in improving app speed.

Conclusion

By following these 10 proven steps, you can significantly improve the speed and performance of your Django app. From optimizing database queries to leveraging caching, using a Content Delivery Network (CDN), and implementing asynchronous tasks, these techniques can make a noticeable difference in your app's performance. Remember to regularly monitor and benchmark your app's performance to ensure that it continues to run smoothly and efficiently. By investing time and effort into optimizing your Django app, you can provide a better experience for your users and keep them engaged with your app. So go ahead and implement these steps to double the speed of your Django app and take it to the next level!

Microservices in 12 steps: A Short Guide with Docker for Monolithic Code Base Migrations (with some code)

Evandro Miquelito — Tue, 25 Apr 2023 17:14:20 +0000

Intro

Modern software development practices are shifting towards microservices architecture, which offers benefits such as improved scalability, flexibility, and maintainability. If you have a monolithic code base and are looking to embrace microservices, Docker can be a powerful tool to help with the migration process.

In this article, we will provide you with a step-by-step guide on how to migrate a monolithic code base to a microservices architecture using Docker. With Docker's containerization capabilities, you can encapsulate individual microservices, ensure consistency in packaging, and achieve portability across different environments. So, let's dive in and learn how to make this transition in a structured and efficient manner.

Step 1: Understand the Current Monolithic Code Base

Gain a deep understanding of the existing monolithic code base, its architecture, dependencies, and functionalities. Identify the different components or modules that can be decoupled and isolated as microservices.

Step 2: Define a Microservices Architecture

Design the target microservices architecture, including the desired granularity of microservices, communication patterns, data management, and deployment strategies. This includes defining the boundaries and interfaces of each microservice.

Example, including defining the boundaries and interfaces of each microservice:

1. Defining the Microservice Boundaries and Interfaces:

# Example of defining the boundaries and interfaces of a User microservice in Python using FastAPI

# user.py - User microservice

from fastapi import FastAPI

app = FastAPI()

@app.get("/users/{user_id}")
async def get_user(user_id: int):
    # Logic to fetch user data from database
    return {"user_id": user_id, "name": "John", "age": 30}

@app.post("/users")
async def create_user(user_data: dict):
    # Logic to create a new user in the database
    return {"user_id": 1, "name": user_data["name"], "age": user_data["age"]}

2. Communication Patterns between Microservices:

# Example of communication patterns between User and Order microservices using RESTful APIs in Node.js with Express

// user-service.js - User microservice

const express = require('express');
const app = express();

app.get('/users/:userId', (req, res) => {
  // Logic to fetch user data from database
  res.json({ userId: req.params.userId, name: 'John', age: 30 });
});

// order-service.js - Order microservice

const express = require('express');
const app = express();

app.post('/orders', (req, res) => {
  // Logic to create a new order in the database
  // and communicate with User microservice to fetch user data
  // Example of making API call to User microservice
  axios.get(`http://user-service/users/${req.body.userId}`)
    .then(response => {
      // Process user data and create order
      res.json({ orderId: 1, userId: req.body.userId, productName: req.body.productName });
    })
    .catch(error => {
      // Handle error
      res.status(500).json({ error: 'Failed to create order' });
    });
});

3. Data Management in Microservices:

// Example of data management in microservices using a message queue (RabbitMQ) in Java with Spring Boot

// user-service - User microservice

@RestController
public class UserController {

  @Autowired
  private UserService userService;

  @GetMapping("/users/{userId}")
  public User getUser(@PathVariable("userId") int userId) {
    // Logic to fetch user data from database
    return userService.getUserById(userId);
  }
}

// order-service - Order microservice

@RestController
public class OrderController {

  @Autowired
  private OrderService orderService;

  @Autowired
  private RabbitTemplate rabbitTemplate;

  @PostMapping("/orders")
  public Order createOrder(@RequestBody OrderDto orderDto) {
    // Logic to create a new order in the database
    // Publish order data to RabbitMQ message queue for processing
    rabbitTemplate.convertAndSend("order-exchange", "order.create", orderDto);
    return orderService.createOrder(orderDto);
  }
}

@Component
public class OrderConsumer {

  @RabbitListener(queues = "order-queue")
  public void processOrder(OrderDto orderDto) {
    // Logic to process order data, communicate with User microservice, and create order
  }
}

Step 3: Containerize Microservices with Docker

Use Docker to containerize the microservices. Create Docker images for each microservice, which encapsulate the application code, runtime dependencies, and configuration. Docker containers provide consistency in packaging and portability across different environments.

Example:

# Example of deployment strategy using Docker Compose for a microservices architecture with multiple services

version: '3'
services:
  user-service:
    build: ./user-service
    ports:
      - "8001:8001"
    networks:
      - my-network
  order

Step 4: Set Up Docker Infrastructure

Set up the Docker infrastructure, including Docker Engine, Docker Compose, and Kubernetes, depending on the desired deployment approach. Docker Compose can be used for local development and testing, while Kubernetes can be used for container orchestration in a production environment.

Examples:

1. Docker Engine setup:

# Example of installing Docker Engine on Ubuntu using Docker's official installation script

# Update package index
sudo apt update

# Install dependencies
sudo apt install apt-transport-https ca-certificates curl software-properties-common

# Add Docker repository
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/docker.gpg
sudo add-apt-repository "deb [arch=$(dpkg --print-architecture signed-by /etc/apt/trusted.gpg.d/docker.gpg)] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

# Update package index again
sudo apt update

# Install Docker
sudo apt install docker-ce docker-ce-cli containerd.io

# Start and enable Docker service
sudo systemctl start docker
sudo systemctl enable docker

# Verify Docker installation
docker --version

2. Docker Compose setup:

# Example of Docker Compose configuration for local development and testing

version: '3'
services:
  user-service:
    build: ./user-service
    ports:
      - "8001:8001"
    networks:
      - my-network
  order-service:
    build: ./order-service
    ports:
      - "8002:8002"
    networks:
      - my-network

networks:
  my-network:

3. Kubernetes setup:

# Example of installing Kubernetes using Minikube for container orchestration

# Install dependencies
sudo apt update
sudo apt install curl

# Install kubectl
sudo snap install kubectl --classic

# Install Minikube
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

# Start Minikube cluster
minikube start

# Verify Kubernetes installation
kubectl version

Detailed setup instructions for Docker Engine, Docker Compose, and Kubernetes may vary depending on the operating system and environment you are using. Please refer to official documentation for accurate installation steps.

Step 5: Develop and Test Microservices

Refactor and rewrite the monolithic code into individual microservices. Develop and test each microservice in isolation using Docker containers to ensure they are functioning correctly and communicate effectively with each other.

Step 6: Implement Service Discovery

Implement a service discovery mechanism to allow microservices to discover and communicate with each other dynamically. Tools such as Consul, etcd, or Kubernete's built-in DNS-based service discovery can be used for this purpose.

Examples:

1. Refactor and rewrite monolithic code into microservices:

Example of monolithic code before refactoring:

# Monolithic code for an e-commerce application

def process_order(order):
    # Process order logic here
    # ...
    return result

def get_customer_info(customer_id):
    # Get customer info logic here
    # ...
    return customer_info

def calculate_shipping_cost(order):
    # Calculate shipping cost logic here
    # ...
    return shipping_cost

# Other functionalities...

Example of microservices after refactoring:

# Microservice 1: Order Service

def process_order(order):
    # Process order logic here
    # ...
    return result

# Other functionalities specific to order service...

# Microservice 2: Customer Service

def get_customer_info(customer_id):
    # Get customer info logic here
    # ...
    return customer_info

# Other functionalities specific to customer service...

# Microservice 3: Shipping Service

def calculate_shipping_cost(order):
    # Calculate shipping cost logic here
    # ...
    return shipping_cost

# Other functionalities specific to shipping service...

# Each microservice is a separate module or application that can be developed, tested, and deployed independently.

2. Develop and test each microservice using Docker containers:

Example of Dockerfile for a microservice:

# Dockerfile for Order Service microservice

# Use a base image
FROM python:3.9

# Set working directory
WORKDIR /app

# Copy source code into the container
COPY . .

# Install dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Expose necessary ports
EXPOSE 8001

# Run the microservice
CMD ["python", "app.py"]

Example of Docker Compose configuration for local development and testing:

# Docker Compose configuration for local development and testing

version: '3'
services:
  order-service:
    build: ./order-service
    ports:
      - "8001:8001"
    networks:
      - my-network

# Other services and networks...

The above examples are simplified and may not cover all aspects of developing and testing microservices with Docker.

Step 7: Set Up Logging and Monitoring

Implement logging and monitoring mechanisms to collect and analyze logs, metrics, and traces from microservices. Tools such as ELK Stack (Elasticsearch, Logstash, and Kibana), Prometheus, or Zipkin can be used for this purpose.

Examples:

1. Implement logging using ELK Stack (Elasticsearch, Logstash, and Kibana):

Example of logging configuration in a microservice using Logstash:

# Logstash configuration file

input {
  beats {
    port => 5044
  }
}

filter {
  # Filter logic here
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "my-microservice-%{+YYYY.MM.dd}"
  }
}

Example of logging code in a microservice using a logging library like Log4j2 (Java):

// Log4j2 logging code in a Java microservice

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

public class MyMicroservice {
  private static final Logger LOGGER = LogManager.getLogger(MyMicroservice.class);

  public void processOrder(Order order) {
    // Process order logic here

    // Log an info level message
    LOGGER.info("Order processed successfully: {}", order);
  }
}

2. Implement monitoring using Prometheus:

Example of monitoring code in a microservice using Prometheus client library (Python):

# Prometheus monitoring code in a Python microservice

from prometheus_client import Counter, start_http_server

# Define a counter for tracking order requests
ORDER_REQUESTS = Counter('order_requests_total', 'Total number of order requests')

def process_order(order):
    # Process order logic here

    # Increment the order requests counter
    ORDER_REQUESTS.inc()

    # ...

3. Implement distributed tracing using Zipkin:

Example of distributed tracing code in a microservice using OpenZipkin library (Java):

// Zipkin distributed tracing code in a Java microservice

import brave.Span;
import brave.Tracer;

public class MyMicroservice {
  private final Tracer tracer;

  public MyMicroservice(Tracer tracer) {
    this.tracer = tracer;
  }

  public void processOrder(Order order) {
    // Process order logic here

    // Start a new Zipkin span
    Span span = tracer.newTrace().name("processOrder").start();

    try {
      // ... Processing logic ...
    } finally {
      // End the Zipkin span
      span.finish();
    }
  }
}

Actual implementation may vary depending on the specific tools, libraries, and frameworks used in your microservices architecture.

Step 8: Implement Deployment and Scaling Strategies

Define and implement deployment and scaling strategies for microservices using Kubernete, such as rolling updates, blue-green deployments, or canary releases. This allows for seamless deployment and scaling of microservices in a distributed environment.

Step 9: Implement Fault Tolerance and Resiliency

Implement fault tolerance and resiliency measures in microservices to handle failures gracefully. This may include retry mechanisms, circuit breakers, and fallback strategies to ensure the overall system's reliability.

Step 10: Monitor and Optimize

Continuously monitor and optimize the microservices architecture using Docker monitoring and logging tools, and make adjustments as needed to improve performance, scalability, and reliability.

Step 11: Gradual Rollout

Plan for a gradual rollout of microservices in production, starting with less critical services and gradually moving towards more critical ones. Monitor the system's performance and stability during the rollout, and make necessary adjustments as needed.

Step 12: Train and Educate Teams

Provide training and education to development and operations teams on Docker, microservices, and the new architecture. Ensure that teams are proficient in using Docker and understand the best practices for developing, deploying, and managing microservices.

Conclusion:

Migrating from a monolithic code base to a microservices architecture using Docker can be a challenging but rewarding journey.

By following the step-by-step guide provided in this article, you can effectively decouple dependencies, containerize microservices, implement service discovery, and scale your applications with Kubernetes.

It's important to carefully plan, test, and monitor the migration process to ensure a smooth transition. With the right approach, Docker can be a valuable tool to enable the adoption of microservices architecture and unlock the benefits of improved scalability, flexibility, and maintainability in your software development practices. So, get started with Docker and embark on the path towards a modern and efficient microservices architecture for your applications.

7 Shared Traits of Ineffective Engineering Teams

Evandro Miquelito — Mon, 05 Oct 2020 13:39:23 +0000

Why is your engineering team ineffective? In this article you will learn to recognize seven bad team traits.

Ineffective engineering teams are not all the same, and the problems are not always obvious. However, there are some red flags to look for when evaluating a team. Team leaders and CEOs need to keep an eye out for the following red flags.

1. Bad hires and partnerships

Having an effective engineering team starts with hiring the right people and the right companies with whom to do business with. You need to hire people who fit into your work culture. That means hiring people who share the company values and buy into the mission. All new hires should be well-trained, courteous, hard-working, and professional. However, correct hiring practices don’t end there. What follows are some common hiring mistakes.

Prioritizing diversity over effectiveness

Diversity hiring creates a vibrant culture in which different perspectives bring new solutions to the table. However, team leaders and CEOs need to balance the need for diversity against the need to keep systems running quickly and efficiently.

Consider the following example: If one engineer writes unique code that others on the team find difficult to read, that slows the efficiency of the team. With complicated tasks, such as APIs, other engineers may not be able to find their way through the unusual code. The engineering team wants the product or the service to be navigable by more than one engineer for optimal flexibility and efficiency.

Hiring the most brilliant candidate and not the most needed candidate

When hiring a new engineer for an established team, leaders want to hire the person whose expertise and experience fill a gap on the team. It may seem counter-intuitive, but that means hiring the person with the right competence, not the most over-all competent candidate.

The members of the hiring team need to know exactly what competencies are needed before advertising the position. And they should steer interviews to determine that the candidate meets the exact needs of the opening.

For example, if the team needs a front-end engineer for developing software, the ideal candidates are those who know JavaScript, CSS, and HTML. You might meet a candidate who is a genius in machine learning. But that genius is not right for your team unless he also knows the needed technologies.

Not making the engineers part of the process when hiring a vendor

When selecting a vendor, top executives will meet multiple potential companies to determine which best meets the organization’s needs. Loop the engineers in. They should have a voice in which suppliers will vendor their work. It is a good idea to put at least one engineer on the hiring team to ensure clear communication between the engineers and the vendor.

Vendors who have experience working for companies similar to yours, as well as experience on analogous projects, may do a better job for you, because they talk the same language as your engineers. When engaging a vendor, also take the trouble to read reviews from previous clients.

2. No culture of excellence

Think about the most successful corporations in the world. All of them possess unique corporate cultures.

But they have one thing in common: the drive to be excellent. To achieve excellence for the whole company, each department or team should be pursuing it.

What follows are the characteristics of teams that are not pursuing excellence.

Individual and team apathy

Every company employee and team should be passionate about their work. You will frequently hear the word “passion” from candidates interviewing for a job. They are, indeed, passionate about getting high-paying employment. Even the oldest, most reliable engineers used the word “passion” when they were interviewing.

However, there can be a decline in creativity and eager problem solving (aka “passion”) among your most experienced engineers. It is up to CEOs and top executives to cultivate passion in even the most placid employees. Studies show that the following practices improve employee engagement:

Get rid of arrogant/bad team players fast, even if they are total rockstars. Nothing kills a team’s culture and performance faster than allowing weeds to grow internally. On a side note, remember that quite often employees leave their managers, not their companies.
Praise good work
Publicly acknowledge good work
Maintain a happy, upbeat company culture
Take an interest in employees’ lives outside the office; show compassion
Share company success on a regular basis so that employees can be proud of where they work
Give back to the community, and allow employees to participate in volunteer work during office hours

Doing the minimum to get by

For any company, the end goal is to succeed and grow. The engineer’s role is to help the company succeed and grow through technology and design.

Unfortunately, some engineers are content to complete some elegant code without considering the user experience. An effective engineer will want to serve the customer. An ineffective engineer considers the customer a nuisance who just adds to his workload.

An effective engineering team cultivates customer empathy and a passion for solving problems on the user end as well as the development end. Engineers should understand the company’s big picture rather than isolating themselves from accounting, sales, and marketing.

Lacks curiosity; dislikes learning new things

New tech tools emerge daily. Each company must carefully evaluate new tools, deploying those that add value to that particular company. An engineering team that is resistant to new ideas will make itself obsolete. Therefore, an effective engineering team should create an environment for its engineers to embrace new programming languages and tools, even new technology trends. The curiosity to continue learning is the key to a lifelong career path. Once an engineering team becomes averse to learning new things, it will lose its creative edge.

3. Communication breakdowns

George Bernard Shaw rightly said, "The single biggest problem in communication is the illusion that it has taken place."

It’s easy to let bad communication and communication failures slide. But everyone in your organization needs to know the facts that relate to his/her job as soon as possible.

One example of poor corporate communication occurs when developers do not get honest feedback from their peers in customer service and support. Or when developers simply ignore the feedback.

Customer service employees and developers are on the same side. If they lose sight of this fact, the results are disastrous. The support team is in trouble when developers neglect to fix a bug or develop a new feature that is badly needed to make the product succeed. Simultaneously, developers feel lost when the support team doesn’t tell them how the new product is doing on the customer end.

When these two departments have good communication, the developers understand the bug’s context and its actions. The customer support department should not be monitoring a bug, and the developers should have enough information to know how the bug acts and how it is interfering with user experience. There should be an open and continuous feedback loop between development and customer support.

Failing to learn end-user experience

No matter how good a technology is, it will not succeed on the market if the end-users can’t figure it out. Tech teams need to train end users when there is a product update or new feature. Tech teams should plan for this training upfront and during the development stage.

Ideally, training will begin before the product or service hits the desktops. If an engineering team members do not have the right training to attract users, they need to be brought up to speed. User training should always match the standard training models. Training end users should always confirm that the product can be used as expected.

Failing to say “no”

Who wants to be viewed as the department that says "no"? Nobody. Yet, in choosing new projects, the engineering team can say "yes" too often.

Taking on too many new projects is irresponsible. An engineering team that is swamped with new projects will fail to update their peers in other departments. That may make the team seem aloof to the rest of the company.

The end result of taking on too many new projects is chaos, burnout, and misunderstanding. By not managing its projects wisely, the engineering team can put stress on other company departments.

Therefore, the engineering team should not accept every assignment from other departments or end-users. A "no" means they are striking a balance with other demands. Before complying with a request, the engineering team must evaluate the request’s urgency, its importance to the company, and whether it is even achievable. All engineering teams must have the discretion to decline a request or new project.

4. Overusing new technology

We are in an era of rapid development and innovation. It's crucial to stay on top of updates, particularly within the technology team. However, indiscriminately pursuing every new technology is not good for business.

It is risky to deploy cutting-edge or unproven technologies on a daily basis. Before bringing a new technology or tool on board, a company needs to be sure that the new tech will improve its product or its profitability.

You need to weigh the possible benefits of a new technology against the expense, the time it will take employees to learn the new tech, and possible disruptions to business from undisclosed bugs.

In other words, new is not necessarily always better. Sometimes it’s just new. Why would you switch to a new technology that is not yet proven? Especially if your existing tech gets the job done? The old adage, “If it ain’t broke, don’t fix it” may not apply to every aspect of business in the 21st century, but it still applies sometimes.

There is also a right time to adopt an emerging technology. Knowing the right time to bring in a new technology can be challenging because of the sheer speed at which new tech is being developed. It’s common for an engineering team to adopt an emerging technology too early or too late.

Adopt too early, and the team will encounter unexpected bugs, outages, and imperfect processes. Adopt too late, and the company loses the competitive edge that this new tech could have given it upon earlier adoption.

There can be no substitute for carefully evaluating all new technology for possible benefits to your specific industry and company. Take the time and energy to make sure that any new tech will repay its costs with interest.

5. Bad code review

All engineering teams need to periodically review code. But code review should occur in such a way that it fixes bugs rather than creating personal strife and office squabbles.

Let's first look at what a bad code review looks like:

Some code was poorly designed and it is slowing development. Developers are not happy. An entry-level engineer is assigned to review the code The review seems endless and takes time away from delivering new features.

The entry-level engineer who is reviewing code looks at the feedback and requests, then prepares his review. Each engineer who reads the review responds with his own nitpicky criticism.

Furthermore, the reviewing engineer has no tools with which to flag programming errors, bugs, stylistic errors, or suspicious constructs. And there is no consistency within the code. One of the reviewers prefers noun-verb naming, and another prefers verbNoun. All colleagues have different preferences.

During a bad code review, reviewers always argue about the format, such as tabs vs. spaces, and placing the brace in the same line or next line. They compete for the best one-liner or even more layers of abstraction and indirection. Reviewers cannot reach an agreement with variable, method, class, or package names. Reviewers ignore the industry standard norms, like SOLID principles, but require others to comply with their habits.

To prevent your code review from turning toxic:

Clearly establish the goals for the review. Make sure the reviewers understand what needs to be improved.
Establish protocols for fixing defects in advance.
Make code authors annotate the source code before starting the review.
Don’t throw new hires at code they didn’t have any part in writing.
Don’t review more than 500 lines of code at a time.
Don’t let an engineer spend more than an hour at a time on the review.

6. Rewriting code when refactoring is more efficient

Let's get familiar with code rewrite and code refactor. When you rewrite code, you know the flaws of the original system and what you need to fix. Yet, this means you have to maintain two systems at the same time: the old code and the new.

Also, since the old system has not been ideal, you still need to fix the bug or add new features in the old one while writing code for the upgrade. You have to do repetitive work for both new and old systems.

Code refactoring is a less drastic approach. When you refactor code, you work directly with the original code, replacing bad code and updating segments as needed. This more incremental approach means you can save on the time and energy involved in going all the way back to the drawing board.

The drawback is that you can't alter the fundamental framework or use another programming language. Engineers can't create miracles with code refactoring, but it may still be the best option if the framework code is still fully functional.

Developers often prefer to rewrite code. And rewriting is definitely the best option if the developer can’t make sense of the original code. That’s why it is imperative to read through the original code in its entirety before choosing between refactoring and rewriting.

While reading the code, engineers need to look for bugs. If it is possible to identify all the bugs that are hurting functionality, refactoring makes more sense. If the code all looks like Greek to a non-Greek speaker, then a rewrite is in order.

When determining whether to rewrite or refactor, also consider the following issues:

Time

It will take you much more time to rewrite the whole code, fix bugs, and even add new features. You should consider not only your own time constraints, but those of users as well. If client need is urgent, consider refactoring to avoid customer dissatisfaction.

Market Changes

The market is not going to hold steady while you are rewriting. If innovations in competing products emerge while you are rewriting code or rebuilding a software, clients will ask you to add the same innovation to your product. Now, in addition to rewriting an entire product, you are also madly keeping up with market innovations in both the new and old codes. You could get caught in a rewriting loop.

The advantage of refactoring is that you can clean up and respond to new market demands at the same time with minimum resources. Should a massive change happen in the market, you will have space to react with refactoring.

7. Using microservices for the wrong reasons

Yet one more trait of the ineffective engineering team is using microservices for the wrong reasons.

Engineers should not deploy microservices just because they are cool. Using microservices where there is no good reason for them is worse than being uncool. It’s possible that a more straightforward approach will give you the right results. For many businesses, writing the simplest code possible and getting to market quickly is the better course of action. Engineers sometimes use microservices because they assume they will need to scale the product for more users. But here’s the problem: No one can see the future. That means you don’t know how much (and if) you will have to scale the product or how you will scale it.

Some people consider microservices optimal, but premature optimization can hold up development of minimally viable products. At an early stage in product development, engineers should focus on developing products or services that satisfy end-users' needs rather than building stunning infrastructure.

Microservices should not be used for theoretical improvements to functionality. You don’t know what improvements you will need to make to functionality because that’s in the future. Building microservices can take time away from vital testing and other quality control practices.

Developers need to build infrastructure and develop products at the same time. Other departments might not understand the implications of microservices vs. quick product development, so it is crucial that engineers do not implement innovations they know will hurt the company’s productivity. Engineers may need to educate their peers in other departments about best practices. It is important that an engineering team not be railroaded into acting against its own best interests.

Engineers who insist on microservices create overly complex code that could cause problems later for other engineers who are now having to troubleshoot bugs in the code. Before investing up front in microservices, consider how much time it will take to create such complex architecture.

8. A bad software engineering process or the lack thereof (bonus item)

I left this as a bonus item although it’s just as important as all items listed above. From my personal experience this is actually the single most important thing that can make or break a software team.

A good software engineering process translates to quick feedback loops and lots of automation in order to catch bugs early in the process. Here are some of the core pieces present in a good process:

A solid CI/CD tool integrated and used by all team members. These days I consider CI/CD just as important and basic as a version control system. It’s hard to work properly without this.
When a team member pushes code the CI/CD automatically triggers a build of your app/libraries/systems and runs your entire test suite. Some teams go the extra mile to implement Continuous Delivery and are able to ship to production multiple times a day. Continuous Delivery is an indication of how serious you are about quality. If you are onboarding a new engineer there is nothing more appealing than being able to get him up and running and perform his first production release right on the first day (or say the first few days) and with a very high confidence that it’s going to be a high quality release.
Code Review - this is not a nice to have feature. High performing teams know how important peer reviews are and how it translates to reduced technical debt, less bugs and much better software. Code reviews should be a mandatory step before code gets pushed to production, it’s usually best to ensure each Pull Request (PR) gets reviewed to more than one team member.
Did I mention CI/CD and automation? This also means adding more layers of automation besides the basic unit and integration tests. For instance:
- Security tests such as static code scanners and library vulnerability checks. Some teams can also take advantage of security as code best practices.
- Linters
- UI tests, including screenshot comparison tests when applicable.
Going Agile, while also understanding it’s pitfalls.

Conclusion

It’s never been a more exciting time to work in software development. Engineering teams have many challenges, but also many opportunities. By tracking the above red flags of inefficiency, companies can leverage the creativity and excitement of their engineers. Happy hacking!

The Path to Speed: How to Release Software to Production All Day, Every Day (Intro)

Evandro Miquelito — Tue, 29 Sep 2020 20:22:05 +0000

One of the biggest challenges most software companies face nowadays is getting their software to market more quickly, and without sacrificing quality. Users expect more, thanks to the help of modern technology and applications that allow many software businesses to update their products at the click of a button, or even automatically. As a result, expectations are high to deliver more features to customers at a faster rate.

In order to compete, most enterprises must be able to release updated features within days, or, even hours. That means they need to stop using those hit-or-miss, cumbersome and ineffective release procedures that struggle to release an update every few months. So how can they ship high quality software even quicker? To shorten the time between idea creation and the software release date, many companies are turning to continuous delivery using automation.

Delving into Continuous Delivery

As you may know, continuous delivery is about being able to deliver changes like new features, bug fixes and configuration updates quickly and sustainably. The goal is to have the ability to perform any deployment on-demand, from apps to large-scale distributed systems, but in a routine and predictable way.

With continuous delivery, your code is deployable 24/7, no matter how many developers are working on it at the same time. By doing so, there’s no need for code freezes as the entire process is streamlined in such a way that there are enough checks and balances in place to provide high quality releases.

However, issuing software and features on a more frequent basis doesn’t mean users should get a lower quality product. The fact is, continuous delivery is capable of providing fast, consistent results, giving companies a winning edge over their competition.

The Benefits of Continuous Delivery

Continuous delivery offers the following important benefits:

Faster Market Times: It can take weeks or even months for the integration and test/fix phase of the traditionally-phased software delivery process. But when you can automate certain processes such as building and deployment, and application environment and regression testing, developers can remove integration and regression testing from their daily tasks and focus on other phases. That can also eliminate the burden of frequent re-work that can muddy the phased methodology.
Lower Risk Releases: The goal of continuous delivery is to ease the pain of delivering software. It lowers the risk of the process and makes it so developers can add features and updates at any time. Leveraging patterns like “blue-green” deployment makes it easier to offer a zero-downtime deployment that users won’t even notice.
Reduced Costs: When a service or software project is a success, it is sure to evolve over its lifetime. But by automating the build, test, deployment and environment phases, you can greatly lower the expense of making incremental updates. That is because automation removes many of the fixed costs that come with the process of software release.
Consistently High Quality: Your teams can focus their efforts on high-level testing and other activities when they have automated tools that can detect regressions in minutes. This frees them up to do usability, exploratory, performance and security testing. Developers can build a quality deployment pipeline that allows them to perform these operations continuously during the delivery process.
Quicker Onboarding Process: New team members can onboard and are more productive quicker since there’s no need to learn a complicated developing and testing process.
Better Feedback and Testing: Continuous delivery makes it more financially-viable to work on smaller batches. That allows developers to receive feedback from the working software throughout the delivery cycle. Being able to A/B test potential features with users before building them out allows teams to use a hypothesis-focused approach to developing their products. This reduces building out a number of features that lack value. This ends up saving not only time but also effort and money, too.
Lowers Team Burnout: Continuous delivery makes releases easier, lowering team burnout. Also, when you release more often, your software delivery department is able to work more actively with users. That means they can quickly and easily determine which concepts work and which don’t. Also, they get to witness the results of their efforts. By lessening those difficult, less-value activities that come with software delivery, your team can focus on more important tasks.

The fact is that continuous delivery is not always easy at first. It’s about perpetual, everyday improvements and striving for a higher quality product. The end result of time, money and effort savings makes the initial investment to implement it well worth it.

Common Stages of the Delivery Pipeline

Before you can create and automate your delivery pipeline, you need to identify each stage. Remember, every pipeline varies, so these stages aren’t set in stone. But the ones listed below are common for most software projects. Here are the main stages: build, staging and production.

Build: The build stage is where you build, package and archive the software (if applicable). It is also where you should run any unit tests. For the build stage, the input is normally a source-code repository and the output is an artifact stored in an artifact repository - oftentimes this is also where you collect output for all your unit tests, which is a critical stage. It is common to configure the build stage to react to changes in the source-code repository, for instance you may want to automatically run your test suite after pushing changes to the repository.
Staging: This is when you install or deploy the build artifact to the staging environment, which is a clone of the production environment. This is also when you automate tests to check the new environment, as well as do integration or functional tests on the new capabilities - some teams go as far as adding a layer of security automation/checks in this step.. Most importantly, this is where regression and performance tests take place to ensure the new version doesn’t affect any other capabilities. Staging deployments are usually triggered by simply merging code into a “staging” branch name (or whatever branch name is used to represent the Staging code).
Production: The production stage is when you install or deploy the software into the production environment. Additional tests take place to make certain that the new version is running as it should. Certain techniques, which are known as "blue-green" or "red-black" deployments can help prevent downtime during deployment. Similar to Staging, production deployments are usually triggered by simply merging code into a “production” branch.

How to Deploy the Process of Continuous Delivery

Now that you know the benefits and stages, here are three steps to start using continuous delivery in your operations. Here are some tips on setting goals you can measure, as well as leveraging automation for continuous delivery and testing your product to perfection.

Step 1: Ask the Tough Questions and Set Measurable Milestones

The first step is to set milestones or goals that you can measure with ease. To do that, you need to ask yourself some tough questions, including:

How will you produce high-quality software that doesn’t hinder the user experience while considerably speeding up the development process?
What are the areas you need to automate the most or the soonest?
Which method and tools will you use to add automation to your processes?
What are the financial implications and long-term ROI for implementing automation?

Think about and list the milestones you want to reach to get to your ultimate goal. There’s no need to make your goals perfectly align with what you have heard or read about continuous delivery, DevOps or other practices and methodologies. Tailor it to your specific needs, instead.

Your focus should be on creating small milestones for building the best infrastructure. It should be an automation process that allows your DevOps team to build out, test and provide small increments of features or updates that will perform in the finished product.

Step 2: Add Automation for Continuous Delivery

The road to the software market is full of problematic areas that you can improve on with automation. They include automating application provisioning, configuration and deployment. You can initiate continuous delivery and DevOps-type projects by incorporating the following tools throughout the production cycle:

Continuous integration and application release automation.
On-demand virtualized environment provisioning.
Measuring test automation.
Configuration of virtualized environments.

It is possible to use continuous delivery to automate the build, testing, deployment and environment provisioning phases all the way to production. However, if you work for a larger organization with multiple departments and teams, you still may need both manual and automated tools. But no matter the size of the project or company, you have to be flexible enough to adjust your plan in an auditable and moderate way.

You want to produce a continuous delivery production process or pipeline that is fully-automated and streamlined to support your DevOps team. So, instead of just matching each of your specific tasks with your automation technology and tools, be sure to also use it to enhance team collaboration and project management.

Adding Automation: New Projects Vs. Existing Ones

If you happen to be involved with a "greenfield" project, build and automate your delivery pipeline before you write a lot of the feature code. Create a basic "hello world" application in the project’s programming language. Next, identify each stage of your pipeline, so you can implement automation from one end to the other. This way, you can write your feature code much faster and confidently because you can automatically build and test every bit of code.

If you are involved in a project that has already started, look at what automation is currently in place. Find the places where the biggest “clogs” take place in the delivery pipeline. They are where automation could make the most difference.

Step 3: And Finally, Test, Test and Retest

When you go from releasing products and updates once a month or once a week to multiple times a day, there just isn’t enough time or manpower to do manual testing. Even if it’s just basic regression testing, it would be impossible to keep up, at least not accurately. Imagine how much harder it would be to check new features while maintaining that level of speed and accuracy, even with more testers.

The answer is automation. One of the most beneficial and important features of an automated continuous delivery pipeline is automated testing. Be sure to plan out your testing strategy for all the stages. Also, include both non-functional and functional testing. A Test Driven Development (TDD) strategy might be a good idea for some teams.

Product testers and developers must closely collaborate to identify each repetitive test, like performance, security and code, and then automate it. But you’ll also want to think about what your users want in terms of features, playability and more.

Why It Takes a Team to Build an Automated Delivery Pipeline

Because creating a continuous delivery pipeline takes a lot of effort and money, it can affect your entire company’s productivity. For that reason, you should manage it as a separate project, just the same as you would when building a software project for a customer. That’s because it requires the same attention, discipline, quality and care as any other software development project. So depending on the number and size of projects your company handles, you should create a team dedicated to setting up, monitoring and maintaining your automation tools. Avoiding some common software development wastes is also a good idea.

Low quality and unreliable automation will waste more money, time and effort than it saves. Your team should be able to trust their testing tools, so they don’t have to spend too much time debugging flaky tests and mistakes. As your delivery pipeline evolves, you’ll be able to use it for more and more projects. A team that focuses on maintaining and optimizing code to keep it working to speed up delivery is priceless.

Remember, automation and tools are merely a means to an end. Every day, newer, upgraded technologies, tools and techniques arrive nearly every day. So, try not to purchase an automation tool just because it’s something new. You want to make sure it works well with the other automation tools in your delivery pipeline. On the other hand, avoid getting stuck in a rut. Stop using old automation tools that no longer work for the productivity and delivery of your product.

Keeping the Flow Going Between Stages

In the most basic delivery pipelines, the stages are run sequentially. That means an error in one stage will hold the next stage back. In a more complex pipeline, there are multiple instances of some stages. For instance, you could have a production stage for each data center or region. That means you will want to run each production stage in parallel. Most production runs are often secured by a manual approval step to prevent accidental deployments as well as for auditing purposes.

To provide the quickest feedback, the early stages should be as basic and run as fast as possible. The later stages should run progressively more complicated tests in a progressively more production-like environment. Remember, it takes longer for the later stages to run. Also, to run in a production-like environment, they need more dependencies. However, as they pass every successive stage, it helps to raise the confidence to be ready for release to the market.

Choosing the Right Tools to Build Your Delivery Pipeline

Building the most basic delivery pipeline means finding several automation tools and platforms. With so many automation tools, your team needs to have the know-how to automate them to work well in conjunction with each other. You will also need someone who can monitor and maintain the delivery pipeline. Discuss all the available tools with your team to determine the best ones to use.

The tools you choose also depend on the categories you plan to automate. Here are some common categories in automated delivery pipelines:

Build Tools: Make, Ant, Gradle, Yarn and Maven.
Continuous Integration (CI) Server Tools: CircleCI, Travis-CI and Jenkins.
Source-Code Management Tools: Git.
On Demand Test Environments: Squash.io
Configuration Management Tools: SaltStack, Chef, Ansible, Puppet and SaltStack.
Testing Framework Tools: Selenium, Cypress, TestCafe. Most testing frameworks use a specific programming language.
Deployment and Provisioning Tools: Terraform, Chef and Ansible.

Consider combining several open-source tools, such as Gradle and Jenkins to form the foundation for your framework. For storage purposes, many pipelines also have an artifact repository to hold items like install packages and binaries from the build stage and other stages, as well.

Finding Your Framework: Tying All Your Automation Tools Together

One final aspect of creating a delivery pipeline is building a framework to tie all of your automation tools together. An example of that is IBM Cloud Continuous Delivery. It’s a framework that uses sets of integrated tools called “toolchains,” like GitHub and Delivery Pipeline.

If you want to use an open-source framework or tool or framework, look for one that comes with community support, such as Stack Overflow. These resources work with other tools, support multiple environments, and offer a wide selection of plug-ins or extensions. Also, you can find online sources for bug fixes and other help. You’ll find other people who know how to use your automation tools who can offer some valuable advice and information.

Conclusion

While it’s advantageous to release software to production multiple times a day, it only works if you can maintain high-quality standards. However, this is possible by setting measurable milestones, using continuous delivery, and adding automation tools for thoroughly testing your software.

These three steps will help lead you to faster, more successful product releases to market. Don’t let your competition leave you in the dust. Automate your delivery pipeline so they won’t even show up in your rearview window.

Intro to Security as Code

Evandro Miquelito — Fri, 25 Sep 2020 20:49:25 +0000

A report by the United States' Federal Bureau of Investigation (FBI) indicates that its Internet Crime Complaint Center (IC3) received over 467,000 complaints related to cybercrime in 2019. The cost of losses due to this class of crimes is estimated to be about U$3.5 billion (Source). An analysis of the numbers from the FBA shows that the losses had doubled since 2015 when they stood at $1.1 billion. This indicates that something needs to change if the security of company and customer assets is to be improved. The situation calls for a shift in thinking, right from early stages of the software development process.

For organizations dealing with code, it's time to ensure that software developers have the skill to design code that will be a step ahead of the criminals. This may be easy to say, but in practice, it will require a commitment to change our way of thinking. We will have to ditch the way of thinking where testing the security of code is something done right at the end - like some kind of an after-thought. The silos will have to fall, and security will need to be taken seriously by developers.

In this article, I want to suggest how organizations can change their way of thinking so that they can protect their assets and those of their clients. I will start by looking at the idea of DevOps and traditional security practices. My focus will be on how we can begin to think of security as code and get developers to consider security and security practitioners to think like developers.

Security in The Age of DevOps

To understand the shift into an era where security and code are seen in a new way starts with comprehending the idea of DevOps. Between practitioners and academics, there is no coherent definition of what the concept of DevOps represents. However, there is a loose agreement that it denotes a concept where software development (represented by the Dev. part) and IT Operations (represented by the Ops) part are combined.

The aim is to reduce the time required to develop systems while ensuring that the delivery of software is continuous and of high quality. In practice, the idea can be divided into segments. These include product delivery, ongoing testing, quality testing, and development of features, and maintenance releases. The aim is to ensure that the released software is not only reliable, but it is also secure. All this has to be done in less time than the traditional methods. The best DevOps teams are able to ship high quality software multiple times per day.

Adopting a DevOps attitude involves commissioning cultural changes in an organization. Organizations that embrace this framework will have to reconsider their operations, the roles of its developers, operators, and testers during the development and delivery of software. These groups will need to be trained to work in collaboration. This will involve dismantling the silos that are currently a feature in many organizations.

In brief, I could say that the idea of DevOps sees security as code. This is the idea that I want to look at in this article. I aim to help an organization that seeks to shift into this way of thinking to identify some of the elements that it will need to address.

It’s Not Business As Usual

To create the paradigm shift into a DevOps way of thinking, your organization will need to start by understanding the prevailing practices and that something needs to change. In most companies today, the software development cycle begins with the development team getting a brief about what needs to be done. Off they go to do what they know best. Once they are done, the security team needs to review the new product to ensure that it does not have any security issues. This usually takes place a few weeks before the application is released.

As expected, the security team will identify several concerns that will need to be addressed during this process. With just a few weeks left before release, everybody goes into a panic mood because it becomes clear that there will be no way the deadlines will be met. At this stage, the company can either go ahead and release the application with the security glitches or postpone the release. Both choices will cost money and leave a few people unhappy.

As you can see from the above scenario, the two teams in the organization (development and security) are concerned about different things. The development team wants to see their product going to the market within the deadlines. On the other hand, the security teams are aware that it will be impossible to release the software before the security issues are resolved. This is where the DevOps mentality comes in to assist.

Dismantling the Silos and Thinking of Security as Code

To solve the situation where the teams working on the development of software are reading from different pages, we need to start thinking about security as code. But how do we do that? We need to understand what the idea of security as code means.

Thinking of security as code is a framework where we start to make security part of the DevOps workflows and tools. This will involve changing systems so that we make changes to infrastructure and code so that security is incorporated into various stages of development. This means that the security team will not only see the code when it is finished but also during the entire development cycle. The collaboration between the teams will ensure that security problems will be identified and resolved early.

It is essential, however, to note that thinking of security as code should not add layers that will make the process more expensive and time-consuming. How do we ensure this? We ensure this by changing our security policies. The result of these changes should be the incorporation of tests into the pipeline and code itself.

When security tests are introduced at the time when the code is written, both time and money will be saved. If such a process is managed correctly, issues will be identified as they develop and corrected. This is better than having them accumulate so that they all need to be resolved just before the application is released.

When security is seen as code and defined right at the beginning of the product development cycle, it becomes codified so that development teams can use it in the future. This creates a self-service solution for developers who want to make sure that they are creating secure code. This not only makes the process more efficient, but it also ensures that the process is less expensive. There are no unnecessary delays in releasing the product.

Introducing Security into Code

Now that it's clear that there are benefits to thinking about security as code, where do we start? We need to ensure that all stakeholders understand the shift in thinking. This will be achieved by setting up practices that will be followed by all teams. Having appropriate policies in place will ensure that everyone knows the secure coding practices to follow. I will look at some of the practical measures your organization can implement.

Keep It Simple

One of the reasons why people, in general, tend to ignore rules and guidelines is that they tend to be challenging to follow. In the same way, if you make your practices complex, likely, the people that are supposed to follow them will simply ignore them.

Part of the process of simplifying things is to have as few tools and processes as possible. Examples of how you can do this include reusing those components you have come to trust and ensuring that approaches are centralized by making them fundamental parts of the design.

It's also essential to make sure that the security tools are integrated into environments that developers are already used to. Examples include bug tracking systems, their IDE, source repository, and build environment.

Outline Security Requirements First

To ensure that everyone is clear about the security issues that could impact the final product, you will need to define these requirements early. This will include determining the threat modelling issues.

Outlining the security requirements will involve not only defining the security requirements but also codifying them right at the start of the project. Store these in a source code repository.

To ensure easy access to the requirements, security policies need to be automated. To evaluate an application's security policies, users click a button at any phase of the development. This is a fundamental step in thinking of security as code as it ensures ease of reuse of protection for different applications within an organization. This situation also ensures that there is no need for each application owner or department to define their separate policies; a time-consuming process.

Ongoing Compliance

The paradigm shift into thinking of security as code will rely on seeing continuous compliance as a non-negotiable item. Start by ensuring that your policies and rules are clear to all stakeholders, making sure that everybody understands why they need to do something.

For example, compliance measures can include ensuring that codes are checked against specific standards such as well-known threats modelling. Other actions could involve having all codes being reviewed by peers before committing them to the primary repository. These are examples of the front line of defence that you could implement to streamline security into the coding process.

It will be easier for stakeholders to comply with the processes if they know them, and they are clear about why the processes need to be followed.

It will not be simple

Of course, from the way I put it in this article, it may look like a simple thing to create a mind shift where security starts to be seen as code. However, changing the way of doing things is not always going to be easy.

Developers writing code have a lot of pressure. There are several things they will need to focus on. For instance, they have to implement the feature, write code that is clean, testable, and maintainable, and other such tasks. Even though they may be keen to embrace secure coding practices, they may not have the time for it. All these are issues you will need to address as you patiently take your team on a new path.

Time to Shift Left

As can be seen from this article, if organizations are to secure their assets and those of their customers, they will need a paradigm shift. The old days of looking at security as an afterthought at the end of the development process are gone. We need to start seeing security as a fundamental element at the beginning of the application release. This is called shifting left. It denotes a situation where teams are being required to move faster and testing comes earlier in the process.

An organization that shifts left is likely to be better at identifying problems. This will reduce the time required to deal with these issues before an application is deployed. Such a mind shift ensures that DevOps teams have the chance to authenticate security needs in shorter periods. Thinking of security as a code can facilitate this process. It assists in ensuring that the secure deployment is automated, a situation that does not only make the process easier but faster too.

While I support the idea of seeing security as code, there is a caveat. In essence, there are a lot of other issues that still need to be taken into account. While the idea will help in ensuring the security of code, no organization should make the mistake of thinking that this is the magic bullet that will solve all its security problems.

Security is a complex challenge that will not be fixed by one element. Hence, it is essential to realize that while thinking of security as code is a big step in securing released applications, it forms only one of the pillars. The real answer lies in creating a culture of sharing of information between the teams, defining clear deployment policies, and making it simple for stakeholders to be compliant. It involves automating processes so that users have access to the security protocols of any application and looking for ways of continuous improvement.

The issue with Monorepos

Yuelin Wen — Fri, 03 Apr 2020 20:49:50 +0000

A monorepo (also called a monolithic repository) is an arrangement where a single version control system (VCS) repository is used for all the code and projects in an organization. The alternative to monorepo is having multiple repositories for different projects or even functionalities. In that case, the system will be called a polyrepo. In this article, we will define the idea of a monorepo and identify some of its benefits and some famous companies using it. Finally, we will focus on whether a monorepo is suitable for everyone.

By the end of this piece, you’ll notice that even though the monorepo system is used by many tech giants such as Google, Facebook, and Uber, it still has its challenges. I found the observation by Google's spokesman interesting in this regard. She acknowledges that using monorepo isn't for everyone. But how does she support that argument? I’ll be focusing on some of her reasons by looking at the advantages and disadvantages of the system.

Why Some Tech Giants Employ a Monorepo setup

The fact that some of the biggest tech giants of our time are using a monorepo system implies that it has some benefits. What could be some of the great attributes of this setup which attract them to it? Let's look at the leading three.

Extensive Code Sharing and Collaboration Across Teams

The main advantage of a monorepo is that all code exists in a single repository. Therefore, team members across an organization work in the same place. This results in a lowered sense of ownership and blurred boundaries. Consequently, there is an increased likelihood of code reuse, less code duplication, and more teams working on shared infrastructure.

The unclear boundaries which such a system creates make changes made by other developers feel less invasive, a situation that fosters more interaction among developers and teams. As a result of these advantages, more useful libraries build-up and more projects get to share the same dependencies.

Simplified Dependency Management With a Single Version of Truth

Since a monorepo system places all the code in the same place, everything is built and updated at the same time. There is no need to reach out to other repositories and constantly check for potential breaks. For instance, technical debt generates in polyrepo when changes occur, but dependency updates don't immediately follow. In a monorepo, changes in a dependency immediately lead to an update of the dependent code and no technical debt is accrued, assuming best practices are in place.

In a monorepo, library updates are also made easier when files are not hosted in different repositories. Moreover, because everything is updated constantly, there will be only one version for all the dependencies, avoiding the diamond dependency problem. The diamond dependency problem occurs when two projects depend on one dependency. However, the challenge comes about when the projects depend on different versions of the dependency. The diamond dependency makes release without breakage very difficult. In certain instances, it can make the building of a project impossible.

Atomic Changes/Large-scale Refactoring

When developers have one repository, they search through one database to make atomic changes across the system. On the same basis, code refactoring is made easier since developers do not have to locate different repositories. The layout of the codebase will be organized in a single tree so that developers can easily trace dependencies. For instance, Google gives each source file a unique identification by assigning the file path as their namespace.

If the Tech Giants are Using a Monorepo Setup, Shouldn’t We All?

So, if the monerepo system is employed by some of the biggest tech giants in the world, shouldn’t we be falling over each other to use the same system? Aimee Lucide, a mobile engineer on Uber’s Driver Signups team, seems to have provided the answer. She gave a speech about how Uber started from a monorepo and then switched to a polyrepo. However, Uber would later switch back to monorepo when the business grew huge and successful.

In her presentation, Lucide explained that the reason the monorepo became the rideshare company’s ultimate choice is that their system comes with an Android app studio, it has a single source of truth, and it makes code sharing easier. However, she also admits that her team quickly encountered some serious problems with the monorepo when their codebase scaled. As Uber’s mobile codebase got bigger, Aimee explains, her team started to experience an integrated development environment (IDE) lag. Eventually, developers could no longer scroll without the code freezing up. She explained that every time they tried to build, git slowed down. The reason was that the whole exact clone would have to pull out.

When the codebase is huge, it consumes a huge portion of the developer’s time when they try to work on a project; even the tiniest one. When everyone is working on a single repository, a broken master sometimes occurs. Broken master happens when one person accidentally breaks the code. Consequently, others get unexpected errors that they can’t figure out.

So why did Uber switch back to a monorepo while they still didn't have the answer to the problems encountered on the system? Uber got big enough to throw its resources at the problem. In Lucide’s own words: “As your company starts to get bigger, you can actually invest resources to mitigate some of the original downsides.”

From Lucide’s answer, it is clear that the monorepo challenge persists. What’s different is that Uber can now throw money at them. So, the real question for anyone who wants to use a monorepo is simple: Are you willing to invest as much as Google, Facebook, Twitter, and Uber? For many of us the answer is equally simple: No.

What’s Good For Google is Not Good For All

In the same way buying out Chelsea Market may not be as much a smart move to you as it could be to Google, the benefits of a monorepo to Google may just not be advantageous to you. As noted above, the tech giants are able to use the monerepo system because they have the resources to throw at the challenges. Hence, one could say the reason is not based on the fact that the system has more benefits for everyone.

Let us take a closer look at the weaknesses of some of the supposed benefits that the tech giants like Google see in the monorepo system. This will make you realize why a monorepo may not be for everyone.

Extensive Code Sharing and Collaboration Across Teams

Because a single repository hosts all the projects, code reuse must be more frequent, and code duplication must be lower right? This is not always the case. Why?

Even in a medium-sized monorepo codebase, much like the case for Uber, mentioned above, it doesn’t really make sense for a developer to grab the whole extract clone of the entire repository. Not only does it take a long time, but it also creates a lot of unnecessary trouble in either searching or editing. Therefore, there are typically two methods required for any monorepo that hopes to scale.

Virtual File System (VFS) that Enables Developers to Access Only a Portion of the Code Locally

According to Rachel Potvin, engineering manager at Google, Google has about 2 billion lines of code power. She was speaking at a Silicon Valley conference. This is for all the Google internet services such as Google Search, Google Maps, Google Docs, Google+ and so on. Despite years of experiments, there was still not an optimal open-source version control system that supported their code in one repository.

To solve the challenge, Google built its own version control system called Piper. This system came with features that help developers to browse and change codes in this vast pool of lines. One feature mentioned by Potvin in her Silicon Valley conference presentation which is used by developers to access Piper is CitC. CitC allows developers to browse through the repository without having to clone or sync. While only being able to modify files stored in the developer’s machine, developers can browse and edit files freely across the whole repository.

Sophisticated Source Code Indexing and Tooling for Code Searching and Discovery

Because of the complexity of a monorepo version control system, a tool to help perform a search across the entire repository is crucial. Google invested in building a custom plug-in for the Eclipse IDE to make handling billions of lines of code possible. The Code-indexing of Google also supports static analysis cross-referencing. These are the type of investments required when codebases scale in a monorepo.

Even with tools that satisfy the requirements, developers will only get to access small portions of the code when working on a project. How different is it to view a portion of the tree using a Virtual File System when compared to looking at different repositories? Tools for easily iterating over many different repositories and collecting the results are readily available. The scaling of the codebase, to a large extent, makes working with a single VCS much similar to working with multiple repositories because of the tool requirements. Collaboration and code sharing has, in fact, more to do with engineering culture than ways to store files.

Simplified Dependency Management/One Version of Truth

It is unreasonable to believe that since a monorepo has all code in a single repository and everything is built at the same time, dependency management is unnecessary. At scale, huge efforts are required to help update on a large scale and run automated tests to ensure code health in place of the usual dependency management in polyrepo.

Because it is easy to create dependencies, developers put less effort into establishing well-thought-out APIs. Abandoned projects can also be under the radar and continue to be maintained and updated, leading to an unnecessary loss in productivity. How do the giants deal with this challenge?

Google invested in creating new tools beyond standard tools and human efforts to deal with codebase complexity, underused/unnecessary dependencies, and code discovery. On top of that, more investment is put into making tools and indexing. This creates better search and browsing.

Moreover, not all the projects are deployed at the same time, previously deployed software can be redeployed if needed. This means that in a monorepo, multiple versions of code exist, build artifacts will have to be carefully tracked, much like what happens in polyrepo, where different versions of dependencies need to be reconciled and managed.

Atomic Change/Large Scale Refactoring

A monorepo system may be counter-intuitive. Editing or searching across the whole codebase at the local machine is a much more difficult task than you might expect.

While implementing a very sophisticated VFS, as mentioned above, makes using monorepo or polyrepo the same, making changes to a fundamental library API and having it reviewed by affected teams can be impossible because of the merge conflict in the case for a monorepo. When it happens, developers, much like they do in a polyrepo, will do one of two things: working around the API issue or deprecate the current API and replace it with a new one.

To make refactoring across large amounts of codes, automated tooling for refactoring will need to be in place to make the whole process possible.

More Issues to Look out for in a Monorepo

Anyone who wants to use a monorepo system will need to be conscious of other issues with the system. I focus on some of the leading ones below.

Tight Coupling

Because of the nature of monorepos, certain issues can be experienced. These include unclear boundaries, less sense of ownership, and collaboration. Consequently, projects in a monorepo are prone to depend on each other, creating tight coupling. This leads to brittle software. Brittle software happens when any small fixes can imply alternation in many other parts of the codebase making maintenance impossible without fracturing the whole system.

A polyrepo, on the other hand, can mitigate this kind of issue by offering defined boundaries and responsibilities to each team.

Broken Master

Broken master is also a result of the unclear boundaries in a monorepo. All failing tests on the master are considered to be a broken master. The broken master can seriously reduce productivity if left unsolved. Therefore, before any monorepo projects start, ways or tools to eliminate broken master are required.

The broken master does a lot less harm in a polyrepo because a break in one or two repositories doesn’t have huge impacts on others.

Is it Good for Everyone?

The answer to our question regarding whether a monorepo is good for everyone is simple: No.

No one can question the fact that a monorepo system has its advantages. That is the reason why big tech companies line up to use it. However, there are some who believe the amount of effort put into simply keeping projects in a monorepo VCS running is impractical for smaller organizations. I will leave you with some questions if you may want to switch to the monorepo system. How much is the sense of collaboration worth to your business? Is your company willing to invest more to get the benefit of the system? After all, like Rachel Potvin said in her Silicon Valley conference presentation, monorepo is not for everyone. Therefore, corporations have to look at their specific contexts, think hard, and select the system that best meets their unique needs.

24 influential books programmers should read (2020 updated)

Yuelin Wen — Thu, 02 Apr 2020 22:31:01 +0000

The fast-paced world of programming demands that developers remain up-to-date. In fact, getting ahead of the curve makes a programmer stand out in his professional field. Don’t let yourself fall behind. Innovate so you can offer your company the kind of programming tailored for their needs. Here is a curated list of books that can support any software engineer.

Part 1: Books for Writing Code

Code Craft: The Practice of Writing Excellent Code (Pete Goodliffe)

All programmers know how to write code that works. But only the top coders and programmers can create outstanding code - clean software code that is comprehensible and lasting. Code Craft can help you join their ranks. Move beyond just writing whatever works and write outstanding, award-winning programming. This book teaches the best code presentation style, problem solving techniques, variable naming, error handling, and security.

Code Craft takes readers beyond their own problems and forces them to look at reality in order to evaluate the best way to move forward. The real world requires a level of skill in teamwork, documentation, communication, and process. The advice given is accessible to all programmers and is based on practical experience. Questions and answers at the end of each chapter allow readers to reinforce the ideas presented to them.

Solid Code (Developer Best Practices) / (Donis Marshall)

Efficiency is key in any code developing session. Solid Code looks to help programmers create robust, bug-free code. Two Microsoft .NET experts teach valuable lessons with examples based on real world problems that programmers have to face every day. Learn to navigate expenses and teamwork on a regular basis. Improve your designs, prototypes, debugging, and testing through the myriad of tips offered in this programming book. Start to create incredible, world-changing products and code.

Solid Code teaches you to:

Use metaprogramming to reduce code complexity, while increasing flexibility and maintainability.
Treat performance as a feature - and manage it throughout the development life cycle.
Apply best practices for application scalability.
Employ preventative security measures to ward off malicious attacks.
Practice defensive programming to catch bugs before run time.
Incorporate automated builds, code analysis, and testing into the daily engineering process.
Implement better source-control management and check-in procedures.
Establish a quality-driven, milestone-based project rhythm—and improve your results!

The Clean Coder: A Code of Conduct for Professional Programmers (Robert C. Martin)

In The Clean Coder: A Code of Conduct for Professional Programmers, legendary software expert Robert C. Martin advises starting programmers to code with discipline and technique, instead of simply following directions. He crammed each page with enormous value: learn how to estimate, code, refactor, and test with ease.

But Martin does so much more than just teach rudimentary technique. He also explains the importance of attitude. The Clean Coder focuses on approaching software development with honor, self-respect, and pride.

Coders will learn to:

Work well and work clean.
Communicate and estimate faithfully.
Face difficult decisions with clarity and honesty.
Understand that deep knowledge comes with a responsibility to act.

Readers will also learn:

What it means to behave as a true software craftsman.
How to deal with conflict, tight schedules, and unreasonable managers.
How to get into the flow of coding, and get past writer’s block.
How to handle unrelenting pressure and avoid burnout.
How to combine enduring attitudes with new development paradigms.
How to manage your time, and avoid blind alleys, marshes, bogs, and swamps.
How to foster environments where programmers and teams can thrive.
When to say “No”–and how to say it.
When to say “Yes”–and what yes really means.

Debugging the Development Process: Practical Strategies for Staying Focused, Hitting Ship Dates, and Building Solid Teams (Steve Maguire)

Steve Maguire tells the story of how his Microsoft software teams learned how to develop the best software without wasting a dime or a day. Some of his methods could be considered controversial, but they worked for Maguire. He does not hold back about what succeeded and what failed for his team at Microsoft.

Maguire also details how to take the energy that one programmer might have for his job and channel it to the whole team to guarantee effective, enjoyable work.

Other lessons include:

Why you might want to kick your star programmer off your team.
How to avoid corporate snares and overblown corporate processes.
How to make tiny changes that produce major results.
How to deliver on schedule without overwork.
How to pull twice the value out of everything you do.
How to get your team on a creative roll.
How to raise the average programmer level at your company.

Refactoring: Improving the Design of Existing Code (Martin Fowler)

As the application of object-oriented programming has become commonplace, a new problem confronts the software development community.

The market has been flooded with poorly designed software that have been developed by programmers who are not as experienced. This means the applications are not only inefficient, but overall a waste of money and intellectual resources. Only now developers are realizing how difficult these systems are to work with. Books for software engineering have not addressed this problem as much.

Expert-level programmers have found better ways to solve these problems with complex yet brilliant methods meant to overwrite the performance of the program but uphold its structural integrity. These practices have not been passed down to programmers who follow in their footsteps. This book seeks to change that.

In Refactoring: Improving the Design of Existing Software, Martin Fowler breaks the silence. The programming book demystifies these master practices and demonstrates how software practitioners can realize the significant benefits of this new process.

Code Complete: A Practical Handbook of Software Construction, Second Edition (Steve McConnell)

This is widely considered as one of the best practical guides to programming, effectively demonstrating the art and science of software development. The author, Steve McConnell, integrates many techniques and must-follow principles and turn them into actionable guidance. He pulls them from research, academia, and daily business practice.

This guidance is suitable for readers with various experience levels, different backgrounds, or different project sizes. Readers can learn to develop the highest quality clean software code.

These are the main topics in this book:

Design for minimum complexity and maximum creativity.
Reap the benefits of collaborative development.
Apply defensive programming techniques to reduce and flush out errors.
Exploit opportunities to refactor—or evolve—code, and do it safely.
Use construction practices that are right-weight for your project.
Debug problems quickly and effectively.
Resolve critical construction issues early and correctly.
Build quality into the beginning, middle, and end of your project.

Clean Code: A Handbook of Agile Software Craftsmanship (Robert C. Martin)

Clean Code is divided into three parts. The first teaches the necessary elements of clean software code: principles, patterns, and practices.

The second part is a series of case studies detailing the problems and solutions in varying order. Each case study is an exercise in cleaning up code: you will learn to transform a code base that has some problems into one that is sound and efficient.

The third part is a study in thought: a contemplation of heuristics that were a part of the case studies but require more understanding. This programming book is a must for any developer, software engineer, project manager, team lead, or systems analyst with an interest in producing better code.

Readers will come away from this book understanding:

How to tell the difference between good and bad code.
How to write good code and how to transform bad code into good code.
How to create good names, good functions, good objects, and good classes.
How to format code for maximum readability.
How to implement complete error handling without obscuring code logic.
How to unit test and practice test-driven development.

Code Simplicity: The Fundamentals Of Software (Max Kanat-Alexander)

Good software development results in simple code. But because the value of simple code was not recognized until recently, there is almost no simple code in the digital world. Programmers were encouraged to flood the market with their incomprehensible code. This book seeks to change that. It is a guide meant to help programmers understand what is needed to make good code and details universal principles that can be used for any programming language or project.

Skill level and experience don’t matter here. You’ll learn how to build and adhere to a plan for software development. You will also become better at managing your time in determining where your programming will go and what path it will take.

Learn what differentiates great programmers from poor programmers.
Understand the ultimate purpose of software and the goals of good software design.
Determine the value of your decisions now and in the future.
Examine real-world examples that demonstrate how a system changes over time.
Learn to allow for the most change in the environment with the least change in the software.
Make easier changes in the future by keeping your code simpler now.
Understand why to write tests and what tests to write.

Part 2: Books for Software Engineering

Rapid Development: Taming Wild Software Schedules (Steve McConnell)

Everyone in the technical world is looking to gain control over their development schedules. In RAPID DEVELOPMENT, author Steve McConnell addresses that concern head-on with overall strategies, specific best practices, and valuable tips to help shrink and control development schedules, as well as keep projects moving. Inside this programming book, you’ll find:

A rapid-development strategy that can be applied to any project and the best practices to make that strategy work.
Candid discussions of great and not-so-great rapid-development practices - estimation, prototyping, forced overtime, motivation, teamwork, rapid-development languages, risk management, and many others.
A list of classic mistakes to avoid for rapid-development projects, including creeping requirements, shortchanged quality, and silver-bullet syndrome.
Case studies that vividly illustrate what can go wrong, what can go right, and how to tell which direction your project is going.

Rapid Development is a real-world guide to more efficient application development.

Software Project Survival Guide (Developer Best Practices) / (Steve McConnell)

The author of the classics CODE COMPLETE and RAPID DEVELOPMENT has more wisdom to impart to his readers. Steve McConnell draws on solid research and a career's worth of experience to map the surest path to your goal - what he calls, "one specific approach to software development that works pretty well most of the time for most projects."

Nineteen succinct chapters teach how to attack and overcome the development process. Readers will learn to plan, design and manage their programming, as well as guarantee quality, testing, and archiving. Simple, elegant code is not out of reach for all programmers in the field.

Agile Software Development: The Cooperative Game (Alistair Cockburn)

Alistair Cockburn begins by updating his powerful model of software development as a “cooperative game of invention and communication.” Among the new ideas, he introduces:

harnessing competition without damaging collaboration.
learning lessons from lean manufacturing.
balancing strategies for communication.

Cockburn also explains how the cooperative game is played in business and on engineering projects, not just software development.

Next, he systematically illuminates the agile model, shows how it has evolved and answers the questions developers and project managers ask most often, including:

Where does agile development fit in our organization?
How do we blend agile ideas with other ideas?
How do we extend agile ideas more broadly?

Cockburn takes on crucial misconceptions that cause agile projects to fail. You’ll also find a thoughtful discussion of the controversial relationship between agile methods and user experience design.

This edition contains important new contributions on these and other topics:

Agile and CMMI.
Introducing agile from the top down.
Revisiting “custom contracts.”
Creating change with “stickers.”

Getting Real: The Smarter, Faster, Easier Way to Build a Successful Web Application (Jason Fried and others)

Through the example of 37signals, readers will learn business, design, programming, and marketing principles that apply to their programming. This book packs a powerful punch with simple, straightforward insights, contrarian points of view, and an outside-the-box view of software design. But don’t be fooled: this is not simply a tutorial. It’s a programming book of ideas.

Digital web developers and programmers of all kinds can find the answers they seek in this book. 37signals had wild success in five apps: Basecamp, Campfire, Backpack, Writeboard, and Ta-da Last, as well as their open-source web application framework, Ruby on Rails. All this was built in just two years with no outside funding, no debt, and only seven people (distributed across seven time zones).

This process was successful for over 500,000 people around the world who all sought to use these applications. Access the knowledge needed to gain success in the pages of this programming book.

Perfect Software: And Other Illusions about Testing (Jerry Weinberg)

Perfect Software and Other Illusions About Testing seeks to destroy negative ideas about testing and testers. Jerry Weinberg, the author of The Psychology of Computer Programming, dismisses the terrible notions that surround that step in software development. Testing isn’t a stigma, it’s necessary.

Be entertained and informed by Weinberg’s expert story-telling while learning about what is possible in the world of software testing. Fallacies and rumors are swept aside and readers learn to steer clear of common pitfalls.

Meanwhile, test strategy development that works for any project is taught here.

Topics include:

Why Not Just Test Everything?
Information Immunity.
What Makes a Test Good?
Major Fallacies About Testing
Determining Significance.
Testing Without Machinery.
and much more.

Coder to Developer: Tools and Strategies for Delivering Your Software (Mike Gunderloy)

Coder to Developer helps you excel at the many non-coding tasks entailed, from start to finish, in just about any successful development project. Furthermore, it equips you with the mindset and self-assurance required to pull it all together, so that you see every piece of your work as part of a coherent process.

Inside, you'll find plenty of technical guidance on such topics as:

Choosing and using a source code control system.
Code generation tools--when and why.
Preventing bugs with unit testing.
Tracking, fixing, and learning from bugs.
Application activity logging.
Streamlining and systematizing the build process.
Traditional installations and alternative approaches.

To pull all of this together, the author has provided the source code for Download Tracker, a tool for organizing your collection of downloaded code, that's used for examples throughout this book. The code is provided in various states of completion which allows readers to dig deep into the actual process of building software. But you'll also develop "softer" skills, in areas such as team management, open-source collaboration, user and developer documentation, and intellectual property protection.

If you want to become someone who can deliver not just good code but also a good product, this programming book is the place to start. This book is essential for building successful software projects.

Software Estimation: Demystifying the Black Art (Developer Best Practices) / (Steve McConnell)

In his highly anticipated programming book, acclaimed author Steve McConnell unravels the mystery to successful software estimation. Software Estimation distills academic information and real-world experience into a practical guide for working software professionals.

Instead of arcane treatises and rigid modeling techniques, this guide highlights a proven set of procedures, understandable formulas, and heuristics that individuals and development teams can apply to their projects to help achieve estimation proficiency.

Discover how to:

Estimate schedule and cost—or estimate the functionality that can be delivered within a given time frame.
Avoid common software estimation mistakes.
Learn estimation techniques for you, your team, and your organization.
Estimate specific project activities - including development, management, and defect correction.
Apply estimation approaches to any type of project—small or large, agile or traditional.
Navigate the shark-infested political waters that surround project estimates.

Software Engineering: A Practitioner's Approach (Roger S. Pressman)

The programming book is organized in five parts: Process, Modeling, Quality Management, Project Management, and Advanced Topics. The chapter count will remain at 32, unchanged from the sixth edition. However, eight new chapters have been developed and another six chapters have undergone major or moderate revisions. The remaining chapters have undergone minor edits/updates.

Facts and Fallacies of Software Engineering (Agile Software Development) / (Robert L. Glass)

While reading Facts and Fallacies of Software Engineering, you may experience moments of “Oh, yes, I had forgotten that,” alongside some “Is that really true?”

The author of this book doesn’t shy away from controversy. In fact, each of the facts and fallacies is accompanied by a discussion of whatever controversy envelops it. You may find yourself agreeing with a lot of the facts and fallacies, yet emotionally disturbed by a few of them! Whether you agree or disagree, you will learn why the author has been called “the premier curmudgeon of software practice.”

These facts and fallacies are fundamental to the software building field.

Part 3: Books for Mentality

The Intentional Stance (Daniel C. Dennett)

How are we able to understand and anticipate each other in everyday life, in our daily interactions? Through the use of such concepts as belief, desire, intention, and expectation asserts Daniel Dennett. The author spent twenty years developing this full-scale presentation of a theory of intentionality.

According to Dennett, humans adopt a stance,a predictive strategy of interpretation that presupposes the rationality of the people - or other entities - we hope to understand and predict.

These principles of radical interpretation have far-reaching implications for the metaphysical and scientific status of the processes referred to by the everyday terms of folk psychology and their corresponding terms in cognitive science. While Dennett's philosophical stance has been steadfast over the years, his views have grown richer and more refined.

The Intentional Stance brings together both previously published and original material: four of the book's ten chapters - (Chapters One, Eight, Nine, and Ten) - appear here for the first time the remaining six were published earlier in the 1980s but were not easily accessible. The older chapters are followed by reflections - essays reconsidering and extending the claims of the earlier work.

These reflections and the new chapters represent the vanguard of Dennett's thought. They reveal fresh lines of inquiry into fundamental issues in psychology, artificial intelligence, and evolutionary theory, as well as traditional issues in the philosophy of mind. It is one of the most crucial books for software engineering.

The Back of the Napkin: Solving Problems and Selling Ideas with Pictures (Dan Roam)

When Southwest Airlines co-founder Herb Kelleher brainstormed a solution to beat the traditional hub-and-spoke airlines, he grabbed a bar napkin and a pen. He drew three dots to represent Dallas, Houston, and San Antonio and three arrows to show direct flights. The image made it easy to sell Southwest Airlines to investors and customers. It also solved his problem.

Used properly, a simple drawing on a humble napkin is more powerful than Excel or PowerPoint. It helps crystallize ideas, force people to think outside the box, and communicate ideas in a direct fashion. In this book, Dan Roam argues that everyone is born with a talent for visual thinking, even those who swear they can’t draw.

This book shows readers how to clarify a problem or sell an idea by visually breaking it down. It promotes a simple set of visual tools that take advantage of everyone’s innate ability to look, see, imagine, and show. These tools are supported by twenty years of visual problem solving, combined with the recent discoveries of vision science.

THE BACK OF THE NAPKIN proves that thinking with pictures can help anyone discover and develop new ideas, solve problems in unexpected ways, and dramatically improve their ability to share their insights. This book will help readers to literally see the world in a new way.

The Timeless Way of Building (Christopher Alexander)

Christopher Alexander believes that the current theory of architecture in our world is bankrupt.

More and more people are aware that something is deeply wrong. But these ideas seem so powerful that critics are uncomfortable, even afraid, to openly say how they feel. Disliking what is happening might make them look foolish.

Now, at last, there is a coherent theory which describes in modern terms an architecture as ancient as human society itself.

The Timeless Way of Building is the introductory volume in the Center for Environmental Structure series. Christopher Alexander presents a new theory of architecture, building, and planning. This theory has at its core that the age-old process by which the people of a society have always pulled the order of their world from their own being.

Alexander writes, "There is one timeless way of building. It is thousands of years old, and the same today as it has always been. The great traditional buildings of the past, the villages and tents, and temples in which man feels at home have always been made by people who were very close to the center of this way. And as you will see, this way will lead anyone who looks for it to buildings which are themselves as ancient in their form as the trees and hills, and as our faces are."

Hackers and Painters: Big Ideas from the Computer Age (Paul Graham)

"The computer world is like an intellectual Wild West, in which you can shoot anyone you wish with your ideas if you're willing to risk the consequences. " --from Hackers & Painters: Big Ideas from the Computer Age, by Paul Graham

We live in the computer age. Our world, our society, is built and perfected by computer programmers and software designers. These people call themselves hackers. Who are these people, what motivates them, and why should you care?

Consider these facts: Everything around us is digitized. Typewriters don’t exist. They have been replaced by a computer. Your phone functions like a computer. So has your camera. Your television is a glorified computer. Your car was not only designed on computers but has more processing power in it than a room-sized mainframe did in 1970. Letters, encyclopedias, newspapers, and even your local store are being replaced by digital products. The world that existed fifty years ago looks very different from the world that exists now.

Hackers & Painters: Big Ideas from the Computer Age gathers historical evidence to draw a meta picture of the phenomenon of digital progress. In an easy-to-read style, Paul Graham explores the “intellectual Wild West.”

This book teaches programming best practices, including the importance and necessity for order and symmetry in software design, how to become profitable, the separation between heresy and free speech, the history of programming language and its current peak today, the open-source movement, digital design, and internet startups. It ranks highly among books for software engineering.

The Productive Programmer (Theory in Practice (O'Reilly)) / (Neal Ford)

In a software developer career, a proven method to produce the code in a more efficient manner is key. The Productive Programmer teaches methods that will tweak your current system for developing software. It is simple to read and even simpler to improve your work.

Master developer Neal Ford discusses how to work smarter, using all of the tools and resources you have to create the best code. He also shows how to stay motivated, get the most out of your tools, and avoid doing the same task over and over again. Become more valuable to your team!

You'll learn to:

Write the test before you write clean software code.
Manage the lifecycle of your tools.
Build only what you need now, not what you might need later.
Apply ancient philosophies to software development.
Question authority, rather than blindly adhere to standards.
Make hard things easier and impossible things possible through meta-programming.
Be sure all code within a method is at the same level of abstraction.
Pick the right editor and assemble the best tools for the job.

The Effective Engineer: How to Leverage Your Efforts In Software Engineering to Make a Disproportionate and Meaningful Impact (Edmond Lau)

Effective engineers have a certain system that they employ in order to get work done in a fast, efficient manner. Wouldn’t it be nice if you had the same system? Make the attempt to avoid trial and error and learn how to be a wanted commodity, not just another employee.

Edmond Lau illustrates his points with interviews of top engineers at the best tech companies. He tries to get them to answer his question: How do engineers become the best in their field? What tips should every engineer know in order to advance himself.

These stories and anecdotes were collected from engineering VPs, directors, managers, and experienced leaders at modern software companies. Some of these companies include Google, Facebook, Twitter and Linkedin. Other up and coming companies include Dropbox, Square, Box, Airbnb, and Etsy. A handful of startups, like Reddit, Stripe, Instagram, and Lyft, were also included.

From their stories, there are some lessons that readers can follow to reach success.

This book teaches readers to

Prioritize the right projects and tasks to increase your impact.
Earn more leeway from your peers and managers on your projects.
Spend less time maintaining and fixing software and more time building and shipping new features.
Produce more accurate software estimates.
Validate your ideas cheaply to reduce wasted work.
Navigate organizational and people-related bottlenecks.
Find the appropriate level of code reviews, testing, abstraction, and technical debt to balance speed and quality.
Shorten your debugging workflow to increase your iteration speed.
Use metrics to quantify your impact and consistently make progress.

The Pragmatic Programmer: From Journeyman to Master (Andrew Hunt)

Learn how to grow and develop your career by adding responsibilities and techniques to your portfolio. If you can write flexible and adaptable code, you can work anywhere. This book is a series of lessons that stand alone and allow you to pick it up from time to time to learn something new. The author uses examples to illustrate his points. You’ll learn the best practices for software development while sharpening your mental tools to use to help you rise in the field.

The Pragmatic Programmer teaches readers to:

learn how to Fight software rot.
Avoid the trap of duplicating knowledge.
Write a flexible, dynamic, and adaptable code.
Avoid programming by coincidence.
Bullet-proof your code with contracts, assertions, and exceptions.
Capture real requirements.
Test ruthlessly and effectively.
Delight your users.
Build teams of pragmatic programmers.
Make your developments more precise with automation.

Agile Shortfalls and What They Mean for Developers

Yuelin Wen — Wed, 18 Mar 2020 23:38:59 +0000

What is the best software development methodology to use? This question is the topic of hot debate during the project implementation stage. However, what you choose depends on many specific factors. To organize the flow of a software development project, there are several methodologies to use.

Two of the most widely used software development methodologies are Waterfall and Agile. While there are many devoted Waterfall fans out there, Agile has gained its own dedicated following. But Agile doesn’t work for every software project, so keep reading to find out why.

How Agile Was Born

Agile is newer than the traditional Waterfall method. Unlike Waterfall, Agile is a type of Rapid Application Development or RAD often paired with Scrum. Several years prior to 2001, people were looking for new ways to replace this traditionally-established, unresponsive framework. Several approaches of a new methodology had been invented in the 12 to 18 months before 2001, when Agile debuted. They included the following:

Extreme Programming
Crystal Methodologies
Scrum
Adaptive Software Development
Feature-Driven Development
Dynamic Systems Development Methodology

The hope was to add “agility” to the development process by breaking it up into smaller sections. The goal was to help get new products to the market quickly while gathering consumer feedback during development. Also, it made it easier to try out new features and functionalities before completing the final product.

On February 13, 2001, at The Lodge at Snowbird Ski Resort in the Wasatch Mountains of Utah, a group of 17 people proposed the principles of Agile software development. Agile methodology, instead of specific ways of doing things, is a set of rules to follow for a flexible, efficient, people-centric software development process.

Reactive Vs. Predictive: A Comparison of Agile and Waterfall Methodologies

First, Let’s talk about the Waterfall method. The normal routine of a Waterfall project has conventionally five stages. It usually goes from planning, building, testing, reviewing and, finally, into deploying. What is important in a Waterfall project is that the team cannot proceed until the current stage is over.

Also, like a Waterfall, the order is usually from planning to deploying, with no skipping of any steps in the middle. There is no way for anyone to start coding until planning is over, then no one is able to start testing until programming is fully complete. Moreover, if things don’t work out, the current status gets bounced back to the previous stage.

For example, if there is a problem about a certain function that is impossible to build, the whole team goes back to the planning stage where everyone has to stop everything and return to the conference room.

Most of the steps in Waterfall project management can take a substantial amount of time. Also, the discovery of any problems at each step can send the project back to base 1. If there are a variety of problems, the project must be repeated over and over until it runs out of money.

Worse yet, by the time the project is finally complete, the end product can be a total misunderstanding of the product owner’s needs. Or, even worse it is rendered useless or obsolete because of changes in technology.

The Waterfall method represents a plan-driven approach. That means it attempts to map out the whole detailed process of the software development in the planning phase. The goal is to make software development more predictable and efficient. Therefore, when changes happen, the Waterfall method won’t be able to react to them without starting the project all over again.

In this increasingly fast-paced world, the Waterfall method is becoming less and less effective in building software that meets the high demand. The Agile methodology, with its more reactive approach instead of predictive, is set out to thrive on the changes necessary to meet the needs in this ever-evolving world.

Scrum Vs. Agile Vs. Agile Scrum

We often hear people talking about “Agile” and “Scrum” as if they were two different things. However, we can also hear the term “Agile Scrum” used in many places. So, are they the same or are they different?

Agile is a framework that aims to foster individual interaction in the software development process and create customer collaboration. Also, it can quickly respond to changes and generate working software. There are 12 principles in the Agile Manifesto that explain in greater details the goals, the mindset and the requirements to practice agile software development.

According to the Agile Manifesto, Agile is just a framework without any specification on how to proceed with software development. However, under the umbrella of the Agile philosophy of software development, Scrum is one of the most common approaches to practice the Agile methodology.

Scrum was developed between the ‘80s and ‘90s with a focus on the management process of software development. In Scrum, the project is divided into manageable chunks, known as Sprints. Sprints are estimated to take under 30 days to be developed into the ship-ready stage. There is also a daily Scrum and burndown chart to ensure communication and monitor progress.

Agile Scrum Software Development: Common Roles, Steps and Procedures

The two specific roles in Agile Scrum software development are as follows:

Product Owner: While this is obviously the person who owns the product, it also represents the user and customer of the end product. They are also the person who sets the direction and identifies the right features to release.
Scrum Master or Project Manager: The Scrum master sets up the procedure, arranges meetings and makes sure the development process goes smoothly.

The rest of the team usually consists of executives, developers, testers and customers.

Understanding the Agile Scrum Software Development Process

This is the Agile Scrum software development process in just five steps:

Creating the User Backlog: The user backlog is a collection of features,known as user stories in Scrum because the features are written from the end user’s perspective, that will make the software great.
Identifying Which User Stories to Release: The product owners choose which user stories in the user backlog will be included in the product release.
Prioritizing User Stories and Estimate the Workload: The chosen user stories are prioritized and given an estimated workload.
Developing Sprints: User stories are combined to form a sprint. Sprints are short duration milestones that normally take under 30 days to achieve. The end product of each sprint must be a ship-ready deliverable. In each sprint, there will be four stages: plan stage, build stage, test stage and review stage.
Monitoring progress. The Scrum master will have to monitor the progress to see if the team is on the right track. That way, they’ll know if they will finish the project on time. Other ways to keep the team on track and in the know include:

Burndown Chart: A burndown Chart created at the beginning of the project lists the remaining workload. Workloads on the burndown chart should head toward zero, although some fluctuations can occur in between each day.
Daily Scrum: Daily stand up meetings for team members to report their progress, identify problems, and communicate with each other.

The Disadvantages of the Agile Methodology

Agile methodology came about because there were many things that the Waterfall methodology couldn't do. However, that doesn’t make Agile the best at everything. Here are some of the disadvantages to this methodology:

More Dedication and Time Investment

While focusing more on people and the interactions between people, Agile needs clients to commit more time and energy into the project. Clients can’t just show up once at a specific time to provide their plan and then wait for the project to be complete.

Because the project development becomes an ongoing process for the client as well, they will have to commit to giving up a portion of their time each month, sometimes each week or even each day, to the project.

Too Much Focus on the Details

Since what makes Agile efficient is short planning and quick reaction to change, sometimes the project may end up with a ton of features without a clear connection between them. Moreover, some functions are more suited for seeing the big picture in the future, which makes them shine less in the user review phase. This results in a shortsighted project. Also, the bigger projects may be too large for chopping into smaller, more manageable milestones.

A New Concept to Some Clients

Before Agile, there was already a solid-structured software development process that people had been using for years. Agile is relatively new, so it may take more effort from the client side, as well. So, it shouldn’t come as a surprise when some clients need some more help in understanding their role in an Agile development project.

Less Predictability and Documentation

Compared to the Waterfall method, which is more plan-oriented or document-oriented, Agile tends to react when a problem presents itself, rather than predicting the problem. This can leave projects with the Agile approach with much less documentation before the project even starts.

Some people argue that the key part of documentation in Agile software development is its source code. This property of Agile software development can be much harder for new members to understand. Furthermore, the team can sometimes fall off track because of insufficient details for direction before the project begins.

Tough to Maintain a High Level of Collaboration

Because Agile separates a project into blocks and assigns tasks to different individuals at the same time,collaboration can be difficult. Frequent updates and daily Scrums are crucial to the entire team.

While some still turn to the traditional Waterfall approach, Agile is a viable alternative, thanks to the flexibility at every stage of the project. The Agile Scrum development process works well, but it doesn’t work for every single project. Huge, complicated ones may be too much to break up into manageable Sprints.

However, the fixed, “all or nothing” approach of Waterfall can make it too cumbersome and time-consuming for a successful project finish. The decision on which to use often is dependent on the size of the project, as well as how much involvement the product owner wants to commit.

Agile and Waterfall: A Quick Comparison Study

Here is a quick comparison of Agile and Waterfall software development methodologies:

Waterfall works best for projects where the requirements are well-defined and have no expected changes. Agile works better when there is a bigger chance of frequent project adjustments.
Waterfall is simple to manage yet rigid because of its sequential approach. Agile is more flexible, yet not as defined since it allows changes at any stage. To mitigate that, teams should create a clear roadmap for delivering the finished product to avoid uncertainty during each phase of the project.
In Agile, the project description and each specific step of the project can change often. In Waterfall, they are definite from the start. The requirement of each step must be fulfilled before moving on to the next.
With Waterfall, the product owner only has to be involved in the beginning. But, with Agile, they must approve each phase of the project before the development team can move on to the next.

Conclusion

Will you use Agile, Agile Scrum or Waterfall for your next software development project? While it may depend on your team’s preferences, it also depends on the specifics of the project, such as the size and scope. Be sure to consider all factors in order to make the best choice.

16 Amazing Python libraries you should be using now (2020 updated)

Yuelin Wen — Tue, 14 Jan 2020 15:23:15 +0000

In this article we will get familiar with several amazing Python libraries being used by the best software teams. With the exception of HTTPX (which is in beta), the libraries listed below are being actively developed & maintained and are backed by a strong community.

HTTPX

HTTPX was developed by Tom Christine, who is a software engineer specialized in API design and development.

The async paradigm is increasingly becoming more common in high-performance modern applications, but when you are using the async paradigm, the Requests library cannot do its job very well.

Therefore, HTTPX comes to solve this problem. HTTPX is an asynchronous HTTP client based on the well-established usability of Requests and gives you HTTP/2 and HTTP/1.1 support. It gives you an API compatible with the Requests library as much as possible, and it has a tight control on the timeouts. HTTPX can also call directly into a Python web application using the ASGI protocol and it is fully type annotated. The library is also equipped with all the standard features of Requests, such as International Domains and URLs, Keep-Alive & Connection Pooling, and Sessions with Cookie Persistence, etc.

Moreover, you can use either asyncio or trio for HTTPX and apply this in high-performance async web frameworks. As a result HTTPX is able to handle a large numbers of requests.

Arrow

As many Python developers know, one way to work with date & time objects is by using the uncompleted date, time, and timezone functions in the Python's standard library and some other low-level modules. However, they don't have high performance and good usability; for example, they have too many modules and types, which is hard for users to remember and distinguish. The conversions between timezones and timestamps are also long-winded.

Luckily, a sensible and human-friendly Python Library called Arrow can help users create, manipulate, format, and convert dates. It mainly aims to reduce your imports and code for dealing with dates and times.

Arrow supports Python 2.7, 3.5, 3.6, 3.7 and 3.8, and broadly supports ISO 8601. Of course, it can do timezone conversion, and the timestamp is also a property in it.

Some features of Arrow can give you a clue of its capability. Arrow has a drop-in replacement for DateTime, and is Timezone-aware by default. Users will get easy-to-use creation options for many general input scenarios. The shift method is beneficial for relative offsets, such as offsets for weeks. It's possible to automatically format and parse strings, which saves a lot of time. One more attractive feature is that Arrow can create periods, ranges, floors, and ceilings for time frames, ranging from microseconds to years.

Python Fire

Python Fire can automatically generate CLIs for any projects. The library makes the process of creating CLIs super simple. You only need to write the functionality at the command line as a function, module, or class, and once you call Fire, the CLI that you desire is ready for you.

You probably want to test early on during the process of writing a Python library. Without Python Fire, you have to write a primary test method to run the functionality you desire to achieve. Although you can do the test in this way, you need to change the primary method whenever you have new features you want to test, updating the primary method again and again, which is time-consuming and annoying. Using Fire you don't need to change your primary test method continuously when you test your command line.

It is usually not super quick to understand a function by looking at its code, particularly when the function was written by someone else. A better way is to call Fire on the module. This feature allows you to easily inspect all module functions/methods.

Moreover, Fire can let you transit directly between Bash and Python so that you can use the unix tools at the same time.

Starlette

Starlette is a lightweight ASGI framework or toolkit for building high-performance asyncio services.

This production-ready library has many features, including support for WebSocket and GraphQL. Starlette can do in-process background tasks, CORS, GZip, Static Files and Streaming responses. All of these features have extensive test coverage and a code base that is 100% type annotated with zero hard dependencies.

Starlette is meant to be used as a complete framework, or as an ASGI toolkit, providing users with the flexibility to apply any of the components independently. Moreover, reusable components can be shared between any ASGI framework, creating an ecosystem of shared middleware and mountable applications.

Mypy

Mypy is an optimal static type checker for Python 3 and Python 2.7, and it is similar to a static analyzer, or a lint-like tool. By adding some typing annotations when you write your program, Mypy can help you type check your code and look for general bugs. Those annotations you leave in your code guide Mypy to do its job without interfering with your application because the annotations are viewed as comments that have no effects in the execution of your code.

Mypy gives developers the flexibility to decide the workflow. The purpose of Mypy is to combine the advantages of dynamic typing and static typing in the applications. Therefore, users can use dynamic typing as a backup when static typing doesn't work, such as for legacy code. When you run Mypy in your program like a linter, the errors will be reported in a compiler-style format. Mypy provides programmers with a robust and consistent check for a project and its dependencies.

Another advantage of Mypy is its learning curve, which is minimal. Most new users of Mypy should be able to annotate the code correctly for the first time. Moreover, the Mypy cheat sheet is a perfect start. One more advantage of Mypy is that It has a much lower false-positive rate compared to most static analyzers.

FastAPI

FastAPI is a high-performance web framework for API development. FastAPI is based on standard Python type hints for Python 3.6+.

FastAPI comes with many interesting features, perhaps the most important one is speed since it is one of the fastest Python frameworks available. The speed of coding is also 200% faster compared to other frameworks.

And if that wasn't enough, FastAPI can help to keep a low bug rate and can even reduce human related errors by almost 40%. The framework is easy to learn and has an interactive documentation.

FastAPI is based on open standards, for instance OpenAPI. It also comes with declarations of path operations, parameters, body requests, and security. Features for automatic client code generation in different languages are also available.

Immutables

Immutables is an immutable mapping type for Python. A Hash Array Mapped Trie (HAMT) used in Clojure, Scala, Haskell, and other functional languages.

Immutable mappings based on HAMT have O(log N) performance for both set() and get() operations, which is essentially O(1) for relatively small mappings.

Expiring Dict

Expiring Dict is a very handy Python caching library. It provides a dictionary with ordering and auto-expiring values for caching purposes. Dictionary elements have a TTL (max age) and max length, which are checked on each access.

VCR.py

Have you ever used Ruby's VCR library? It records your test suite's HTTP interactions and replays them during future test runs for fast, deterministic and accurate tests.

VCR.py is similar to Ruby's VCR library. It makes the tests with HTTP requests more straightforward and quicker. If your code is in a VCR.py context manager or decorated function, when you run it VCR.py will record all the HTTP interactions during your tests. Then VCR.py will serialize and write the HTTP interactions to a cassette, which is a flat file. When you execute the recorded code it will replay the serialized requests and responses from the cassette file.

This process has many benefits because the requests will not generate any HTTP traffic. Therefore, VCR.py can do its job offline, help to generate deterministic tests as well as significantly increase their execution speed.

When you make changes to your tests all you need to do is to delete the cassette file. Then when you re-run the code, VCR.py will record again the HTTP interactions and generate a new cassette.

Transformers

Transformers (former pytorch-transformers and pytorch-pretrained-bert) provides Natural Language Understanding (NLU) and Natural Language Generation (NLG) with more than 32 prepared models in over 100 languages and deep interoperability between TensorFlow 2.0 and PyTorch. Those architectures generally are BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet, and others.

Transformers is an easy-to-use library with no barriers to get familiar with. It is even more powerful and concise than Keras, with the added benefit that it has extraordinary high performance when dealing with NLU and NLG projects.

The goal of Transformers is to become the state-of-the-art NLP for everyone. No matter if you are a deep learning researcher, hands-on practitioner, or AI/ML/NLP teacher and educator, you can get a lot of support from Transformers. The library is very efficient as it reduces the compute costs. For example, you can share your trained models with your colleagues so that they do not need to spend extra efforts retraining models. It comes with more than 30 preset models, and over 100 languages which can save a lot of time on computing and thus help to reduce production costs.

Lastly, Transformers can use the most suitable framework for each part of your applications over time. Specifically, Transformers can use only three lines of code to train a state-of-the-art system for you. The deep interoperability between TensorFlow 2.0 and PyTorch models is an added benefit.

Modin

If you are a data scientist you are probably familiar with the Pandas library. Modin can scale your Pandas workflows by changing only one line of code. Sounds great, right?

Before proving more details on Modin we first need to introduce Ray, which is the key to how Modin works. Ray is a fast and accessible framework for building and running distributed applications. Ray is packaged with Tune, RLlib, and Distributed Training.

The relationship between Modin and Ray is that Modin uses Ray to speed up the Pandas notebook, scripts, and libraries in an effortless way. A weakness of other distributed DataFrame libraries is the regular conflicts among Pandas code. Modin can solve this problem by integrating the Pandas code seamlessly.

Using Modin, you do not need to know details about the cores of your system and the distribution of the data. You do not need to abandon the Pandas notebooks you previously used to take advantage of the enormous acceleration provided by Modin.

Moreover, the modin.pandas is an extremely lightweight parallel DataFrame. After you install Modin, there is no need to stop using the previous Pandas API because Modin can do transparent distribution of the data and computation.

Sometimes, when facing different data sizes it may be difficult to scale 1KB DataFrames to 1TB, there is often a significant overhead. With Modin, you can directly get DataFrames at both 1KB and 1TB.

Dash

If you are building web apps with complex or large datasets, Dash might be for you. Dash is very suitable for visualizing data and it also provides apps with customized user interfaces in pure Python.

Dash can even help you to build user interfaces with Python code in a short amount of time. Dash replaces many of the tools and technologies used for building an interactive web application and it does its job in a simple way.

Detectron 2

Detectron is an object detection platform. It is one of the open-source projects adopted by the Facebook AI Research group.

Detectron 2 is the second generation of this library with many performance enhancements. The library is flexible and extensible and makes training on GPU servers a very quick process. It also comes with state-of-the-art object detection algorithms, allowing developers to do advanced research without the whole complete dataset.

Detectron 2 was rewritten from Scratch in PyTorch, which is a great tool for deep learning. The vast and active community behind PyTorch is an added benefit for Detectron2 users.

With Detectron2, users can insert their customized code into the object detection system as they see fit. Under this situation, hundreds of lines of code can successfully develop a new research project, and the core Detectron2 and the brand-new research achievement can be divided clearly. Moreover, Detectron2 also supports semantic segmentation and panoptic segmentation.

The reason why Detectron2 is faster than the original version is due to moving the entire training pipeline to GPU. Distributing the training to different GPU servers makes the scaling process of large data sets much easier.

Streamlit

Streamlit is a library that can provide you with the fastest way to build custom Machine Learning tools.

The library embraces Python scripting resulting in clean code and fast prototyping. Each change in the Python code directly reruns the code from top to bottom. The cache primitive in Streamlit is consistent, immutable-by-default so users can reuse the information effortlessly.

Imbalanced-learn

Imbalanced-learn offers many re-sampling techniques for machine learning projects. This python library is widely used in datasets to show a robust between-class imbalance.

Imbalanced-learn is compatible with scikit-learn. Scikit-learn is a simple and efficient tool for predictive data analysis. It is built on NumPy, SciPy,and matplotlib. It is open source and very reusable. The library is available for Python 3.6+.

PyTorch

PyTorch offers rapid prototyping for dynamic neural networks and strong GPU support.

The compositions of PyTorch are torch, torch.autograd, torch.jit, torch.nn, torch.multiprocessing, and torch.utils. Component torch is a NumPy like Tensor library, which has the strong GPU support. Torch.autograd is a tape-based automatic differentiation library that is accessible to all various Tensor operations in torch. Torch.jit can work as a compilation stack to create serializable and ideal models. Troch.nn is aggregated with autograd as a neural network aiming for maximum flexibility. Torch.multiprocessing is useful for data loading and Hogwild training. Lastly, torch.utils has a data loader and other utility functions for convenience.

What is Test-Driven Development? (And How To Get It Right)

Evandro Miquelito — Mon, 06 Jan 2020 21:54:11 +0000

Testing is one of the most crucial parts of any software development project. When you create a robust, comprehensive software testing suite, you can detect the bugs, mistakes, and incorrect assumptions that you’ve made while building the code base. By using many different types of testing, from unit and integration testing to user interface and regression testing, you’ll be more likely to release a higher-quality final product.

However, the unfortunate reality is that testing is far too often an afterthought, or treated merely as a box to be checked. Development teams struggling to meet the pressures of deadlines and budget constraints don’t have the time to devote to testing that the software fully deserves.

In order to counteract this trend, software developer Kent Beck reintroduced the concept of “test-driven development.” This article will provide an in-depth discussion of test-driven development when building software applications, as well as the advantages and disadvantages of doing so.

The Definition of Test-Driven Development

As the name suggests, test-driven development (abbreviated as TDD) is a software development practice that places testing first and foremost in the development process. To understand the definition of test-driven development, we first need to define unit testing, which is an essential concept in TDD.

What is unit testing?

Unit testing is a software testing method that breaks an application down into small parts known as units. Each unit is evaluated individually and independently by a thorough series of tests that are written specifically for that unit. In order to assist with debugging, the unit tests should be as short and simple as possible, so that developers can more easily discover the error.

For example, suppose that we’re writing code for a calculator application, and we want to test the program using unit testing. We could write one unit test for each of the calculator’s functions: addition, subtraction, multiplication, division, exponentials, etc. We would also need unit tests for activities such as clearing the calculator screen and composing multiple operations.

The goal of unit testing is to find bugs and problems early on in the development life cycle. A unit testing suite for a large software application may have hundreds or thousands of unit tests, each of which needs to pass in order for development to continue.

What is test-driven development?

TDD is the practice of writing software by making unit testing the most important concern. TDD is a highly structured, highly regimented approach to software development. The principles of TDD are as follows:

Tests are always written before the code that will make them pass. The test anticipates the correct behavior of the code.
Development proceeds one test at a time. Once a test goes from failing to passing, the next test is written and development can continue.
Again, tests should be as simple as possible, only long enough to break the application in its current state. After the test is written, all development must focus on making the software pass the test.

TDD proceeds in cycles which are usually called “red, green, refactor.” This cycle is usually done once for each unit test that you write. The cycle consists of three stages:

Red: Write a unit test that fails due to missing functionality in the software (since the color red usually denotes failure).
Green: Write code that fixes the issue by making the software pass the test (since the color green usually denotes success).
Refactor: Clean up the code base to account for the presence of this new code.

It might be helpful to compare TDD to the process of writing a long essay. You start writing the essay by creating a detailed outline of the topics that you want to cover. Next, you write one of the sections of the essay based on the outline, and you make any necessary adjustments to the rest of the essay based on the new text that you’ve just written. By having the structure in mind ahead of time, it becomes much easier to create the final product.

How to Do Test-Driven Development

In the previous section, we discussed the main ideas of test-driven development. We’ll now provide an in-depth guide about how to implement TDD in a software development project.

1. Write a test

Naturally, the first step in TDD is to create a unit test that evaluates some part of your code base. The “unit” in unit testing is often a method, a class, or a member function of that class.

For example, suppose that we’re creating a driving simulator as part of an educational course, and we want to have a Car class to represent the car that the user is driving. This Car class will include methods such as startCar(), turnOffCar(), changeGear(), changeSpeed(), etc. It will also have variables that hold information such as the car’s current status (on or off), current gear, and current speed.

The very first test that we would write would be to create an instance of the Car class:

Car c = new Car();

Of course, this test will fail at compile time, since we haven’t written the Car class yet.

Inspiration for writing tests during TDD can come from use case diagrams and user stories.

Use case diagrams are models for how a system should behave based on the actions that a user wants to perform.
User stories are brief text descriptions of the software requirements written by the project’s key stakeholders. In our driving simulator example, two user stories might be “Users can practice parallel parking with the software.” and “Users can practice in a variety of weather conditions.”

2. Run the test

When we run our unit testing suite (so far, consisting of just 1 test), we will receive an error during compilation informing us that the class does not exist. This message provides a clue to the developers telling them how to resolve the issue.

In some cases, the error will appear during runtime and not compile time. You can use assert statements to verify that a given condition is true or fulfilled while the program is executing. You can also throw an exception to check for one or more error conditions.

As the TDD process continues, additional tests will be added to the end of the unit testing suite. Note that in TDD, each unit test should be an independent entity. In other words, no test should depend on the behavior or success of the tests that came before it.

3. Fix the code

With the appropriate error message in hand, developers can now move to fix the problem. At this step, you should focus less on writing the perfect solution and more on writing a solution that will satisfy the test conditions.

For instance, in our Car example, the code needed to fix the failing test would be a definition of the Car class:

class Car {

}

As we mentioned above, this code is the minimum amount of effort required to pass our first test. We haven’t yet defined any member variables or member functions that belong to this class—because we don’t need to do so in order to pass the test.

The additional unit tests that we write will look for the presence of the functions and variables that we need in the Car class (e.g. turnOn() and currentState). Once we write each test, we will be able to add the function or variable that it tests.

4. Rerun the test

Once coding is complete, rerun the testing suite to see if you can now pass the test. In our basic Car example, for instance, the application will create the object and then exit silently. If all goes well and you’re following the principles of TDD, all of your tests should now be passing.

5. Refactor the code

In this (optional) step, you’ll refactor the code that you wrote in step 3 so that it integrates with the existing code base. This may involve making the code more readable, separating it into more logical parts, and renaming or moving variables and methods.

6. Repeat

TDD should continue incrementally, gradually expanding the features and functionality of the software.

You may find during TDD that you can’t easily make a new test pass, or that you break previous tests with the new code you’ve written. In these cases, the TDD best practice is usually to revert the changes you’ve made, rather than waste time on a lengthy debugging process.

Note that if you’re using any third-party libraries or frameworks, there’s no need to test the functionality of these external resources. You should only test the code that you plan to write yourself. In addition, good libraries and frameworks should already have their own unit tests defined in their code base.

The 5 Best Tools for Test-Driven Development

As a popular practice in programming, TDD has no shortage of tools to help you during your project. Some of the best tools for test-driven development are:

Travis CI: Travis CI is a tool for testing and deploying your code using the practice of continuous integration, which aligns well with TDD. Continuous integration requires developers to integrate their code into a shared repository multiple times per day and verify it using an automated build tool. Travis CI includes integration with GitHub as well as many databases and services—and it's completely free for open source projects.
CircleCI: Another tool for continuous integration, CircleCI is a highly customizable offering that offers you complete control over the development and testing process. CircleCI includes support for job orchestration and caching, and it's compatible with Docker as well as any language, toolchain, or framework that runs on Linux or Mac.
Squash: Squash replaces the traditional setup of development, staging, and QA servers with a unique virtual machine for each branch of code in your repository. You can preview a fully working version of your application based on the changes that you've made to each branch. This gives you the power to quickly and easily experiment with new tests and features, scaling your usage up and down automatically as you need it.
Selenium : is a free and open source test automation framework intended specifically for web development. You can configure tests so that they simulate different desktop environments and web browsers, and automatically generate reports of the test results.
Cypress: Cypress is another open source front-end web development testing framework that runs on JavaScript. In particular, Cypress performs end-to-end testing, making sure that the flow of your web application is correct from start to finish. Web developers at organizations like DHL, Spotify, and NASA all use Cypress to write, run, and record tests.

3 Benefits of Test-Driven Development

1. Speed

For skilled developers and testers who can move quickly, one major benefit of TDD is speed. By adding tests that fail, and then fixing the code to make them pass, TDD encourages rapid iteration and progress.

TDD may seem like the slower option at first, but the initial effort you put in will pay off later on. Few things are more disastrous for a software development project than discovering that your application logic contains a major flaw that requires the code to be refactored or rewritten.

By taking the time to invest in the quality of the code base early on, TDD can save developers time and effort and reduce the risk of a project that’s failed or delayed.

2. Easier automation

Another related benefit in terms of speed is that TDD and unit testing make it much easier to automate your software testing suite.

Manual checks are often painstakingly slow, especially if you perform functional tests that require you to follow a series of steps. The functional testing process can take seconds or minutes to complete, and must be done every time that you make a change to the system.

Unit testing, on the other hand, is rapid and extremely easy to automate. While both manual and automated testing have a place in a mature, robust software testing program, automated tests can drastically speed up the process. This allows you to run more tests in the same amount of time, improving the quality of the code.

3. Higher-quality code

If you wait to test your code until later in the development cycle, it’s a potential recipe for disaster. Writing hundreds or thousands of lines of code without making a mistake or typo is highly improbable.

As human beings, even the best developers are prone to error every so often, and it’s only a matter of time before a slip-up occurs. Without testing the code at regular intervals, bugs and unexpected behaviors are more likely to be introduced.

TDD is, above all, an incremental approach to software development. In most cases, developers will write only a few lines of code at a time—just enough to make the current test pass. This “slow yet steady” philosophy provides reassurance (although not a guarantee) that your software does not contain errors.

One additional benefit of TDD is that the code itself can serve as documentation. For example, if you want to demonstrate how a function behaves if given exceptional input (such as an empty string or a negative number), all you have to do is write a test that uses this input.

3 Common Pitfalls of Test-Driven Development

1. Time-intensive

The good news is that TDD seems to have real benefits for its practitioners. A survey of multiple studies on the impact of TDD has found that it reduces defects by 40 to 60 percent, while increasing effort and execution time by 15 to 35 percent.

While TDD generally results in higher-quality code, however, it must also be acknowledged that the extra effort isn’t always worth it. The TDD process involves a great deal of overhead in the form of unit tests. Creating and maintaining a test suite, in addition to the software itself, is a significant investment.

As a result, software that is short and/or straightforward to write will likely take longer with TDD, even when accounting for the separate coding and testing stages in traditional development. Businesses that prefer to invest time in manual QA, or that lack the technical resources to implement unit tests, may not be the ideal candidates for TDD.

2. Less flexibility

The overhead created by TDD can often be stifling and paralyzing during the project when developers want or need to make changes. Poor choices of architecture, design, or testing strategy early on can be difficult to recover from later in the project. At this point, it could be difficult or impossible to alter the code base without making dozens or hundreds of existing tests fail.

With the prospect of having to refactor both the code and the test suite, developers are caught between a rock and a hard place. Because even simple changes can be time-consuming to make, you’ll need to decide whether it’s even worth it to continue with TDD at this stage.

For this reason, TDD is difficult to apply to legacy code bases. Having a skilled team of testers and QA staff who know how to write good software tests is essential if you plan to practice TDD.

3. Not a perfect solution

TDD isn’t a guarantee that your code base will be airtight and protected from bugs and mistakes. After all, the tests in TDD are written not by the computer, but by error-prone humans.

This means that it’s nearly as likely that mistakes will be introduced during testing as during development. For example, developers may forget to write a test that covers an important feature or functionality of the software, causing bugs to go undetected. A single incorrect keystroke or lapse in judgment can easily cause an issue during the testing process.

What’s more, TDD can’t protect you from errors in comprehension, which result when developers have a fundamental misunderstanding or incorrect assumption about the problem that they’re trying to solve.

Of course, writing another testing suite for the first testing suite is a silly and impractical idea. The best you can do is to take special care when writing your tests, and to review the testing code at regular intervals.

Test-Driven Development vs. Traditional Development

TDD might sound like an excellent idea, but it hasn’t always been common practice in software development (and isn’t always used even today).

According to the traditional software development model, projects should proceed in a series of consecutive, sequential stages: requirements gathering, analysis, design, coding, testing, and deployment. This concept is often referred to as the “waterfall” model, like a waterfall cascading over a series of rocks at different levels.

In fact, the waterfall metaphor is highly accurate. The water falling in a waterfall continually flows downwards, without the possibility to return to a higher level. Similarly, the waterfall model of software development generally discourages returning to a previous stage of development. All coding and implementation must be completed before testing can begin.

Seeing the flaws with the waterfall methodology isn’t difficult. If new requirements emerge halfway through the project, or you discover a critical flaw in your assumptions, you may have no choice but to start all over again. As a result, software developed with the waterfall model can be susceptible to delays and budget overruns.

Rather than the traditional waterfall methodology, TDD fits in well with the agile and lean methodologies that are currently in vogue with software developers. Agile prioritizes flexibility, adaptability, and customer satisfaction over strict rules and regulations. In particular, Agile uses iterative development, in which software is constantly released in its newest deployable state in order to get valuable customer feedback.

The lean methodology, meanwhile, originates in concepts taken from automobile manufacturing. Lean uses the idea of "just-in-time" (JIT) development, in which the components of a system are created or ordered just in time for them to be used in the final product. JIT reduces the need to maintain excess inventory and ensures that employees are always working to add value to the product.

It's not hard to see how TDD, agile, and lean share similar philosophies about software development. With TDD, code and tests are written incrementally, using the minimum effort required in order to get feedback and start a new iteration. In fact, the practice of TDD was originally taken from extreme programming (XP), a software development methodology that falls under the agile umbrella.

Test-Driven Development vs. Behavior-Driven Development

Unit testing is an essential part of TDD, and you’ll often see the two ideas mentioned in the same breath. In this section, we’ll discuss another software testing concept that’s highly related to TDD: behavior-driven development (also known as BDD). In fact, TDD and BDD are so similar that some developers assume that they’re two terms for the same thing.

Like TDD, BDD prioritizes testing above all else during development. However, BDD goes beyond TDD by posing the question: “Is this the correct way to be testing in the first place?”

BDD considers issues such as:

What parts of the code should and should not be tested
How testing should be conducted
How to understand why a test has failed or passed.

BDD encourages the use of names for unit tests that describe the expected behavior in natural language. For example, if we were testing our calculator application using BDD, we might say that a test “should (or does, or shall) return 2 when given 1+1.” The “should/does/shall” construct is extremely common for BDD test names.

Another difference between BDD and TDD is that tests in BDD evaluate the code’s expected behavior, rather than the specifics of the implementation.

For example, our Car class might have a speed variable that is initialized to 0 when creating a Car object. We’ve also written a unit test that verifies that the speed is correctly increased when using the increaseSpeed() function.

Instead of verifying that the Car’s speed is 10 after calling increaseSpeed(10), BDD would verify that the new speed is equal to the Car’s initial speed plus 10. This helps the unit test function independently of the code that it tests. For example, we can change the Car’s starting speed to 20 when the object is initialized without breaking the test.

Final Thoughts

TDD and its cousin BDD represent a departure from traditional approaches to software testing, in which tests are run only after the programming work is complete. Instead, TDD emphasizes the value of testing by binding it closely together with development.

This novel way of thinking forces developers to understand how each part of the code base should function, and helps them catch errors before it’s too late in the development process.

Being able to describe the software’s expected output and behavior has a variety of benefits. Communication improves, errors decline, and key stakeholders can be sure that their requirements for the project are being met.

DEV Community: Squash.io

Smoke Testing Best Practices: How to Catch Critical Issues Early

Intro

I. Understanding Smoke Testing

A. Definition and purpose

B. Importance in the software development lifecycle

C. Differences between smoke and sanity testing

II. Identifying Critical Functionalities

A. Analyzing the application

B. Prioritizing key features

C. Involving stakeholders

III. Designing Effective Smoke Test Cases

A. Creating comprehensive test scenarios

B. Focusing on positive test cases

C. Ensuring test cases are easy to understand and maintain

IV. Automating Smoke Tests

A. Benefits of automation

B. Choosing the right automation tools

C. Integrating automation into your development process

V. Integrating Smoke Testing into the Development Pipeline

A. Identifying the appropriate stage for smoke tests

B. Coordinating with the development team

C. Implementing continuous integration and deployment

VI. Tracking and Reporting Smoke Test Results

A. Establishing a clear reporting process

B. Monitoring test results over time

C. Communicating results to stakeholders

VII. Continuous Improvement of Smoke Tests

A. Regularly reviewing and updating test cases

B. Adapting to changes in the application

C. Incorporating feedback from the development team and stakeholders

VIII. Case Study: Successful Smoke Testing in Practice

A. The challenges faced

B. Implementing smoke testing best practices

C. Results and lessons learned

Conclusion

10 Proven Steps to Double the Speed of Your Django App

Introduction

1. Optimize database queries

2. Use caching

1. Using Django's built-in caching framework:

2. Using Memcached as an external caching tool:

3. Using Redis as an external caching tool:

3. Optimize view functions

1. Using select_related() to reduce database queries:

2. Utilizing Django's built-in caching framework:

3. Using Django's Prefetch to optimize related object queries:

4. Optimize templates

1. Utilizing Django's template caching:

2. Utilizing Django's template inheritance:

3. Creating custom template tags for complex logic:

5. Enable Gzip compression

1. Enabling Gzip compression using Django middleware:

2. Enabling Gzip compression using web server configuration (e.g., Nginx):

6. Use a Content Delivery Network (CDN)

1. Utilizing a CDN with Django:

2. Utilizing a CDN with a web server (e.g., Nginx):

7. Optimize database connection management

8. Use asynchronous tasks

1. Offloading tasks to Celery:

2. Offloading tasks to Django Channels:

9. Optimize server configuration

10. Monitor and analyze app performance

Conclusion

Microservices in 12 steps: A Short Guide with Docker for Monolithic Code Base Migrations (with some code)

Intro

Step 1: Understand the Current Monolithic Code Base

Step 2: Define a Microservices Architecture

1. Defining the Microservice Boundaries and Interfaces:

2. Communication Patterns between Microservices:

3. Data Management in Microservices:

Step 3: Containerize Microservices with Docker

Step 4: Set Up Docker Infrastructure

1. Docker Engine setup:

2. Docker Compose setup:

3. Kubernetes setup:

Step 5: Develop and Test Microservices

Step 6: Implement Service Discovery

1. Refactor and rewrite monolithic code into microservices:

2. Develop and test each microservice using Docker containers:

1. Using `select_related()` to reduce database queries:

3. Using Django's `Prefetch` to optimize related object queries: