DEV Community: shalomtash

Simulink server crash incident report

shalomtash — Tue, 01 Nov 2022 13:57:20 +0000

__## _By the Simulink backend developer team

Earlier this week we experienced a crash in our servers that led to the interruption of over 500,000 client transactions

_.
_

How could we make such a massive fail?

We are providing an incident report that details the nature of the outage and our response.

The following is the incident report for the Simulink server crash that occurred on November 1, 2022. We understand this service issue has impacted our valued developers and users, and we apologize to everyone who was affected.

Issue summary

From 12:10 PM to 8:50 PM EAT, all server requests to Simulink resulted in internal server error messages. This issue affected 100% of traffic to our servers. Users could only access certain APIs that run on separate infrastructures. The root cause of this outage was an invalid configuration change that exposed a bug in a widely used internal library.

Timeline

11:05 AM: System update released
12:05 PM: Servers restart after update
12:10 PM: Outage begins
14:30 PM: Pagers alerted teams
6:00 PM: Identification of issue
6:30 PM: Failed configuration change rollback
6:50 PM: Successful configuration change rollback
7:10 PM: Server restarts begin
8:50 PM: 100% of traffic back online

Root cause

At 11:05 AM EAT, a biannual system update was released to our production environment. The update was accompanied by an undetected configuration change that specified an invalid address for the authentication servers in production. This exposed a bug in the authentication libraries which caused them to block permanently while attempting to resolve the invalid address to physical services. In addition, the internal monitoring systems permanently blocked on this call to the authentication library. The combination of the update, bug, and configuration error quickly caused all of the serving threads to be consumed. Traffic was permanently queued waiting for a serving thread to become available. The servers began repeatedly hanging and restarting as they attempted to recover and at 12:10 PM EAT, the service outage began.

Resolution and recovery

At 2:30 PM EAT, the DataDog monitoring systems alerted our backend engineers who investigated the issue. The ongoing start and restart of servers however led to a delay in resolving the problem. An abrupt stop of all server processes would have led to the inadvertent loss of client data from our servers. As such care had to be taken to backup and transfer all data to an availability zone before proceeding to stop all processes and identify the underlying issue. By 6:00 PM, the incident response team identified that the monitoring system was exacerbating the problem caused by this bug.

At 6:30 PM, we attempted to roll back the problematic configuration change. This rollback failed due to complexity in the configuration system which caused our security checks to reject the rollback. These problems were addressed and we successfully rolled back at 6:50 PM. We then redid the system update with the servers still in offline monitoring mode.

We decided to restart servers gradually (at 7:10 PM), to avoid possible cascading failures from a wide-scale restart. By 8:00 PM, 25% of traffic was restored and 100% of traffic was routed to the API infrastructure at 8:50 PM.

Corrective and preventative measures

In the last two days, we’ve conducted an internal review and analysis of the outage. The following are actions we are taking to address the underlying causes of the issue and to help prevent recurrence and improve response times:

Disable the current combined system update and configuration release mechanism until safer measures are implemented. (Completed.)
Change monitoring alert process to be quicker and more robust.
Fix the underlying authentication libraries and monitoring to correctly timeout/interrupt on errors.
Programmatically enforce staged rollouts of all configuration changes.
Improve process for auditing all high-risk configuration options.
Add a faster rollback mechanism and improve the traffic ramp-up process, so any future problems of this type can be corrected quickly.
Develop better mechanism for quickly delivering status notifications during incidents.
Simulink is committed to continually and quickly improving our technology and operational processes to prevent outages. We appreciate your patience and again apologize for the impact to you, your users, and your organization. We thank you for your business and continued support.

Sincere apologies

Python for Web Development

shalomtash — Sun, 01 May 2022 20:59:09 +0000

web development

Web development - It broadly refers to the work involved in designing, developing, and managing websites for intranet or internet hosting. Web programming, database management, web design, web publishing, and other aspects of website creation are all covered.

All of the codes that determine how a website works are included in web development. We can divide the entire web development process into two categories.

Front-end

Back-end

Though frontend and backend web development are distinct, they are similar in that they are two sides of the same coin. Each side of a complete website relies on the other to communicate and operate as a unified entity. In web development, both the front-end and back-end are equally crucial.

Real world examples of python web development include:
Netflix, Reddit, Spotify, Dropbox, Quora etc.

python framework

Python is one of the most widely used computer programming languages today. It has Web frameworks, such as Django and Flask, make Python considerably easier to use and remove a lot of the heavy lifting from backend writing.

Django

Django is a high-level Python web framework that enables designers to create fast, clean, and pragmatic designs. Django takes care of many of the intricacies of web development, allowing you to concentrate on developing your app rather than reinventing the wheel. It's open source and free.

Django utilizes ORM to map objects to database tables and to transfer data from one database to another.

It supports the majority of popular databases, including Oracle, MySQL, PostgreSQL, SQLite, and others.

Django is the major framework for backend development on a large number of websites in the business.

Django's characteristics

The following are some of the outstanding features of this Python web framework:

-Routing URLs

-Authentication

-Migrations of database schema

-ORM stands for Organizational Resource Management (Object-relational mapper)

-Engine for creating templates

A Django framework example:

Django is one of the greatest frameworks to consider if you're planning to construct a highly configurable project, such as a social media website. Django's strength is in user interaction and the ability to exchange various forms of material. Django's ability to leverage a broad community-based support network provides you with highly customisable third-party ready-to-use plugins in your applications.

flask

Flask supports a number of extensions that extend the application's functionality as if they were built into the framework itself. Object-relational mappers, form validation, upload handling, different open authentication protocols, and other framework-related tools all have extensions.

Flask Characteristics

-Support for unit testing that is integrated

-Request dispatching with a restful interface

-This package includes a development server and a debugger.

-Jinja2 templating is used.

-Secure cookies are supported.

-Unicode-based

-WSGI 1.0 compliant to the nth degree

-Documentation is extensive.

-Compatibility with Google App Engine

-Extensions are provided to improve desired characteristics.

example:

To recapitulate, due of its lightweight, Flask is one of the most popular python web frameworks. Despite its diminutive size, it is a Python web framework that may be extended. Flask speeds up the construction of basic web applications by providing the essential features. As a result, Flask is better suited to simpler, smaller applications.

Ultimate Python Tutorial For Beginners

shalomtash — Fri, 22 Apr 2022 20:06:45 +0000

INTRODUCTION TO PYTHON
Python is one of the high level programming languages just like Java. A high level programming language means it is closer to the human language e.g., print('hello world!').

So as to understand what python is with less time wastage, I have outlined various key concepts to read about such include;

Variables
Operators
Data Types
Conditions
Loops
Exception Handling
Lists
Dictionary
String Operations
File Operations

VARIABLES
-This are whole numbers [0,1,2,3,4,5,6,7,8,9].

DATA TYPES
-Data types are the classification or categorization of knowledge items. This represents the type useful to be performed for a specific data.
There are 4 data types. This includes:

string - This are alphabetic letters str('a', 'b', 'c', 'd').
Boolean(bool) - This are True or False statements. They can even be represented using 0 to mean False and 1 to mean True.

3.Float - This are decimal numbers. e.g., 0.12 .

4.Complex - This is a building function in python which is rarely used . Complex number(real number + the imaginary number) .

LOOPS
Loop is iterable object such as a list, tuple, set etc.

CONDITIONS
They are statements that are used to show if a statement is true or not true . e.g., if... else .

STRING OPERATIONS
They are special string operators .
example
* its used to concatenate multiple copies of the same string .
[] they give a character of the same index .
[:] it gives character from a given range .

LISTS
They are used to store multiple items in a single variable. e.g., grocery = ['kales', 'tomatoes', 'onions'].

FILE OPERATIONS
They are inbuilt functions

e.g.,
r used to open a file for reading .