DEV Community: Larapulse Technology

Understanding GraphQL: A Guide for Backend Developers

Sergey Podgorny — Wed, 11 Oct 2023 09:56:00 +0000

GraphQL is a powerful query language for APIs that provides a flexible and efficient way to request and manipulate data from a server. If you're a backend developer looking to learn about GraphQL, this guide will walk you through the basics, implementation, and its relevance to your work.

What Is GraphQL?

GraphQL was developed by Facebook and has gained popularity as an alternative to RESTful APIs. It provides the following key features:

Query Language:
GraphQL uses a schema to define the types of data available and the relationships between them. Clients can send queries to request exactly the data they need, and they receive a JSON response containing only that data.
Single Endpoint:
GraphQL typically has a single endpoint for all data operations, simplifying the API surface and allowing clients to request various data in a single request.
Strongly Typed:
GraphQL APIs are strongly typed, meaning they have a defined schema with clear data types. Clients can introspect the schema to understand what data is available and how to query it.
Real-time Data:
GraphQL can be used to request real-time data through subscriptions, allowing clients to receive updates when data changes.

Implementing GraphQL

Here's a step-by-step guide to implementing GraphQL in your backend:

Define a Schema:

Create a schema that describes the types of data your API will expose. You'll specify object types, queries, mutations (for write operations), and potentially subscriptions.

type User {
  id: ID!
  name: String
  email: String
}

type Query {
  getUser(userId: ID!): User
}

type Mutation {
  updateUser(userId: ID!, input: UserInput): User
}

Set Up Resolvers:

For each field in your schema, implement resolver functions. These resolver functions determine how to fetch the data for each field. These can be as simple as fetching data from a database or involve more complex operations.

const resolvers = {
  Query: {
    getUser: (parent, args, context, info) => {
      // Fetch user data based on 'args.userId'
      // Return the user data
    },
  },
  Mutation: {
    updateUser: (parent, args, context, info) => {
      // Update user data based on 'args.userId' and 'args.input'
      // Return the updated user data
    },
  },
  User: {
    name: (user) => user.name, // Resolver for the "name" field
    email: (user) => user.email, // Resolver for the "email" field
  },
};

Middleware:

Set up middleware for handling tasks like authentication, authorization, and caching.

Server Implementation:

Choose a GraphQL server library or framework that fits your tech stack. Popular options include Apollo Server, Express GraphQL, and GraphQL Yoga for JavaScript/Node.js.

Testing and Documentation:

Write unit tests for your resolvers and document your GraphQL schema so that clients can understand how to use it.

Integration with Frontend:

Implement the GraphQL client on your frontend to send queries and mutations to the GraphQL server.

Resolvers in GraphQL

Resolvers are central to GraphQL, as they determine how to fetch the data for specific fields. Each field in your schema typically has its resolver function. These resolver functions can optimize data fetching.

If multiple fields can be efficiently fetched in a single query from the same data source, you can use a single resolver to retrieve all the necessary data.
GraphQL clients request multiple fields in a single query, and the resolver functions are invoked only for the fields requested in that query.
Resolvers allow you to tailor the database query to fetch only the data that corresponds to the requested fields.

Do You Need GraphQL?

The decision to implement GraphQL depends on your project's requirements. You might benefit from GraphQL if:

Your project involves complex data fetching, with data from multiple sources.
You want to avoid versioning your API, as GraphQL allows clients to request only the data they need.
Real-time data updates are necessary through subscriptions.
Frontend and backend teams work in parallel, as GraphQL enables frontend developers to specify their data requirements.
Mobile apps are part of your project, as GraphQL's ability to fetch only the necessary data is advantageous for mobile apps.

However, if your project has simple data retrieval needs or is small in scope, you may not necessarily need to implement GraphQL.

Conclusion

GraphQL is a powerful tool for backend developers to create flexible and efficient APIs. Its ability to allow clients to request exactly the data they need and its strong typing make it a valuable addition to your API toolkit. By understanding GraphQL's principles and implementing it effectively, you can improve data retrieval and enhance your development process.

Start exploring GraphQL today, and you'll find it to be a valuable addition to your backend development toolkit.

CPU Cache Basics

Sergey Podgorny — Mon, 09 Oct 2023 09:21:00 +0000

CPU caches are the unsung heroes of modern computing, silently speeding up your computer's performance. These small but incredibly fast memory storage areas play a vital role in ensuring that your CPU can access frequently used data and instructions with lightning speed. In this article, we'll explore the world of CPU caches, uncovering their design, optimization strategies, and their indispensable role in enhancing the performance of software and systems.

Here's what you should know as a software engineer:

L1 Cache (Level 1 Cache):

L1 cache is the smallest but fastest cache located closest to the CPU cores.
It's divided into two parts: L1i (instruction cache) and L1d (data cache). L1i stores instructions, and L1d stores data.
The purpose of L1 cache is to store the most frequently used instructions and data to speed up the CPU's operations.
It has low latency (the time it takes to access data) and is usually separate for each CPU core.

L2 Cache (Level 2 Cache):

L2 cache is larger than L1 cache but slightly slower.
It is shared among CPU cores in many multi-core processors.
Its role is to store additional frequently used data and instructions that couldn't fit in L1 cache.
L2 cache is still faster than accessing data from RAM (main memory).

L3 Cache (Level 3 Cache, if available):

L3 cache is even larger but slightly slower than L2 cache.
It is shared across all CPU cores in a multi-core processor.
L3 cache acts as a backup storage for frequently used data and instructions that couldn't fit in L1 or L2 cache.
Having an L3 cache can help reduce bottlenecks when multiple CPU cores are accessing memory simultaneously.

How it works:

When the CPU needs data or instructions, it first checks if they are in the L1 cache.
If the needed information is in the L1 cache (or L2 cache if not found in L1), it's called a cache hit, and the CPU can quickly retrieve it.
If the data is not in the cache, it's called a cache miss. In this case, the CPU has to fetch the data from slower main memory (RAM), which takes more time.
The goal of the cache hierarchy is to reduce the number of cache misses by storing the most frequently used data and instructions in the faster, smaller caches.

When it is used:

L1, L2, and L3 caches are used constantly as the CPU executes programs.
They are especially beneficial for speeding up frequently executed code and data access patterns.
The cache hardware manages what gets stored in the cache, so as a software engineer, you generally don't interact with it directly.

Optimization principles

Caches like L1, L2, and L3 are managed by the hardware, and as a software engineer, you don't have direct control over which programs or data are stored in them. However, you can follow certain programming and optimization principles to increase the likelihood that your program benefits from cache usage. Here's how:

Locality of Reference: Caches work best when your program exhibits good locality of reference. There are two types of locality:
- Temporal Locality: This means that if you access a piece of data once, you're likely to access it again in the near future. To leverage temporal locality, try to reuse data that you've recently accessed.
- Spatial Locality: This refers to the tendency to access data located near recently accessed data. To benefit from spatial locality, try to access data in a sequential or predictable pattern.
Cache-Friendly Data Structures: Use data structures and algorithms that are cache-friendly. For example, when iterating over an array, processing elements that are stored close to each other in memory is more cache-efficient than jumping around in memory.
Cache Line Awareness: Cache systems typically work with fixed-size cache lines (e.g., 64 bytes). Be aware of this when designing your data structures. If you only need a small portion of a cache line, avoid loading the entire line to reduce cache pollution.
Compiler and Compiler Flags: Compilers can optimize code to improve cache locality. Use compiler flags (e.g., -O2 or -O3 in GCC) to enable optimizations. Additionally, understand how your compiler optimizes code for your target architecture.
Profiling and Benchmarking: Use profiling tools to analyze cache behavior in your program. Tools like perf (on Linux) or performance analyzers in integrated development environments (IDEs) can help you identify cache-related issues.
Thread Affinity: If you're working with multi-threaded programs, consider using thread affinity techniques to bind threads to specific CPU cores. This can help minimize cache contention between threads.

Cache sizes

Regarding the sizes of cache levels, they can vary widely depending on the CPU architecture. However, here's a rough estimate:

L1 Cache: Typically ranges from 16KB to 128KB per core.
L2 Cache: Can range from 256KB to 1MB per core or be shared among multiple cores.
L3 Cache: Usually shared among multiple cores and can range from 2MB to 32MB or more in high-end processors.

Keep in mind that these numbers can change with different CPU models and generations. You can usually find the specific cache sizes for your CPU in its documentation or by checking the manufacturer's website. Understanding cache sizes can help you make informed decisions when optimizing your code for specific hardware.

Cache latencies

Let's compare the latencies of different memory levels, including CPU caches and RAM (main memory):

L1 Cache Latency:

L1 cache is the fastest and has the lowest latency among all memory levels.
Typical latency ranges from 1 to 3 cycles, which is extremely fast.
Accessing data from L1 cache is significantly faster than any other memory level.

L2 Cache Latency:

L2 cache has slightly higher latency compared to L1 cache.
Typical latency ranges from 4 to 10 cycles, depending on the CPU architecture.
It is still much faster than accessing RAM.

L3 Cache Latency:

L3 cache has higher latency compared to L2 and L1 caches.
Typical latency ranges from 10 to 40 cycles, depending on the CPU and cache design.
While slower than L1 and L2, it is still much faster than RAM.

RAM (Main Memory) Latency:

Accessing data from RAM is significantly slower than accessing any level of cache.
RAM latency can vary widely, but it typically ranges from 60 to 100 cycles or more.
RAM access times are several orders of magnitude slower than L1 cache.

Assuming a CPU clock speed of 3 GHz (3 billion cycles per second):

L1 cache access time:
- Fastest case (1 cycle): 1 / 3 * 10^9 = 0.33 nanoseconds
- Slowest case (3 cycles): 3 / 3 * 10^9 = 1 nanosecond
L2 cache access time:
- Fastest case (4 cycles): 4 / 3 * 10^9 = 1.33 nanoseconds
- Slowest case (10 cycles): 10 / 3 * 10^9 = 3.33 nanoseconds
L3 cache access time:
- Fastest case (10 cycles): 10 / 3 * 10^9 = 3.33 nanoseconds
- Slowest case (40 cycles): 40 / 3 * 10^9 = 13.33 nanoseconds
RAM access time:
- Fastest case (60 cycles): 60 / 3 * 10^9 = 20 nanoseconds
- Slowest case (100 cycles): 100 / 3 * 10^9 = 33.33 nanoseconds

To put these numbers into perspective, accessing data from L1 cache can be over 10 times faster than accessing the same data from L2 cache, and it can be more than 100 times faster than accessing it from RAM.

Efficient use of CPU caches is crucial for optimizing software performance because minimizing cache misses and utilizing cache-friendly algorithms can help reduce the impact of slower RAM access times. This is why understanding cache behavior and optimizing for cache locality is a key consideration in high-performance computing and software development.

Summary

The importance of CPU caches in modern computing cannot be overstated. These small, high-speed memory storage areas play a pivotal role in enhancing software performance by reducing the latency of data access. Cache-aware programming, which involves optimizing code and data structures to maximize cache utilization, has a profound impact on software performance.

In summary, caches like L1, L2, and L3 are crucial for optimizing CPU performance by reducing memory access times. As a software engineer, understanding the basics of how caches work can help you write more efficient code, such as optimizing data access patterns and minimizing cache thrashing (when cache contents change frequently). However, the specifics of cache management are typically handled by the hardware and the operating system.

Dealing with RabbitMQ exchange types

Sergey Podgorny — Tue, 20 Apr 2021 14:53:05 +0000

Exchanges control the routing of messages to queues. Each exchange type defines a specific routing algorithm which the server uses to determine which bound queues a published message should be routed to.

RabbitMQ provides four types of exchanges: Direct, Fanout, Topic, and Headers.

Fanout Exchanges

The Fanout exchange type routes messages to all bound queues indiscriminately. If a routing key is provided, it will simply be ignored. The following illustrates how the fanout exchange type works:

The Fanout exchange type is useful for facilitating the publish-subscribe pattern. When using the fanout exchange type, different queues can be declared to handle messages in different ways. For instance, a message indicating a customer order has been placed might be received by one queue whose consumers fulfil the order, another whose consumers update a read-only history of orders, and yet another whose consumers record the order for reporting purposes.

Direct Exchanges

The Direct exchange type routes messages with a routing key equal to the routing key declared by the binding queue. The following illustrates how the direct exchange type works:

The Direct exchange type is useful when you would like to distinguish messages published to the same exchange using a simple string identifier. This is the type of exchange that was used in our Hello World example. As discussed in part 3 of our series, every queue is automatically bound to a default exchange using a routing key equal to the queue name. This default exchange is declared as a Direct exchange. In our example, the queue named "hello-world-queue" was bound to the default exchange with a routing key of "hello-world-queue", so publishing a message to the default exchange (identified with an empty string) routed the message to the queue named "hello-world-queue".

Topic Exchanges

The Topic exchange type routes messages to queues whose routing key matches all, or a portion of a routing key. With topic exchanges, messages are published with routing keys containing a series of words separated by a dot (e.g. "word1.word2.word3"). Queues binding to a topic exchange supply a matching pattern for the server to use when routing the message. Patterns may contain an asterisk ("*") to match a word in a specific position of the routing key, or a hash ("#") to match zero or more words. For example, a message published with a routing key of "honda.civic.navy" would match queues bound with "honda.civic.navy", "*.civic.*", "honda.#", or "#", but would not match "honda.accord.navy", "honda.accord.silver", "*.accord.*", or "ford.#". The following illustrates how the fanout exchange type works:

The topic exchange type is useful for directing messages based on multiple categories (e.g. product type and shipping preference), or for routing messages originating from multiple sources (e.g. logs containing an application name and severity level).

Headers Exchanges

The Headers exchange type routes messages based upon a matching of message headers to the expected headers specified by the binding queue. The headers exchange type is similar to the topic exchange type in that more than one criteria can be specified as a filter, but the headers exchange differs in that its criteria is expressed in the message headers as opposed to the routing key, may occur in any order, and may be specified as matching any or all of the specified headers. The following illustrates how the headers exchange type works:

The Headers exchange type is useful for directing messages which may contain a subset of known criteria where the order is not established and provides a more convenient way of matching based upon the use of complex types as the matching criteria (i.e. a serialized object).

Conclusion

That wraps up our introduction to each of the exchange types. Next time, we’ll walk through an example which demonstrates declaring a direct exchange explicitly and take a look at the push API.

Interesting in reading more about digital literacy? Follow this link to find more articles 😉

Event-driven architecture over standard client-server aproach

Sergey Podgorny — Fri, 16 Apr 2021 14:32:08 +0000

Just 20 years ago, most of the business was offline and web development was just starting to gain traction. But gradually everything develops, the whole business is trying to go online. And along with it, web development is actively developing, there are more and more services. Web development is an integral part of any modern business.

To keep our business running fast, we need to consider all the third-party services on which our business depends. First, let's take a look at the standard architecture we're used to.

Standard Client-Server architecture

With a standard synchronous architecture, most changes have a large number of dependencies, sometimes on slow and unpredictable systems. This slows down interactions and increases the error rate. Maintainability is troublesome, as there is usually a high amount of possible side effects to consider.

👍 Advantages:

Simple to understand
Easier to debug and test
Better data consistency

👎 Disadvantages:

High rate of client-visible errors
Slow
One thing fails, everything fails - does not work well for complex business processes

Event-Driven architecture

In an event-driven architecture, when a service performs some piece of work that other services might be interested in, that service produces an event — a record of the completed action. Other services consume those events so that they can perform any of their tasks needed as a result of the event.

By decoupling the core change from the side effects performance is greatly increased and error rates go down.

Secondary concerns can be handled independently, this allows easier maintenance, and even involving different teams and/or technologies that are best suited for the respective tasks.

👍 Advantages:

Responsive
Resilient
Scales as a whole as well as only where needed
Development can be distributed between multiple teams

👎 Disadvantages:

Extra work needed to keep data consistency
Higher learning curve

Summary

Event-driven architecture has been gaining momentum in popularity lately, and there are reasons for this: services grow and develop, become larger and acquire new dependencies. Decoupling of responsibility can be a solution in this case and improve performance, but you shouldn't abuse the architecture without a detailed understanding of all its pros and cons. For small and simple projects, a standard synchronous architecture may be a much better solution.

To read more, follow these links:

Interesting in reading more about APIs? Follow this link to find more articles 😉

Go migrator for MySQL databases

Sergey Podgorny — Wed, 02 Sep 2020 18:01:16 +0000

Intention

A few months ago I started working on one of my pet projects powered by Go using MySQL as persistent data storage. After some research to find the best solution to migrate my database schema, the most popular decisions suggested for me were auto-migrator by GORM and golang-migrate utility. Unfortunately, they did not meet my expectations.

Let me explain why...

First of all, I did not want to stick to the one specific ORM and moreover, I didn't want to use ORM in my project at all. I do not want to debate whether GORM is good or not, it was just that in my case it was much more convenient and easier to use the standard library database/sql.

I also wanted to have a long-term solution that would allow me to programmatically write code to the database without delving into sql queries. Therefore, the solution using pure sql did not suit me either.

Progress

Since none of the solutions I found worked for me, I decided to write a code that would fully meet my requirements and expectations.

At first, the solution I wrote was integrated into my migration files, but I immediately noticed that the code that is used to make changes to the database can be split into a separate package. This package was still an integral part of my project, but nevertheless began to grow into something more.

After a few weeks of using my package, I decided that it proved to be quite stable and can be published in the public domain for general use. So it became part of the open-source community and my first public Go project on GitHub 💪

larapulse / migrator

MySQL database migrator

MySQL database migrator designed to run migrations to your features and manage database schema update with intuitive go code. It is compatible with the latest MySQL v8.

Installation

To install migrator package, you need to install Go and set your Go workspace first.

The first need Go installed (version 1.13+ is required), then you can use the below Go command to install migrator.

$ go get -u github.com/larapulse/migrator

Import it in your code:

import "github.com/larapulse/migrator"

Quick start

Initialize migrator with migration entries:

var migrations = []migrator.Migration{
    {
        Name: "19700101_0001_create_posts_table"
        Up: func() migrator.Schema {
            var s migrator.Schema
            posts := migrator.Table{Name: "posts"}

            posts.UniqueID("id")
            posts.Varchar("title", 64)
            posts.Text("content", false)
            posts.Timestamps()

            s.CreateTable(posts

…

View on GitHub

Since this was my first project to Go open-source packages, I didn't know a few things.

How to publish my package to pkg.go.dev? The solution turned out to be quite simple and unusual. There is no need to upload your package anywhere, you just need to request it either with the help of searching for packages, or using the command line.
How to write good documentation for your package? The package manager independently reads the comments in front of functions and variables, and then generates documentation for your package. So be careful what and how you write in the comment before the function. Try to describe in as much detail as possible what exactly is happening and for what exactly and how this or that part of your package is used. Unfortunately, there are currently no mechanisms for editing public documentation, so you cannot change the display order or formatting to your liking. Documentation is a visual display of your comments in front of exported functions and variables. 🤷‍♂️
go.mod file really matters. While the unexported package can be called whatever you like (fe. module migrator), published module should be named the same as the place where it is actually located (fe. module github.com/larapulse/migrator)

Features

Migrate, Rollback and Revert your migration:

m := migrator.Migrator{Pool: migrations}
migrated, err = m.Migrate(db)

if err != nil {
    log.Errorf("Could not migrate: %v", err)
    os.Exit(1)
}

if len(migrated) == 0 {
    log.Print("Nothing were migrated.")
}

for _, m := range migrated {
    log.Printf("Migration: %s was migrated ✅", m)
}

log.Print("Migration did run successfully")

While Migrate runs all unexecuted migrations from the pool, you can also Rollback last executed migration batch, or Rollback whole migration list.

It creates migrations table and tracks there all executed migrations. You can set any name you like if you want.

m := migrator.Migrator{TableName: "_my_app_migrations"}

Defining migrations with schema commands. You can create, drop or alter tables:

var migration = migrator.Migration{
    Name: "19700101_0001_create_posts_table",
    Up: func() migrator.Schema {
        var s migrator.Schema
        posts := migrator.Table{Name: "posts"}

        posts.UniqueID("id")
        posts.Varchar("title", 64)
        posts.Text("content", false)
        posts.Timestamps()

        s.CreateTable(posts)

        return s
    },
    Down: func() migrator.Schema {
        var s migrator.Schema

        s.DropTableIfExists("posts")

        return s
    },
}

Using transactional migrations. In case you have multiple commands within one migration and you want to be sure it is migrated properly, you might enable transactional execution per migration:

var migration = migrator.Migration{
    Name: "19700101_0003_rename_foreign_key",
    Up: func() migrator.Schema {
        var s migrator.Schema

        keyName := migrator.BuildForeignNameOnTable("comments", "post_id")
        newKeyName := migrator.BuildForeignNameOnTable("comments", "article_id")

        s.AlterTable("comments", migrator.TableCommands{
            migrator.DropForeignCommand(keyName),
            migrator.DropIndexCommand(keyName),
            migrator.RenameColumnCommand{"post_id", "article_id"},
            migrator.AddIndexCommand{newKeyName, []string{"article_id"}},
            migrator.AddForeignCommand{migrator.Foreign{
                Key:       newKeyName,
                Column:    "article_id",
                Reference: "id",
                On:        "posts",
            }},
        })

        return s
    },
    Down: func() migrator.Schema {
        var s migrator.Schema

        keyName := migrator.BuildForeignNameOnTable("comments", "article_id")
        newKeyName := migrator.BuildForeignNameOnTable("comments", "post_id")

        s.AlterTable("comments", migrator.TableCommands{
            migrator.DropForeignCommand(keyName),
            migrator.DropIndexCommand(keyName),
            migrator.RenameColumnCommand{"article_id", "post_id"},
            migrator.AddIndexCommand{newKeyName, []string{"post_id"}},
            migrator.AddForeignCommand{migrator.Foreign{
                Key:       newKeyName,
                Column:    "post_id",
                Reference: "id",
                On:        "posts",
            }},
        })

        return s
    },
    Transaction: true,
}

Custom queries on schema and tables.

You may add any column definition to the database on your own, just be sure you implement migrator.columnType interface:

type customType string

func (ct customType) buildRow() string {
    return string(ct)
}

posts := migrator.Table{Name: "posts"}
posts.UniqueID("id")
posts.Column("data", customType("json not null"))
posts.Timestamps()

The same logic is for adding custom commands to the schema to be migrated or reverted, just be sure you implement command interface:

type customCommand string

func (cc customCommand) toSQL() string {
    return string(cc)
}

var s migrator.Schema

c := customCommand("DROP PROCEDURE abc")
s.CustomCommand(c)

Pending features

At the time of this writing, the latest stable version is v1.2.0. As the next steps, it is planned to add more commands for interacting with the schema.

If necessary, in the next major versions drivers can be added for Postgres, SQLite, Microsoft SQL Server etc.

Result

At the moment, this package is a part of the go open source community and is versatile enough to be used in various projects. I really hope that it will also be useful to you when working with the database schema.

It is not popular yet, but I am really proud of it and believe it may grow fast with your support and contributions!

// Detect dark theme var iframe = document.getElementById('tweet-1277288650959736832-585'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1277288650959736832&theme=dark" }

How DNS servers work

Sergey Podgorny — Tue, 04 Feb 2020 11:15:52 +0000

What is DNS

DNS is a global system for translating human-readable domain names to IP addresses. When a user tries to access a web address like "example.com", their web browser or application performs a DNS Query against a DNS server, supplying the hostname. The DNS server takes the hostname and resolves it into a numeric IP address, which the web browser can connect to.

A component called a DNS Resolver is responsible for checking if the hostname is available in the local cache, and if not, contacts a series of DNS Name Servers, until eventually it receives the IP of the service the user is trying to reach and returns it to the browser or application. This usually takes less than a second.

Domain Registrars

Before anything else, it is important to know that your domain registrar is where the entire process begins. Your domain registrar is normally the place where you purchased/obtained your domain. Your registrar could be a business that only does domain registration or even your web host itself.

The registrar is the first piece of our puzzle because whenever a request for a domain is made, the registrar is the first place that gets referenced for information. As it pertains to loading a website, the most important part of the response is what set of "nameservers" the domain is pointed at.

Websites like http://who.is can help you identify your domain's registrar.

You can change your domain's registrar through a transfer process. In general, changing registrars is not necessary to modify your domain's DNS records.

Nameservers

A domain's nameservers are significant because they identify what set of servers any requests should reference in order to obtain a domain's DNS records. Your domain should have its nameservers pointed to wherever you are intending to manage your site's DNS records.

You can point your domain's nameservers anywhere that you can manage DNS, though. This means that if your domain's registrar allows you to manage DNS with them, then you can point nameservers at your registrar's nameservers and use their DNS management panel to set records. If you are interested in having your DNS hosted somewhere other than your web host or registrar, then hosted DNS is a good solution for you if you can find a service offering that fits your need and capacity requirements (e.g. AWS Route 53 that is a scalable cloud DNS web service).

Your domain's nameservers will likely need to change in the following cases:

You have just purchased a domain and need to point nameservers at your web host. The default nameservers when purchasing a domain are normally the ones operated by the domain registrar.
You are currently managing your DNS records at your registrar and want your host to manage them instead.
You have nameservers pointed at a web host that you are transitioning away from. It is necessary to change nameservers because closing your hosting account will also remove your DNS records at that host so you will want to point them elsewhere.

One of the most important things to note when changing nameservers is that they are a setting that can take a while to take effect across the world. In most cases, nameservers will take between 1 and 10 hours to spread across a majority of servers across the world, but you should allow as long as 36 hours for them to take fully.

This is quite a vital piece of knowledge because it is entirely possible to have two different sets of DNS records on two different sets of nameservers. If nameservers have taken effect in some parts of the world, but not others, then you may have some visitors seeing the site on old DNS records and some on new ones.

Query Types

There are three types of queries in the DNS system:

Recursive Query: In a recursive query, a DNS client provides a hostname, and the DNS Resolver "must" provide an answer - it responds with either a relevant resource record or an error message if it can't be found. The resolver starts a recursive query process, starting from the DNS Root Server until it finds the Authoritative Name Server that holds the IP address and other information for the requested hostname.
Iterative Query: In an iterative query, a DNS client provides a hostname, and the DNS Resolver returns the best answer it can. If the DNS resolver has the relevant DNS records in its cache, it returns them. If not, it refers the DNS client to the Root Server or another Authoritative Name Server which is nearest to the required DNS zone. The DNS client must then repeat the query directly against the DNS server it was referred to.
Non-Recursive Query: A non-recursive query is a query in which the DNS Resolver already knows the answer. It either immediately returns a DNS record because it already stores it in the local cache, or queries a DNS Name Server which is authoritative for the record, meaning it definitely holds the correct IP for that hostname. In both cases, there is no need for additional rounds of queries (like in recursive or iterative queries). Rather, a response is immediately returned to the client.

DNS Servers Types

The following are the most common DNS server types that are used to resolve hostnames into IP addresses.

DNS Resolver

A DNS resolver (recursive resolver), is designed to receive DNS queries, which include a human-readable hostname such as www.example.com, and is responsible for tracking the IP address for that hostname.

DNS Root Server

The root server is the first step in the journey from hostname to IP address. The DNS Root Server extracts the Top Level Domain (TLD) from the user's query, for example, www.example.com and provides details for the .com TLD Name Server. In turn, that server will provide details for domains with the .com DNS zone, including "example.com".

There are 13 root servers worldwide, indicated by the letters A through M, operated by organizations like the Internet Systems Consortium, Verisign, ICANN, the University of Maryland, and the U.S. Army Research Lab.

Authoritative DNS Server

Higher-level servers in the DNS hierarchy define which DNS server is the "authoritative" name server for a specific hostname, meaning that it holds the up-to-date information for that hostname.

The Authoritative Name Server is the last stop in the name server query - it takes the hostname and returns the correct IP address to the DNS Resolver (or if it cannot find the domain, returns the message NXDOMAIN).

Record Composition

In general, most every DNS record requires at least three pieces of information:

Record Name
Record Type
Record Value/Data
Time to Live (TTL)

Record Name

A DNS Record's name is the descriptor and effective subdomain of that entry. If you were adding a blog to your domain, you would create a DNS record and assign it the name "blog". This would mean that whenever a request attempts to use blog.larapulse.com, the DNS zone is queried for information about a DNS record with the name of "blog". Although you can assign whatever name you'd like to a record, there exist a few special cases you should be aware of.

Blank Name – A record that has nothing in the name slot gets used for all requests made to the base/naked form of the domain. To refer to a previous example, larapulse.com and www.larapulse.com are two different DNS records with separate values for their name. The www.larapulse.com DNS record uses "www" as its record name and larapulse.com uses nothing/blank for its record name.

@ Symbol – Some DNS management systems use the @ symbol in the "name" slot instead of the blank name entry. This is important because it allows the use of @ as another record's value/data, meaning the other record will refer to the value of the base/naked form of the domain to know where to point.

Let's say for instance you always wanted www.larapulse.com to point at the same place as larapulse.com. You could just set them individually to point at the same place and manage them separately, but since the @ symbol represents the base/naked DNS record, you could set the www DNS record entry to have a value/data of @ and it will always refer to and use the value of the @ name record when being requested. This is useful when multiple subdomains/records need to all point at the same place as the base/naked domain.

* Symbol (Wildcard) – It isn't strange to see the * symbol be used as the name for a DNS record. This symbol is used as an indicator that the record named * should be used in place of any record that isn't specified. These can be used when you would like to direct subdomains that do not exist somewhere where they can be accounted for or handled. This means that you can specify where you would like to send all traffic for the subdomain that does not exist as a DNS record or that you might not expect people to visit. You could consider the * symbol to be the DNS record "for everything else".

Record Types

DNS servers create a DNS record to provide important information about a domain or hostname, particularly its current IP address. The most common DNS record types are:

A Record: Address Mapping record: also known as a DNS host record, stores a hostname and its corresponding IPv4 address.
AAAA Record: IP Version 6 Address record: stores a hostname and its corresponding IPv6 address.
CNAME Record: Canonical Name record: specifies a domain name that has to be queried in order to resolve the original DNS query. Therefore it can be used to alias a hostname to another hostname.
MX Record: Mail exchanger record: specifies an SMTP email server for the domain a DNS domain name.
NS Record: Name Server records: specifies that a DNS Zone, such as "example.com" is delegated to a specific Authoritative Name Server, and provides the address of the name server.
SOA Record: Start of Authority: this record appears at the beginning of a DNS zone file and specifies core information about a DNS zone, such as Authoritative Name Server for the current DNS zone, contact details for the domain administrator, domain serial number, and information on how frequently DNS information for this zone should be refreshed.
TXT Record: Text Record: is used to store any text-based information that can be grabbed when necessary.
PTR Record: Reverse-lookup Pointer records: is used to look up domain names based on an IP address, and allows a DNS resolver to provide an IP address and receive a hostname (reverse DNS lookup).
CERT Record: Certificate record: store certificates and related revocation lists (CRL) for cryptographic keys.
SRV Record: Service Location: is used to specify the location of a service, like MX but for other communication protocols.

Record Value/Data

A DNS Record's value or data is the information that tells the DNS record where you want it to point, or in some cases, what you want it to do. In the case of A Records and CNAMEs, the data/value represents the IP or domain, respectively, that the record name is pointed at and should refer to figure out where to go next.

In MX records, the value/data information indicates what mail servers email should be routed to. SPF records use the value/data field so specify what servers are allowed to legitimately use your domain name for the sending of emails.

Time to Live (TTL)

TTL is the numerical value, in seconds, of how long a DNS record will be cached before it needs to be refreshed. Whenever a nameserver is queried for a DNS record, it will check to see if it has delivered that same DNS record within the time period specified by the TTL and if so, will deliver the cached version of that DNS record. Once that period of time specified by the TTL passes, the nameserver will query the zone for record data and cache it once more for the specified period of time.

The most important thing to know about TTL is that any changes you make to the value/data of a DNS record are subject to that TTL in regard to how long it will take for that record to start taking effect.

TTL also applies to change the TTL itself in a record. For instance, if you are switching hosts and know you are going to need to change the IP address your domain points to, you might want to lower the TTL on your DNS records so that the switch over from your old host to new host happens as quickly as possible.

Simply changing the TTL from 3600 to 60 will not ensure that your IP change will take effect within 60 seconds, it simply indicates to your zone/nameservers that once they request a fresh set of information after the current 3600 second caching period, it should also adjust your TTL from that point on. This means that if you want to switch from a 3600 TTL to a 60 second TTL, you should make the TTL change at least 3600 seconds in advance of your IP change to ensure yourself the 60 second TTL period.

Interesting in reading more about Networks? Follow this link to find more articles 😉

Commonly used DNS record types

Sergey Podgorny — Mon, 13 Jan 2020 16:47:34 +0000

What is DNS

DNS is a global system for translating IP addresses to human-readable domain names. When a user tries to access a web address like example.com, their web browser or application performs a DNS Query against a DNS server, supplying the hostname. The DNS server takes the hostname and resolves it into a numeric IP address, which the web browser can connect to.

What is record types

Before we dive into various types of DNS records, it is important to understand the distinction and concept of DNS Zones and DNS Records.

Zone DNS database is a collection of resource records and each of the records provides information about a specific object. A DNS Zone is like a container of all the DNS records for a specific domain and only that domain.

A DNS Record is a single entry that gives the zone instructions on how to handle any given request based on type.

DNS servers create a DNS record to provide important information about a domain or hostname, particularly its current IP address. Each individual DNS record is assigned a type and information needed for that type of record.

The most common DNS record types are:

Record A: Address Mapping record

The A-record is the most basic and the most commonly used DNS record type. It is also known as a DNS host record, stores a hostname and its corresponding IPv4 address.

It is used to translate human-friendly domain names such as www.example.com into IP-addresses such as 93.184.216.34 (machine friendly numbers).

A-records are the DNS server equivalent of the hosts' file - a simple domain name to IP-address mapping. A-records are not required for all computers, but are needed for any computer that provides shared resources on a network. Assigning a value to an A record is as simple as providing your DNS management panel with an IP address to where the domain or subdomain should point and a TTL.

You'll want to use an A Record for your DNS entry if you have an IP address that the domain/subdomain should point to or if you want to establish a domain/subdomain to be used as the place to point a CNAME.

This record type is defined in RFC1035.

Record AAAA: IP Version 6 Address record

The record AAAA (also quad-A record) specifies IPv6 address for a given host. So it works the same way as the A record and the difference is the type of IP address.

IPv6 is the future replacement for the current IP address system (also known as IPv4). The current IPv4 addresses are 32 bits long (xxx.xxx.xxx.xxx = 4 bytes), and therefore "only" support a total of 4,294,967,296 addresses - less than the global population. With this limitation, there is an increasing shortage of IPv4 addresses, and to solve the problem, the whole Internet will eventually be migrated to IPv6.

IPv6 addresses are 128 bits long and are written in hexadecimal numbers separated by colons (:) at every four digits (segment). A series of zero value segments can be shortened as "::", and leading zeros in a segment can be skipped. For example: 4C2F::1:2:3:4:567:89AB.

This record type is defined in RFC3596.

Record CNAME: Canonical Name record

The CNAME record specifies a domain name that has to be queried in order to resolve the original DNS query. Therefore CNAME records are used for creating aliases of domain names. CNAME records are truly useful when you want to alias a domain to an external domain.

When a DNS client requests a record that contains a CNAME, which points to another hostname, the DNS resolution process is repeated with the new hostname. In other cases, we can remove CNAME records and replace them with A records and even decrease performance overhead.

Computers on the Internet often perform multiple roles such as web-server, ftp-server, chat-server etc. To mask this, CNAME-records can be used to give a single computer multiple names (aliases).

For example, the computer "computer1.xyz.com" may be both a web-server and an ftp-server, so two CNAME-records are defined:

www.xyz.com = computer1.xyz.com and ftp.xyz.com = computer1.xyz.com.

Sometimes a single server computer hosts many different domain names (take ISPs), and so CNAME-records may be defined such as www.abc.com = www.xyz.com.

The most common use of the CNAME-record type is to provide access to a web-server using both the standard www.domain.com and domain.com (with and without the www prefix). This is usually done by creating an A-record for the short name (without www), and a CNAME-record for the www name pointing to the short name.

CNAME-records can also be used when a computer or service needs to be renamed, to temporarily allow access through both the old and new name.

A CNAME-record should always point to an A-record and never to itself or another CNAME-record to avoid circular references.

This record type is defined in RFC1035.

Record MX: Mail exchanger record

It specifies an SMTP email server for the domain a DNS domain name. The information is used by Simple Mail Transfer Protocol (SMTP) to route emails to proper hosts. Typically, there is more than one mail exchange server for a DNS domain and each of them has set priority.

Mail Exchanger (MX) records are used to help route email according to the domain owners preference. The MX record itself specifies which server(s) to attempt to use to deliver mail to when this type of request is made to the domain. They differ from A Records and CNAMEs in the way that they also require a "priority" value as a part of their entry. The priority number is used to indicate which of the servers listed as MX records it should attempt to use first.

If a domain name is handled by multiple e-mail servers (for backup/redundancy), a separate MX-record is used for each e-mail server, and the preference numbers then determine in which order (lower numbers first) these servers should be used by other e-mail servers.
If a domain name is handled by a single e-mail server, only one MX-record is needed and the preference number does not matter.

Some email providers have only one MX record and some have well over two. The number of MX entries you will need to create depends largely on the mail provider and how they expect the load on these email servers to be handled.

When sending an e-mail to "user@example.com", your e-mail server must first lookup any MX-records for "example.com" to see which e-mail servers handle incoming e-mail for "example.com". This could be "mail.example.com" or someone else's mail server like "mail.larapulse.com". After this, it looks up the A-record for that e-mail server name to connect to its IP-address.

Important:
An MX-record must point to the name of a mail server - not directly to the IP-address.
Because of this, it is very important that an A-record for the referenced mail server name exists (not necessarily on your DNS server, but wherever it belongs), otherwise there may not be any way to connect to that e-mail server.

Do not point an MX-record to a CNAME-record. Many e-mail servers don't understand this. Add another A-record instead.

This record type is defined in RFC1035.

Record NS: Name Server records

It specifies that a DNS Zone, such as “example.com” is delegated to a specific Authoritative Name Server, and provides the address of the name server.

A zone should contain one NS-record for each of its own DNS servers (primary and secondaries). This is mostly used for zone transfer purposes (notify messages). These NS-records have the same name as the zone in which they are located.

The more important function of the NS-record is delegation. Delegation means that part of a domain is delegated to other DNS servers.

For example, all ".com" sub-names (such as "example.com") are delegated from the "com" zone. The "com" zone contains NS-records for all ".com" sub-names (a lot!).

You can delegate sub-names of your own domain name (such as "subname.example.com") to other DNS servers the same way. To delegate "subname.example.com", create NS-records for "subname.example.com" in the "example.com" zone. These NS-records must point to the DNS server responsible for "subname.example.com", for example, "ns1.subname.example.com" - or a DNS server somewhere else like "ns1.othername.net".

An NS-record identifies the name of a DNS server - not the IP-address. Because of this, it is important that an A-record for the referenced DNS server exists (not necessarily on your DNS server, but wherever it belongs), otherwise there may not be any way to connect with that DNS server.

If an NS-record delegates a sub-name ("subname.example.com") to a DNS server with a name in that sub-name ("ns1.subname.example.com"), an A-record for that server (""ns1.subname.example.com") must exist in the parent zone ("example.com"). This A-record is called a "glue record", because it doesn't really belong in the parent zone, but is necessary to locate the DNS server for the delegated sub-name.

This record type is defined in RFC1035.

Record SOA: Start of Authority

This record appears at the beginning of a DNS zone file and specifies core information about a DNS zone. A zone contains exactly one SOA-record, which holds the following properties for the zone:

Name of primary DNS server. The hostname of the primary DNS server for the zone. The zone should contain a matching NS-record.

NOTE: For dynamic updates from Windows clients and Active Directory to work correctly, it is important that this contains the correct hostname for the primary DNS server for the zone, and also that an A-record exists for this name pointing to the correct IP address.

E-mail address of responsible person. The e-mail address of the person responsible for the zone. The standard for this is the "hostmaster" alias - such as "hostmaster@example.com".
Serial number. Used by secondary DNS servers to check if the zone has changed. If the serial number is higher than what the secondary server has, a zone transfer will be initiated. You should never decrease the serial number.
Refresh Interval. How often secondary DNS servers should check if changes are made to the zone.
Retry Interval. How often secondary DNS server should retry checking if changes are made - if the first refresh fails.
Expire Interval. How long the zone will be valid after a refresh. Secondary servers will discard the zone if no refresh could be made within this interval.
Minimum (default) TTL. Used by other DNS servers to cache negative responses (such as record does not exist etc.).

This record type is defined in RFC1035.

Record TXT: Text Record

A TXT record is used to store any text-based information that can be grabbed when necessary. We most commonly see TXT records used to hold SPF data and verify domain ownership. They are also often used to hold general information about a domain name such as who is hosting it, contact person, phone numbers, etc.

One common use of TXT-records is for SPF.

This record type is defined in RFC1035.

Record ALIAS: Auto resolved alias

ALIAS-records are virtual alias records resolved by DNS Provider at the time of each request - providing "flattened" (no CNAME-record chain) synthesized records with data from a hidden source name.

This can be used for different purposes - including solving the classic problem with CNAME-records at the domain apex (for the zone name / for "the naked domain").

Similar implementations of virtual alias records ("ANAME", "ALIAS", "CNAME flattening") are offered by various DNS service providers. There is no standard (RFC or similar) for this.

Record PTR: Reverse-lookup Pointer records

As opposed to forward DNS resolution (A and AAAA DNS records), the PTR record is used to look up domain names based on an IP address.

For a reverse IPv4 mapping, the name of the PTR-record is the IP address with the segments reversed and with "in-addr.arpa" appended to the end. As an example, looking up the domain name for IP address "12.23.34.45" is done with a query for the PTR-record for "45.34.23.12.in-addr.arpa".

For a reverse IPv6 mapping, the name of the PTR-record is each hex digit of the IP address in reverse order, with dots between each digit, and with "ip6.arpa" appended to the end. As an example, looking up the domain name for IPv6 address "1234:5678:90ab:cdef:1234:5678:90ab:cdef" is done with a query for the PTR-record for "f.e.d.c.b.a.0.9.8.7.6.5.4.3.2.1.f.e.d.c.b.a.0.9.8.7.6.5.4.3.2.1.ip6.arpa".

This record type is defined in RFC1035.

Record CERT: Certificate record

CERT-records store certificates and related revocation lists (CRL) for cryptographic keys.

CERT-records have the following data elements (see RFC below for details):

Type: the type of certificate/CRL stored. Select one of the predefined values, or enter a numeric value (0-65535).
Key tag: A numeric value (0-65535) used the efficiently pick a CERT record.
Algorithm: the algorithm used. Select one of the predefined values, or enter a numeric value (0-65535).
Certificate or CRL: Base 64 encoded.

This record type is defined in RFC4398.

Record SRV: Service Location

SRV-records are used to specify the location of a service. They are used in connection with different directory servers such as LDAP (Lightweight Directory Access Protocol), and Windows Active Directory, and more recently with SIP servers.

They can also be used for advanced load balancing and to specify specific ports for services, for example, that a web-server is running on port 8080 instead of the usual port 80 (theoretical example - this is not yet supported by any major browsers). This record type is, however, NOT supported by most programs in use today, including web-browsers.

The SRV record identification (record name) is made of 3 parts:

Service: Most internet services are defined in RFC1700 (page 15).
Protocol: Generally TCP or UDP, but also values are also valid.
Domain name.

The "service location" is specified through a priority, weight, port and target:

Priority is a preference number used when more servers are providing the same service (lower numbers are tried first).
Weight is used for advanced load balancing.
Port is the TCP/UDP port number on the server that provides this service.
Target is the domain name of the server (referencing an A-record or AAAA-record).

This record type is defined in RFC2782.

Record HINFO: Host Information records

HINFO records are used to acquire general information about a host. The record specifies a type of CPU and OS. The HINFO record data provides the possibility to use operating system specific protocols when two hosts want to communicate. For security reasons, the HINFO records are not typically used on public servers.

This information can be used by application protocols such as FTP, which use special procedures when communicating with computers of a known CPU and operating system type. Standard CPU and operating system types are defined in RFC1700 (Page 206 / 214). The standard for a Windows PC is "INTEL-386" / "WIN32".

This record type is defined in RFC1035.

Record ISDN: Integrated Services Digital Network records

The ISDN resource record specifies ISDN address for a host. An ISDN address is a telephone number that consists of a country code, a national destination code, an ISDN Subscriber number and, optionally, an ISDN subaddress.

The ISDN phone numbers / DDI (Direct Dial In) used should follow ITU-T E.163/E.164 international telephone numbering standards. The ISDN sub-address is an optional hexadecimal number.

The function of the record is the only variation of the A resource record function.

This record type is defined in RFC1183.

Managing your own DNS can be a tricky endeavour, especially if you haven't ever considered what this means or ever even seen a DNS record. The most important DNS record types you should know is A, CNAME, MX, NS and SOA.

Interesting in reading more about Networks? Follow this link to find more articles 😉

Process monitoring tools you probably didn't know about

Sergey Podgorny — Fri, 20 Dec 2019 12:45:34 +0000

Htop – Linux Process Monitoring

htop is a much advanced interactive and real time Linux process monitoring tool. This is much similar to Linux top command, but it has some rich features like user friendly interface to manage process, shortcut keys, vertical and horizontal view of the processes and much more. htop is a third-party tool and isn't included in Linux systems, you need to install it manually.

Features

it shows a frequently updated list of the processes running on a computer, normally ordered by the amount of CPU usage;
it provides a full list of processes running, instead of the top resource-consuming processes;
it uses color and gives visual information about processor, swap and memory status;
it can also display the processes as a tree;
it provides a convenient, visual, cursor-controlled interface for sending signals to processes.

Instalation

Debian / Ubuntu:

$ sudo apt-get install htop

CentOs / Fedora:

# yum install htop

MacOS:

$ brew install htop

Usage

All you have to do is just run it from the terminal:

$ htop

Navigating throughout this tools is very easy, especially due to the fact that it supports the work of the cursor.

You can also check out other alternatives, like Glances or nmon.

Glances is a good htop alternative as it can adapt dynamically when displaying system information, depending on the terminal size.

nmon is another htop alternative systems administrators tool, for server tuning, benchmarking or viewing detailed system performance information.

IPTraf – Real Time IP LAN Monitoring

iptraf is an open source console-based real time network (IP LAN) monitoring utility for Linux. It collects a variety of information such as IP traffic monitor that passes over the network, including TCP flag information, ICMP details, TCP/UDP traffic breakdowns, TCP connection packet and byne counts. It also gathers information of general and detailed interface statistics of TCP, UDP, IP, ICMP, non-IP, IP checksum errors, interface activity etc.

Features

An IP traffic monitor shows information on the IP traffic passing over your network. Includes TCP flag information, packet and byte counts, ICMP details, OSPF packet types.
General and detailed interface statistics showing IP, TCP, UDP, ICMP, non-IP and other IP packet counts, IP checksum errors, interface activity, packet size counts.
A TCP and UDP service monitor showing counts of incoming and outgoing packets for common TCP and UDP application ports.
A LAN statistics module discovers active hosts and shows statistics showing the data activity on them.
TCP, UDP, and other protocol display filters allowing you to view only traffic you're interested in.
Support Logging.
Supports Ethernet, FDDI, ISDN, SLIP, PPP, and loopback interface types.
Utilizes the built-in raw socket interface of the Linux kernel, allowing it to be used over a wide range of supported network cards.
Full-screen, menu-driven operation.

Instalation

Debian / Ubuntu:

$ sudo apt-get install iptraf

CentOS / Fedora:

# yum install iptraf

Usage

If the iptraf command is issued without any command-line options, the program comes up in interactive mode, with the various facilities accessed through the main menu. Note, that it has to be run with root privileges:

$ sudo iptraf

Ctop - interface for container metrics

ctop provides a concise and condensed overview of real-time metrics for multiple containers, as well as a single container view for inspecting a specific container.

It mostly works same as docker stats, but provides a better interface and single container view and logging.

Features

it shows a frequently updated list of the containers with metrics running on a computer;
it comes with built-in support for Docker and runC;
it uses color and gives visual information about CPU, memory and network usage;
it shows logs per instance;
it has a full-screen, menu-driven operation.

Instalation

Linux:

$ sudo wget https://github.com/bcicen/ctop/releases/download/v0.7.2/ctop-0.7.2-linux-amd64 -O /usr/local/bin/ctop
$ sudo chmod +x /usr/local/bin/ctop

MacOS:

$ brew install ctop

Usage

ctop requires no arguments and uses Docker host variables by default. To run it, just type it in the terminal:

$ ctop

Mytop - real-time threads and performance monitoring tools

mytop is an open source, command line tool used for monitoring MySQL performance. Mytop connects to a MySQL server and periodically runs the show processlist and show global status commands. It then summarizes the information in a useful format. Using mytop, we can in real-time monitor MySQL threads, queries, and uptime as well as see which user is running queries on which database, which are the slow queries, and more. All this information can be used to optimize the MySQL server performance.

Features

htop provides a command-line shell interface to monitor real time MySQL/MariaDB threads, queries per second, process list and performance of databases and gives an idea for the database administrator to better optimize the server to handle a heavy load.

The mytop display screen is really broken into two parts. The top 4 lines (header) contain summary information about your MySQL server:

The first line identified the hostname of the server and the version of MySQL it is running. The right side shows the uptime of the MySQL server process in days+hours:minutes:seconds format (much like FreeBSD's top) as well as the current time.
The second line displays the total number of queries the server has processed, the average number of queries per second, the real-time number of queries per second, and the number of slow queries.
The third line deals with threads. Versions of MySQL before 3.23.x didn't give out this information, so you'll see all zeros.
And the fourth line displays key buffer efficiency (how often keys are read from the buffer rather than disk) and the number of bytes that MySQL has sent and received.

The second part of the display lists as many threads as can fit on screen. By default, they are sorted according to their idle time (least idle first).

Instalation

By default Mytop tool is included in the Fedora and Debian/Ubuntu repositories, so you just have to install it using your default package manager. To install Mytop, run the appropriate command below for your Linux distribution to install it:

$ sudo apt install mytop    # Debian/Ubuntu
# yum install mytop         # RHEL/CentOS
# dnf install mytop         # Fedora 22+
# pacman -S mytop           # Arch Linux
# zypper in mytop           # openSUSE

To prevent inserting connection details, create a customized configuration file for mytop named .mytop in the home directory and add the following content in the file:

user=root
pass=
host=127.0.0.1
db=
port=3306
delay=3
slow=10
socket=
batchmode=0
header=1
color=1
idle=1
long=120

The delay option specifies the amount of time in seconds between display refreshes. The idle parameter specifies whether to allow idle (sleeping) threads to appear in the list in mytop display screen.

Usage

If you configured .mytop file, you can run mytop without any command-line arguments:

$ mytop

If you didn't provide a password in the configuration file, you have to use the --prompt option to mytop, which asks for the password each time:

$ mytop --prompt
Password:

If you need to connect to another database, then in the configuration file, you could run mytop with necessary arguments:

$ mytop -u larapulse -d larapulse_db --prompt

That's definitely not all useful monitoring tools that exist, so you could find more in resources (links) below. Nevertheless, we think that htop, iptraf, ctop and mytop are the most used in the daily development cycle.

Interesting in reading more? Follow this link to find more articles 😉

PHP variables under the hood

Sergey Podgorny — Wed, 18 Dec 2019 12:50:13 +0000

Variables in PHP are some containers that store the type of the variable, its value, the amount of referring variables to this container, and the flag - whether this variable is referenced.

Structures and Pointers

Structures are very similar to classes, but they can not have methods, just data, pointers to data and pointers to functions. Declaring a structure in C, you define the data type. And when defining a variable, you can write the name of this structure in place of the type of that variable, like so:

my_super_struct super_struct_instance;

Pointers are like variables, but their values store an address in memory. Reference variables behave like dereferenced pointers, that means they access values of the pointer. Let's look at an example:

// defining pointer `foo`, that will points to the variable with `int` type
int *foo;
// defining variable with `int` type
int bar = 3;

// taking reference to variable `bar` and assigning it to pointer.
// `foo` stores memory address, where `bar` stores
foo = &bar;

// with an asterisk we dereference the pointer (take the value at its address) and increment the value
(*foo)++;

// we increment the pointer itself, that means the pointer will refer at another value
foo++;

Containers

The container is a structure called zval (short for “Zend value”), it represents an arbitrary PHP value and looks like this:

struct zval {
    zvalue_value value;
    zend_uchar type;
    zend_uchar is_ref;
    zend_ushort refcount;
};

As we can see, there is a value, a type, a flag and an amount of referring variables. PHP zval supports the following 8 types:

BOOL
LONG (signed integer type)
DOUBLE (used to store floating point numbers)
STRING
ARRAY
OBJECT
RESOURCE
NULL

zvalue_value is an union. A union is a special type that may have several members declarations of different types, but only one will be used. That how it defines:

typedef union _zvalue_value {
    long lval; // integer
    double dval; // float
    struct {
        char *val;
        int len;
    } str; // string
    HashTable *ht; // array
    zend_object obj; // object
} zvalue_value;

As a result, when you create a variable of this type, it will take in memory exactly as much as occupies the heaviest element of unions.

Why do we need all this

First, let's figure out why we need refcount. That's very simple: when you assign to a variable value of another variable, they both refer to one zval, and refcount increments.

Now, if you change the value of one of these variables, PHP seeing the refcount greater than 1, will copy this zval, make the changes there, and your variable will point already to the new zval. It will look something like this:

PHP Under the hood

PHP	Under the hood
`$foo = "baz"; $bar = $foo;`	`bar,foo: { type: string, value: str: val: "baz" len: 3 is_ref: 0 refcount: 2 }`
`$bar .= "q";`	`foo: { type: string, value: str: val: "baz" len: 3 is_ref: 0 refcount: 1 } bar: { type: string, value: str: val: "bazq" len: 4 is_ref: 0 refcount: 1 }`

$foo = "baz";
$bar = $foo;

bar,foo: {
    type: string,
    value:
        str:
            val: "baz"
            len: 3
    is_ref: 0
    refcount: 2
}

$bar .= "q";

foo: {
    type: string,
    value:
        str:
            val: "baz"
            len: 3
    is_ref: 0
    refcount: 1
}
bar: {
    type: string,
    value:
        str:
            val: "bazq"
            len: 4
    is_ref: 0
    refcount: 1
}

This technique is called "copy on write" and it allows to reduce memory consumption quite well. Also, refcount is needed for the garbage collector, which removes from memory all zvals, which have refcount = 0.

And what happens with references? And is_ref is working? That's very simple: if you create a reference from a variable, the is_ref flag becomes 1, and the above optimization for this zval will not be applied.

PHP Under the hood

PHP	Under the hood
`$foo = "baz"; $bar = $foo;`	`bar,foo: { type: string, value: str: val: "baz" len: 3 is_ref: 0 refcount: 2 }`
`$baz = &$foo;`	baz,foo: { type: string, value: str: val: "baz" len: 3 is_ref: 1 refcount: 2 } bar: { // variable `bar` was allocated to a separate `zval` type: string, value: str: val: "baz" len: 3 is_ref: 0 refcount: 1 }
`$qwe = $foo;`	baz,foo: { type: string, value: str: val: "baz" len: 3 is_ref: 1 refcount: 2 } bar: { type: string, value: str: val: "baz" len: 3 is_ref: 0 refcount: 1 } // this variable was also allocated to a separate `zval` qwe: { type: string, value: str: val: "baz" len: 3 is_ref: 0 refcount: 1 }

$foo = "baz";
$bar = $foo;

bar,foo: {
    type: string,
    value:
        str:
            val: "baz"
            len: 3
    is_ref: 0
    refcount: 2
}

$baz = &$foo;

baz,foo: {
    type: string,
    value:
        str:
            val: "baz"
            len: 3
    is_ref: 1
    refcount: 2
}
bar: { // variable `bar` was allocated to a separate `zval`
    type: string,
    value:
        str:
            val: "baz"
            len: 3
    is_ref: 0
    refcount: 1
}

$qwe = $foo;

baz,foo: {
    type: string,
    value:
        str:
            val: "baz"
            len: 3
    is_ref: 1
    refcount: 2
}
bar: {
    type: string,
    value:
        str:
            val: "baz"
            len: 3
    is_ref: 0
    refcount: 1
}
// this variable was also allocated to a separate `zval`
qwe: {
    type: string,
    value:
        str:
            val: "baz"
            len: 3
    is_ref: 0
    refcount: 1
}

Interesting in reading more about PHP? Follow this link to find more articles 😉

Datastructures in PHP

Sergey Podgorny — Tue, 17 Dec 2019 12:44:59 +0000

The development process in PHP is mostly related to the receipt and processing of data from different sources, such as databases, local files, remote APIs, etc. Developers spend a lot of time organizing data, getting, moving and processing it. The most used structure for representing data in PHP is an array. However, in some cases, arrays are not suitable for solving problems due to insufficient performance and excessive memory consumption, and therefore more appropriate data structures are required.

Usage of the Standard PHP Library, SPL and knowledge of its composition is an area whose possession can confirm the competence of the PHP developer.

SPL is like patterns, but purely for data. Mostly it doesn't simplify writing of code, but simplify its understanding by others (or by yourself after a while). Of course, if you used it well and right.

Doubly-linked lists

SplDoublyLinkedList - Doubly-linked lists are a variation on "standard" linked lists where each node has a pointer to the previous node as well as a pointer to the next node.

Imagine that you are in the queue at the bank and at the same time you can see only the person in front of you and behind you. This is an analogy of the relationship between the elements in the SplDoublyLinkedList. Inserting an item into the list corresponds to the situation when someone climbed into the queue, and you suddenly forgot who was standing in front of you (and this someone forgot about you). A doubly-linked list allows you to efficiently bypass and add large data sets without re-hashing.

SplDoublyLinkedList
- SplStack
- SplQueue

SplQueue and SplStack are very similar to SplDoublyLinkedList. Both these structures, in fact, are doubly linked lists with different iterator flags (IT_MODE_LIFO - Last In First Out, and IT_MODE_FIFO - First In First Out), which regulate the order of node processing and what to do with these elements after they have been processed. Another difference between these structures is that the SplQueue interface contains more intuitive enqueue() and dequeue() methods, unlike the push() and pop() methods of SplStack.

    $stack = new \SplStack();

    // add items to the stack
    $stack->push('1'); 
    $stack->push('2');
    $stack->push('3');

    echo $stack->count();       // 3
    echo $stack->top();         // 3
    echo $stack->bottom();      // 1
    echo $stack->serialize();   // i:6;:s:1:"1";:s:1:"2";:s:1:"3";

    // retrieve items from the stack
    echo $stack->pop(); // 3
    echo $stack->pop(); // 2
    echo $stack->pop(); // 1

    $queue = new \SplQueue();

    $queue->setIteratorMode(SplQueue::IT_MODE_DELETE);

    $queue->enqueue('one');
    $queue->enqueue('two');
    $queue->enqueue('three');

    $queue->dequeue();
    $queue->dequeue();

    echo $queue->top(); // three

Stacks are widely used in the analysis or processing of nested data structures, in particular in the calculation of mathematical expressions.
Queues could be used when processing the lists of "jobs" or some tasks, as parsing the text input in the form of a list of individual elements, normalizing it in one cycle, and then process the normalized list in the other.

Heaps

Heaps are complete binary tree structures, that should satisfy heap-order property: each node is greater than or equal to the data stored in its children. Each level of a tree is completely filled, except possibly the bottom level (at this level it is filled from left to right).

SplHeap
- SplMaxHeap
- SplMinHeap
SplPriorityQueue

SplHeap is a heap represented as a binary tree, each node of which has no more than two child nodes. This is an abstract class that requires an implementation of the compare() method, which allows real-time sorting while inserting new nodes into a tree.

    $heap = new \SplMaxHeap();
    $heap->insert('111');
    $heap->insert('222');
    $heap->insert('333');

    echo $heap->extract(); // 333
    echo $heap->extract(); // 222
    echo $heap->extract(); // 111
    echo $heap->extract(); // Exception: Can't extract from an empty heap

    $heap = new \SplMinHeap();
    $heap->insert('111');
    $heap->insert('222');
    $heap->insert('333');

    echo $heap->extract(); // 111
    echo $heap->extract(); // 222
    echo $heap->extract(); // 333
    echo $heap->extract(); // Exception: Can't extract from an empty heap

SplMaxHeap and SplMinHeap are concrete implementations of the abstract class SplHeap. SplMaxHeap implements the compare() method so that the tree is sorted in descending order of node values, and SplMinHeap is in ascending order of values.

SplPriorityQueue is a queue similar to SplHeap, but unlike SplHeap, sorting based on the value of the priority property assigned to each node.

    $queue = new \SplPriorityQueue();
    $queue->setExtractFlags(SplPriorityQueue::EXTR_DATA); // extract only values of elements

    $queue->insert('Q', 1);
    $queue->insert('W', 2);
    $queue->insert('E', 3);
    $queue->insert('R', 4);
    $queue->insert('T', 5);
    $queue->insert('Y', 6);

    $queue->top(); 

    while ($queue->valid()) { 
        echo $queue->current(); 
        $queue->next(); 
    }
    //YTREWQ

Arrays

SplFixedArray is an array of fixed length, the indixes of which can be only integers (which are greater than or equals 0). These restrictions provide a higher processing speed of the array, which is achieved, due to the fact that in SplFixedArray there is no hashing of the keys of the elements when they are added (in contrast to the usual arrays). The length could be changed, but this is a costly operation.

Map

SplObjectStorage is an object storage, that provides an interface for mapping objects to data, or can be used as a container for multiple objects. Allows you to use an object as a key of associative array and associate it with some data.

$storage = new \SplObjectStorage();

$o1 = new \stdClass();
$o2 = new \stdClass();

$storage->attach($o1);

var_dump($storage->contains($o1)); // bool(true)
var_dump($storage->contains($o2)); // bool(false)

$storage->detach($o1);

var_dump($storage->contains($o1)); // bool(false)
var_dump($storage->contains($o2)); // bool(false)

$storage = new \SplObjectStorage();
$object = new \stdClass();

$storage[$object] = "data from object";

echo $storage[$object]; // data from object

PHP isn't C. That’s all you should remember while working with data. You can't expect that a super dynamic language like PHP has the same highly efficient memory usage that C has. But if you do want to save memory you could consider using an SPL for large, static arrays.

Interesting in reading more about PHP? Follow this link to find more articles 😉