DEV Community: Tiago Mesquita

When parallelStream is not a big deal in Java

Tiago Mesquita — Wed, 10 Jun 2020 14:29:19 +0000

To understand when you should use or avoid the use of parallelStream is important to understand the concept of stateful and stateless in java streams.

Stateful

Whenever an action in a stream needs to keep a state to finish its works, the operation would be considered a stateful operation, these actions would be the invocation of operations like:

distinct()
sorted()
limit()
skip()

And many others conditions that you can put in your code and will make it stateful. For a fast way to determine if an operation is stateful I use the documentation when I'm not sure:

A stateful operation should not be used in parallel, it will kill performances!

Stateless

Stateless operations keep no state during the pipeline execution, because of this characteristic those operations is much more performative.

Nondeterministic values

Because of the stateful behavioral, it's not possible to determine the execution result when running it in parallel, let's see an example in code:

for (int i = 0; i < 5; i++) {
    Set<Integer> alreadySeen = new HashSet<>();
    IntStream stream = IntStream.of(3, 4, 1, 2, 1, 2, 3, 4, 4, 5);
    int sum = stream.parallel().map(
            // Here we add a stateful behavioral parameter.
            value -> alreadySeen.add(value) ? value : 0).sum();
    System.out.println(sum);
}

And here is the output:

This result is nondeterministic, you gonne receive a different result for each execution you do, so to make it correct we gonna change our code taking off the parallel():

for (int i = 0; i < 5; i++) {
    Set<Integer> alreadySeen = new HashSet<>();
    IntStream stream = IntStream.of(3, 4, 1, 2, 1, 2, 3, 4, 4, 5);
    int sum = stream.map(
            // Here we add a stateful behavioral parameter.
            value -> alreadySeen.add(value) ? value : 0).sum();
    System.out.println(sum);
}

Now we can see a correct result that will be expected to be equal in all new execution:

This is because when executing an action in parallel, it is not possible to guarantee a correct validation with a previous state.

Workarounds

There're some workarounds that would help us with this, but these fix would undermine the benefits of parallelism.
Use of a synchronizedSet:

Set<Integer> alreadySeen = Collections.synchronizedSet(new HashSet<>());

Another one is to use distinct() that will keep the operation stateful, but is safe and deterministic:

int sum = stream.parallel().distinct().sum();

The use of parallel can speed up processing, but it can also cause problems in the application, so it’s important to know if we’re in a stateful or stateless flow to know how to handle it.

Web Scraping in 3 steps with Python

Tiago Mesquita — Wed, 20 May 2020 00:01:37 +0000

Web Scraping is a technique that makes possible to extract data from HTML files. This resource was widely used by Google to create its search engine, based in the href tag they would know the most popular links, following new links and rank the pages based on how many times it was referred.

Here we're going to learn how to make web scraping with Python in just 3 steps, but before we go on, let meet the page we're going to explore.

INUMERÁVEIS

Inumeráveis (means innumerable) is a page created in Brazil to be a memorial dedicated to the history of each of the victims of the coronavirus in Brazil.
The site presents the age of the victim and it's a value that our health system isn't making public. Questioning myself about the lethality by age I thought about making a scraping to get an idea about this number.
The page shows the victim age aside the name (red circle):

We need to identify the HTML tag that holds the value, so we need to inspect it:

Now we know that the value we're looking for is inside a h4 > span tag, we can start constructing our scraping in three simple steps:

STEP 01: Get the HTML

# pip install requests BeautifulSoup4 matplotlib

from requests import get
from requests.exceptions import RequestException
from contextlib import closing
from bs4 import BeautifulSoup
import matplotlib.pyplot as plt

def simple_get(url):
    try:
        with closing(get(url, stream=True)) as resp:
            if is_good_response(resp):
                return resp.content
            else:
                return None

    except RequestException as e:
        log_error('Error during requests to {0} : {1}'.format(url, str(e)))
        return None


def is_good_response(resp):
    content_type = resp.headers['Content-Type'].lower()
    return (resp.status_code == 200
            and content_type is not None
            and content_type.find('html') > -1)


def log_error(e):
    print(e)


html = BeautifulSoup(simple_get("https://inumeraveis.com.br/"), 'html.parser')

This step uses the BeautifulSoup to parse our raw HTML content returned by the simple_get function in a form that we can use it programmaticaly.

STEP 02: Collect and sort the data

ages = {}

for item in html.select('h4 span'):
    ano = int(item.text.replace(' anos', ''))
    if ano not in ages.keys():
        ages[ano] = 1
        continue
    ages[ano] = ages.get(ano) + 1

ages = sorted(ages.items(), key=lambda x: x[0])
print(ages)
xAge = []
yCount = []
for key, value in ages:
    print(f'Idade: {key} Ocorrências: {value}')
    xAge.append(int(key))
    yCount.append(int(value))

The most important step, here we are searching for the data (age) that we need and we are creating a new data that is the number of times that an age appears (count), furthermore we're sorting the list of ages by its age (lambda expression) so we can use it to plot our graphic.

STEP 03: Plot the graphic

def createGraphic(xAge, yCount):
    fig, ax = plt.subplots()
    ax.bar(xAge, yCount, align='center', alpha=0.5)
    ax.set_xlabel('x - idade')
    ax.set_ylabel('y - quantidade')
    plt.title('Quantidade de mortes por idade - COVID-19')
    ax.xaxis.set_ticks(xAge)
    ax.yaxis.set_ticks(yCount)
    ax.legend()
    plt.gca().margins(x=0)
    plt.gcf().canvas.draw()
    tl = plt.gca().get_xticklabels()
    maxsize = max([t.get_window_extent().width for t in tl])
    m = 0.4
    s = maxsize / plt.gcf().dpi * len(xAge) + 15 * m
    margin = m / plt.gcf().get_size_inches()[0]
    plt.gcf().subplots_adjust(left=margin, right=1. - margin)
    plt.gcf().set_size_inches(s, plt.gcf().get_size_inches()[1])
    plt.show()


createGraphic(xAge, yCount)

In this third and last step we're ploting a graphic with the collected data, for it we're using the matplotlib library. Here i'm using a bar graphic.
The reason that this code becomes a bit large is because of some more steps taken to enlarge the labels text. Now the generated graphic:

The project code is accessible here.

Conclusion

Looking at the graphic was possible to see that the age group with highest risk is in its Sixty-ish, but almost all ages represents risk, we could see that the youngest victim was 18 years old only and the oldest 103 years old. The numbers show that there's not a safe zone, but maybe a slight dangerous zone, so what we really learned here is: #StayHomeStaySafe

Learn Docker Network

Tiago Mesquita — Wed, 29 Apr 2020 16:48:54 +0000

What is called network in docker is an abstraction created to make it simple to manage the communication among containers and external nodes. There's nothing much in common with the IP network (as example 10.0.0.1/24...)

Docker's default networks

Docker gives to you three default networks with specific configurations, to see those just use the command:

$ docker network ls

Those networks are: bridge; none; host.
Talking first about the bridge:

Bridge

Each container started in Docker is associated to a specific network, without that association, the default will be bridge, this network received an address that is avaliable within network IP 172.17.0.0/16 (CIDR) and all containers into this network may connect through TCP/IP. But wait, you must know the IP of the container that you wanna connect to, and this is difficult because those IPs are automatically given, so there's this thing called "--link", you gonna use it when you want to specific a DNS to a container that you are starting, a specific container, so the container A can connect to container B.
To explain this brief case, lets create a container A that wanna connect to B, first creating B:

$ docker container run -d --name B [image_name]

And now creating A:

$ docker container run -d --name A --link B [image_name]

Now you can ping A to B simply using this command:

$ docker container exec -it A ping B

But now unlearn it (really), it's an old way (legacy), so now the Docker treats it through DNS resolution automatically. An important thing to mention here is that all containers within this network will have access to all internet resources that the host has, and you can expose ports of the container to the outside world too.

None

This network is simple to explain, its purpose is to only isolate the container to the outside world and for the host. This network simply says that there's no network inside the container and its purpose of use is for containers that don't need it, containers that will use only mapped volumes to make backups, for example.

Host

This kind of network simply expose all host's interfaces to the container. In certain way this host is faster cause there isn't a bridge in the way of the packets, but the bridge still important to secure and manage the network.

User network

In Docker the user can create your own network, that created network is associated to a thing named "network driver", each network created by the user must be associated to a determined driver, and in case where the user don't create a driver, he must choose among the different drivers provided by docker.

Bridge (Driver)

Yes, there's a driver called "bridge" too, and the default network created by the user is similar to bridge network, but pay attention:

Created network have all features present in bridge network, plus it makes unnecessary to provide a "--link" to connect to other container, it uses an internal DNS resolution from Docker to connect its containers, it's specially an interesting thing when we are using the docker-compose to create more than one container that establish communication.
But now, let see how we can create a new network named "my_network":

$ docker network create --driver bridge my_network

And we can see that as a result of the command:

$ docker network list

We get:

NETWORK ID          NAME                DRIVER              SCOPE
3cf03b36d2fe        bridge              bridge              local
42517bb97000        host                host                local
a0ad7b22784f        my_network          bridge              local
1f98998e1711        none                null                local

Now to a better understand of what we learnt until now, let's create a container named "container_a" and another named "container_b" and connect both to our network "my_network" and ping "container_b" from "container_a", first creating "container_b":

docker container run -d --net my_network --name container_b alpine tail -f /dev/null

And then create container_a and tell him to ping container_b:

docker container run -it --net my_network --name container_a alpine ping container_b

And that is our output:

PING container_b (172.18.0.2): 56 data bytes
64 bytes from 172.18.0.2: seq=0 ttl=64 time=0.080 ms
64 bytes from 172.18.0.2: seq=1 ttl=64 time=0.114 ms
64 bytes from 172.18.0.2: seq=2 ttl=64 time=0.084 ms
64 bytes from 172.18.0.2: seq=3 ttl=64 time=0.068 ms
64 bytes from 172.18.0.2: seq=4 ttl=64 time=0.069 ms
64 bytes from 172.18.0.2: seq=5 ttl=64 time=0.197 ms
64 bytes from 172.18.0.2: seq=6 ttl=64 time=0.209 ms
64 bytes from 172.18.0.2: seq=7 ttl=64 time=0.090 ms
^C
--- container_b ping statistics ---
8 packets transmitted, 8 packets received, 0% packet loss
round-trip min/avg/max = 0.068/0.113/0.209 ms

As we saw the Docker automatically created the DNS resolve and gave the IP 172.18.0.2, so it becomes transparent as we just need to refer to container_b by its name.

Now a curious thing is that we can see which containers are within our new created network, to do so run the command:

docker network inspect my_network

And you'll get:

[
    {
        "Name": "my_network",
        "Id": "a0ad7b22784f70f3cd21edc0868fb84be5cf11ebd516dcb760f096f1a683607a",
        "Created": "2020-04-29T14:57:15.2716747Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "5aafff23f51d44836b2de86b7ee7686d8187fbe57a450fa095139b46f9fe7081": {
                "Name": "container_b",
                "EndpointID": "f724a6b2adb8dbd3e04716963058f1959e2684abe2b6dbb17e05cad24e89b9a6",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            },
            "5b2d2708c8d4392c4f1e2bda3009e9498c64c4cd12cecafbbcafc2f6593453d1": {
                "Name": "container_a",
                "EndpointID": "7f59020df5559e77ac8bff04204ede21328a3c55d82695f4aa7f8a311c9b2a85",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

Overlay

I'll not go deep in this topic here, but just to know the purpose of Overlay driver is to establish connection among hosts docker, it has a bit complex configuration but i full recommend you to search about if you gonna need to communicate containers among different hosts.

Ok, we finished for now, I hope you enjoy it!

This post is part of a series about docker, to see more please follow the tag series: docker-by-tiagomac, keep learning and practicing.

Learn Docker Storage

Tiago Mesquita — Wed, 29 Apr 2020 00:55:10 +0000

Here we gonna see the different ways that the docker storage files.

Unification filesystem - AUFS

AUFS is one of the most common among Docker installations, it is based in layers, and when a layer creates a new file it will exists only in this layer or derivate (upper) containers.

Whenever you wanna modify a file that besides in bottom layers the Docker uses a technique called CoW (Copy-on-write), where the file of the lowers layers still unmodified (the files in bottom layers are read only to preserve its state), then Docker handler with this simple creating a new copy of this file in the new layer with the modified part attached, do you saw the problem here? Even if a small part of the file is modified, all the file must be copied.

This is a technique named whiteout, and the new files are marked as whiteout file.

Volume mapping

This is the simplest known way and maybe most used by developers to put the application code inside a container, you just need to give Docker a link between a folder in your host and your container, to do so you need to run your container with a -v param, like in this example:

$ docker container run -v /var/lib/container1:/var ubuntu

With the -v param we are telling the docker to make a link between the /var/lib/container1 folder located in the host with the folder /var located in the container.

This method had a problem that is not being portable, so the host that will execute must have the necessary files.

Mapping through data container

This model creates a container and inside of it is given a volume to be used by other containers. To create this container just do:

$ docker create -v /dbdata --name dbdata postgres /bin/true

To consume the data in other containers just run with:

$ docker container run -d --volumes-from dbdata --name db2 postgres

Now the container db2 gonna have a folder named /dbdata that is the same as in the container dbdata, making the process fully portable. The counterpart of this method is because you gonna need to keep a container just for it.

Mapping volumes

Since Docker 1.9 we can create volumes in an isolate mode, just use this commando:

$ docker volume create --name dbdata

The above command the docker creates a volume that can be used by any container. To use this volume, you just need to use this command when creating a new container:

$ docker container run -d -v dbdata:/var/lib/data postgres

At this point the folder /var/lib/data within the container will point to dbdata volume.

This last model is recommended since the release because it's not easily lost when the container is deleted and still easy to manage it.

This post is part of a series about docker, to see more please follow the tag series: docker-by-tiagomac, keep learning and practicing.