The latest articles on DEV Community by Breno Helfstein Moura (@brenohelf). https://dev.to/brenohelf https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F391055%2F32d8720e-bacc-4fed-b7d2-7235674c3d65.jpeg https://dev.to/brenohelf en Breno Helfstein Moura Wed, 20 May 2020 18:00:27 +0000 https://dev.to/brenohelf/you-learn-the-most-when-you-fail-distributed-programming-project-3aaa https://dev.to/brenohelf/you-learn-the-most-when-you-fail-distributed-programming-project-3aaa <h2> The project </h2> <p>This project was done for network and distributed programming subject of University of São Paulo. The idea of the project was to build a distributed program that would run on different nodes, and it had the goal of sorting an array (using all the nodes in our network). It needed to support the addition of new nodes, death of nodes, and the change of "leader nodes" that would distribute the work load. The array would start in one of the nodes that would act as a "master".</p> <h2> Link to Code </h2> <div class="ltag-github-readme-tag"> <div class="readme-overview"> <h2> <img src="https://res.cloudinary.com/practicaldev/image/fetch/s--vWogaON8--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://practicaldev-herokuapp-com.freetls.fastly.net/assets/github-logo-28d89282e0daa1e2496205e2f218a44c755b0dd6536bbadf5ed5a44a7ca54716.svg" alt="GitHub logo"> <a href="https://github.com/breno-helf"> breno-helf </a> / <a href="https://github.com/breno-helf/Network-Distributed-Prog"> Network-Distributed-Prog </a> </h2> <h3> Porgrams done for "Computer Networks and Distributed Systems" subject from University of Sao Paulo </h3> </div> <div class="ltag-github-body"> <div id="readme" class="md"> <h1> Redes</h1> <p>Porgrams done for "Computer Networks and Distributed Systems" subject from University of Sao Paulo</p> <p>Each folder represent a different project. EP means "Exercicio Programa" in portuguese, and it can be translated by "Programming assingment".</p> <h1> EP1</h1> <p>We had to code a simple FTP server, with some functionalities. This project had to be done in C.</p> <h1> EP2</h1> <p>We had to do a program that would sort an array in a distributed fashion. It had some restrictions, such it had to implement leader election (even though it maybe unnecessary).</p> <h1> EP3</h1> <p>We had to follow a tutorial to implement a switch and firewall with OpenFlow.</p> <h1> EP4</h1> <p>In this assingment we had to explore a security breach, we choose to explore a CRLF of a specific golang version.</p> </div> </div> <br> <div class="gh-btn-container"><a class="gh-btn" href="https://github.com/breno-helf/Network-Distributed-Prog">View on GitHub</a></div> <br> </div> <br> <h2> How I built it </h2> <p>To the project we had freedom of choice. We, (Me and my teammate <a href="https://github.com/matheushw">Matheus Cunha</a>) choose Golang for the language and made some "simple" architecture choices. The first one was how to divide the tasks, we choose to divide the array into chunks, that would be sorted in the nodes with the default sort function of golang. The tasks would come from the master node (that has the data), and the receivers of each task would be defined by the leader node.</p> <p><a href="https://res.cloudinary.com/practicaldev/image/fetch/s--SGPmxROK--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/i/ezha4ravaehripmqz5r8.png" class="article-body-image-wrapper"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--SGPmxROK--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/i/ezha4ravaehripmqz5r8.png" alt="Chunk division"></a></p> <p>Then we would send the array back to a "master" node, (The one that started with the array). The master node would then register the array into its filesystem.</p> <p><a href="https://res.cloudinary.com/practicaldev/image/fetch/s--o5gYdcWj--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/i/9fty9ln2n1z2454yno9f.png" class="article-body-image-wrapper"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--o5gYdcWj--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/i/9fty9ln2n1z2454yno9f.png" alt="Simple Architechture"></a></p> <p>Later, when all chunks are sorted and stored in the master, the master node would perform the last step, that is going though all chunks in order (we would have a pointer for each chunk in memory) and storing them in a heap, doing some sort of "heap merge" that would take O(N * log(chunks)). </p> <p>As you may be expecting by the title of this post, the project "failed" in some way. Although it sorted the chunks correctly, it took way more time distributing the tasks than sorting it, so one node used to work better than N nodes (We tested with 1, 2, 3 and 4 nodes, all raspberry pies). The reason it took so long could be many. The extra "leader" -&gt; "master" step (that was required, due to need of implementing leader election, a requirement of the assignment), can be the issue. Maybe its because we used raspberry pies for testing. However, my favorite candidate is the way we decided to transfer the data, through strings that would be parsed by the other side (Serializing and then de-serializing). Just to "compress" the chunk data, would take longer than to sort the entire chunk, which made the whole idea useless.</p> <h2> Why failing this project was good </h2> <p>Well, during this project I think I learned a lot of things. More than just exercising distributed and parallel concepts in practice, I learned what is important, how to measure and diagnoses a problem in a distributed environment and how to think solutions for those problems. None of this things would be learned if I didn't fail, and in the end I am glad I failed. The project didn't had any major goal or responsibility and I learned how to not fail in more important projects.</p> octograd2020