DEV Community: Emre Savcı

Troubleshooting Kubernetes Application - Istio Service Mesh Upstream Connect Error

Emre Savcı — Sat, 10 Jun 2023 20:55:00 +0000

Hello everyone, in this article I will explain how did we troubleshoot a network error between our microservices.

Content

🚀 Service Mesh in Trendyol
❌ Error Case
📝 Determining Troubleshooting Paths
🚨 Reproducing the Error
🕵️ Sniffing the Pod Network
🔍 Inspecting Code
✅ Problem and Solution
📚 Conclusion

Service Mesh in Trendyol 🚀

In Trendyol, our microservice applications run in Kubernetes along with service mesh Istio. We have thousands of clusters, almost ~200K pods running on it.

We empower microservices with the help of service mesh features like service authorization, load balancing, weighted request routing, traffic mirroring, interrupting incoming and outgoing requests, manipulating request flows etc.

Beyond that, we are extending our service mesh functionality by leveraging Envoy’s extensibility features as we described in Envoy Wasm Filters article.

If you wonder more about Trendyol infrastructure, you can take a look our infra metrics publicly on here.

inframetrics.trendyol.com

Error Case ❌

Couple of days ago, we (Platform — Developer Productivity Engineering team) got a support request from one of our domain teams for a microservice that they developed. They described issue as a connectivity problem between a two services. The strange point is the application was facing with issue while sending a request to existing resource (/resources/1) and working correctly on non existing resource (/resources/133).

To reproduce the error we demanded the example requests and application urls. From now on, I explain the situation and the steps we followed.

Determining Troubleshooting Paths 📝

While troubleshooting an error, visualizing the process with big picture helps a lot to reveal overlooked points. So we started to draw the request flow.

What we know about process is:

We have a reverse proxy in-front of Kubernetes
We have Istio Envoy Proxy in pods
We have two service that run in Kubernetes and access each-others
Gateway service handles the request and makes a call to an API

Let’s see the big picture 👇

When a user make a request to a URL, it is handled by Tengine and then Tengine forwards request to the cluster, request first handled via Istio Ingress Gateway inside clusters. Inside the pod, request handled by istio-proxy (Envoy) container and then forwarded to the application container.

Now we can ready to dig in.

Reproducing the Error 🚨

According to those informations, we started to eliminate components one by one. First we sent requests by bypassing load balancer (Tengine). Because we now that load-balancer can access to Kubernetes because it works on some requests to the same URL.

So we are now removed the Tengine from big picture 👇

We directly sent requests to the Kubernetes cluster’s ip address and Istio Ingress Gateway port. To be sure about Istio configuration, we also sent direct requests to NodePort port of our service, this bypasses Istio Ingress Gateway from our request flow.

But either way, we faced with the same following result 👇

The error says “upstream connect error or disconnect/reset before headers. reset reason: connection termination”.

We wanted to sure that our upstream API works correctly, so we removed the gateway from the picture and we directly sent requests to the API 👇

We got our responses successfully for both existing and non existing resources while sending direct requests to the API.

After bunch of request combinations to the both gateway and API, we began considering the likelihood of an error occurring within the communication of the gateway pod.

Sniffing the Pod Network 🕵️

We could not find any reason to this error just by outlooking by sending requests. We decided to sniff the gateway pod’s network and see whats going on.

Thanks to the wonderful kubectl plugin named “ksniff” we easily sniff our network and track it on Wireshark.

To sniff our pod network let’s run the kubectl command below:

kubectl sniff pod-name -f 'tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst|tcp-push)!=0'

When we start sniffing it opens up a Wireshark with bunch of package traces like in the following screenshot👇

As you can see, it contains a lot of unnecessary informations, we need to track a specific request flow.

We started to send couple of different requests to the application to see whats happening on the network so we can filter out easily by typing a filter like http.request.full_uri contains “/eligibility” and this shows us filtered requests👇

And now we can select a request and follow it’s HTTP or TCP stream by right clicking on it 👇

Now when we follow this process on our invalid request, we can easily see that our TCP stream does not contain any error 👇

When we do the same thing on the failed responses, we faced with the following screen👇

As you can see there is a big difference between two TCP stream. Failed stream contains [RST] flag. This means that somehow the application causes to close connection with istio-proxy (Envoy).

We can actually see that our gateway application communicates with the Eligibility API.

So we can downgrade the error scope to the only gateway application pod. There must be something between gateway container and istio-proxy (Envoy) container.

Inspecting Code 🔍

Finally we ended up with the idea of there must be a difference between response flows of this application.

There is one thing to take a look: code difference between the error and success responses.

When we took a look to source code of gateway application, we notices that application has a global error handler which returns a fresh new response object created by handler.

BUT…

In the controller requests are sent by Feign Client to the API application and the response is directly return to the caller ⚠️

We found the reason of the error is because of this behavior of the controller. It corrupting the response header which expected by istio-proxy.

Problem and Solution ✅

Here is the problematic code that we saw in the controller 👇

@GetMapping("/some/path")
public ResponseEntity<Response> getResponse(
    @RequestParam(name = "id", required = false) Integer id) 
{
    return externalApiClient.get(id); //directly returning the ResponseEntity<Response> from Feign client
}

We made a tiny change to fix this issue, just by wrapping the response body of the API call request with a newly created Response object👇

@GetMapping("/some/path")
public ResponseEntity<Response> getResponse(
            @RequestParam(name = "id", required = false) Integer id)
{
    return ResponseEntity.ok(externalApiClient.get(id).getBody());  //creating a fresh new  ResponseEntity<Response> from Feign client response body
}

It fixed all of our issues.

Just because we got this issue in Java it does not mean that you may not encounter it in another language. This problem is language agnostic. You may encounter it in Go, C# or any other language as well.

Remember the main reason of this issue is directly returning response headers too of external requests. You need to create a custom response 👍

Conclusion 📚

We made couple of inferences after resolving this problem.

Do not directly proxy responses of your external requests to the caller, wrap it with custom response
Code implementation may vary over different flows, do not make assumptions
Always try to make your application observable

We learned a lot while investigating this issue. Thanks a lot to my team mates Onur Yilmaz and Gokhan Karadas.

I hope you find this article entertaining and informative. Thanks for your attention, happy coding ⌨️

Ready to take your career to the next level? Join our dynamic team and make a difference at Trendyol.

Trendyol

jobs.lever.co

Golang -What is Broken Pipe Error? Tcp Http Connections and Pools

Emre Savcı — Mon, 06 Feb 2023 13:39:48 +0000

Have you ever encountered a "broken pipe" error in the applications you have developed, during database operations or when making requests to the systems you have connected over the network?

In this article, we will look at the causes and solutions of some errors that you may encounter in network operations while using Golang, wherever TCP connection is established. These network operations can be operations that you perform by establishing a database connection or by making an HTTP request.

Although Golang is mentioned in the topic and content, you can apply the explained approach here in many programming languages. You can even apply the approaches to be explained when designing a system, communicating with multiple applications, using Reverse Proxy, Load Balancer, and client-server communications.

Connection Pools are Everywhere

Most application developers that make network communication use the connection pool structure consciously or unconsciously. Because it is a costly operation to open TCP connections over and over, "pool" structures are well suited to improve application performance and resource utilization. HTTP packets, database drivers or libraries, web frameworks, load balancers and reverse proxies, in short, most systems that communicate via TCP in the background (if there is no inherent contradiction) embrace "connection pool".

HTTP Client - Server Communication with Golang

Let's take a look at how a connection pool has a place in HTTP operations and how it is used in the Golang HTTP package.

HTTP Keep Alive

When you write an HTTP server with Go and send a request via a standard Go HTTP client, the opened "connection" will be stored on a pool to be used again in future requests, unless you specify otherwise.

Now let's create a simple HTTP server instance with the following code:

The server is running on the "8080" port, and it outputs the connection statuses on the server with the ConnState function as console output.

Now let's write the HTTP client code to send our request:

We use the httptrace package to monitor client connection status. The thing to note here is that the http response is read completely with the io.ReadAll() function. If we do not do this, the connection keep-alive function will not be performed.

Let's run our server and client code and look at the outputs:

Our server application prints the connection status as New for the first request and remains Active throughout the request. When the request is completed, the opened connections switches to the Idle state.

Subsequent requests switch between Active and Idle states. Because a new connection is not opened, the first connection is used again and again.

Let's look at the output of our Client application during this process:

When the first request is sent, we see that the connection start process takes place. When the request is completed, the connection switches to the Idle state and the same connection is used again for subsequent requests.

Go maintains a connection pool on the Transport type within the HTTP client.

In the screenshot below, we can see the idle conn pool in the Transport definition.

type Transport struct {
   idleMu       sync.Mutex
   closeIdle    bool                                // user has requested to close all idle conns
   idleConn     map[connectMethodKey][]*persistConn // most recently used at end
   idleConnWait map[connectMethodKey]wantConnQueue  // waiting getConns
   idleLRU      connLRU
...
}

Idle Connection Settings

If we want, we can make various settings for idle connections on both client and server.

We can make the following settings for idle connections to be kept on the HTTP client:

http.Client{
  Transport: http.Transport{
   DisableKeepAlives:   false,
   MaxIdleConns:        100,
   MaxIdleConnsPerHost: 5,
   IdleConnTimeout:     60 ,
}}

DisableKeepAlives : Does not maintain keep alive connections if set as true

MaxIdleConns : Total connection number can be hold on Transport

MaxIdleConnsPerHost : Total connection number can be hold for a specific host on Transport

IdleConnTimeout : Idle connection timeout on the pool

Creating a TCP Connection in Golang

We made a quick introduction to the connection pool structure with HTTP requests and saw its implementation on a real example.

Now let's create a TCP server ourselves and establish a connection.

We created a TCP server listening on port 8080 with the Golang net package. And we processed all incoming connections with a goroutine.

We created a client that makes TCP connection requests using the net package. We sent the hello data over the connection we created in the loop.

Now let's run our program and look at the application outputs.

We ran our server application and immediately after that we also ran our client application. Our server application has received a new request and printed the received data to the screen.

Our client application also successfully wrote the size of the data it sent to the screen.

Broken Pipe Error

So what happens if we close our server application while both applications are running?

Now, after running the example above, let's end the server application for a while and look at the output.

After closing our server application, our client application gives an output as above.

After the server is shut down, our client application encounters an error as seen. The error message appears exactly as write tcp 127.0.0.1:58766->127.0.0.1:8080: write: broken pipe.

As it can be understood from here, if an opened TCP connection is closed by the server, it cannot be notified until the client side writes.

Lets Create Our Own TCP Connection Pool

We have seen how the Broken pipe error happens. If an opened TCP connection is closed by the server while it is still in use, client applications receive an error as expected.

Let's create a connection pool to bring this error closer to real life examples.

This is quite basic pool. Let's run our client application again.

When we close the server after a while after running the client application, we see that the connection received from the pool on the client side encounters with a broken pipe error.

In our examples we are talking about closing the server application, but this may not always be the case. In fact, the important thing here is to close the connection on the server side.

An idle timeout value can be given for connections created on the server side, and the connections of requests that are satisfied after a certain period of time can be closed automatically.

In such a case, a connection opened and used by the client and stored in the idle pool may be closed by the server. In such a scenario, the client will encounter a broken pipe error when trying to reuse the connection from the idle pool.

Where can we encounter with this broken pipe error?

Database Connections: The libraries we use while connecting to databases can contain pools. We may use packages containing pools. In this case, we should consider that we may receive an error on the application side against the idle connections closed by the database.

Reverse Proxy & Load Balancer Applications: Many reverse proxies and load balancer applications have a connection pool structure on their own. In the configuration of such applications, the idle connection timeout values must be less than the timeout value specified on the server side, so that the connection is terminated by the client before the server shuts down.

See you on the next article, I wish the bug free developments to all of you.

Let's Connect:
Twitter
Github

You can support me on Github: Support mstrYoda on Github

No-Code Tools For Your Next SaaS & Side-Project

Emre Savcı — Tue, 06 Dec 2022 13:25:07 +0000

Hi all, hope you are well 👋 In this post I will introduce you couple of no-code tools that might help you to create your next SaaS or side-project. Now let's take a look into no-code tools 👇

AirTable

Our first tool is AirTable, which is known and used by many as a database tool.

AirTable allows you to create various databases, workflows, charts where you can store your data 💪

BaseRow

Another no-code database platform is BaseRow. They advertise themselves as AirTable alternative 💪

Notion

I am using Notion in my daily basis. It is quite helpful to organize my notes, ToDo lists, even my schedule on a calendar view 💪

Although it is not a database product we can use it as, if we want, we can also open a database page and keep our data on Notion.so.

AwesomeTable

What if you want to use the Google Sheet data on your site as a data source, instead of creating a database from scratch?

Here we come across a very nice product AwesomeTable. It allows us to connect our existing Google Sheets and create a website based on that data 💪

ParseHub

Sometimes you may need data collection (crawling) from various websites and in that case if you don't know how to code ParseHub will help you about it 💪

GlideApps

Let's say you want to create a mobile application with the data that you already have.

glideapps.com is a product that transforms your data from various sources such as Google Sheets, Airtable, Excel, BigQuery into a mobile application 💪

WebFlow

All that's left is to create your website. For this, I will share 2 different tools. The first of these is WebFlow.io.

It offers the opportunity to create a free design using its own editor and publish it under the webflow.io domain 💪

Pory

Another website creation tool is Pory.io.

You can create your website using your existing data by importing them from different data sources such as Airtable and Notion 💪

Now we have almost everything to quickly create our next SaaS product or side-project 🚀 Let's create a website, a landing page, create contents like blog posts, collect some data, visualize them in website and create a mobile application using that data. Good luck 🎉

Golang Blazing Fast Unit Tests - Fiber/fasthttp/http Internals and Optimizing HTTP Server Tests

Emre Savcı — Mon, 12 Sep 2022 07:35:46 +0000

Greetings, in this article, I will tell you how we can increase the test performance of our HTTP servers in the projects we have developed with Go.

Recently, while I was working on a new feature for an application we were developing, I realized that I could commute to get coffee while waiting for unit tests to complete. And it was not possible to run the tests in parallel. If we wanted to run it in parallel in its current form, this test period could have taken much longer.

Now, let's take a brief look at the content and move on to the details of our subject.

Go HTTP Internals

http.ListenAndServe
net.Conn & net.Listener
http.Client Dial Function
net.Pipe

Testing in Different Packages

Fiber

- app.Test Internals
- app.ServeConn

fasthttp

- fasthttputil.NewInmemoryListener

Go HTTP

- httptest.NewUnstartedServer
- Mock InMemListener Example

Our Test Performance Improvement

Conclusion

Go HTTP Internals

Let's take a look at what we need to do when we want to run a simple HTTP server with the Go language.

http.ListenAndServe

When we want to run a server that we wrote using the Go standard library, we use the ListenAndServe method.

Let's take a look at the details of the ListenAndServe method:

Now let's look at the details of the server.ListenAndServe() method:

Finally, let's look at the details of the srv.Serve(ln) method. I've removed some code so it's not too long.

The part that we should pay attention to here is the l.Accept method.

The summary of the process done in the codes I have shared so far is actually:

http.ListenAndServe method creates a server in the background and server ListenAndServe method is run
The server creates a net.Listener object that accepts a request from the port specified by the net.Listen("tcp", addr) method call and runs the Serve method.
The Serve method acts on the net.Conn object that represents the new incoming connections with the Accept() method of the net.Listener object that it receives as a parameter.

So far, we have seen the process of creating an API in its basic form.

Now, I can hear you slowly saying, "Well, sir, what will they do in real life?"

Don't worry, we'll soon see how we can improve our testing performance using this information.

net.Listen & net.Conn

Now let's take a closer look at these two basic types included in the HTTP package.

We see that these two types are actually interfaces.

Well, what does it mean? We can give special structs that we implement these interfaces as parameters to methods that expect these interface types as parameters. This is an important point for us.

http.Client Dial Function

Now let's look at how the http.Client object is used, which allows us to make HTTP requests with Go.
There are two methods of sending requests using the Go HTTP package.

The first method is to use the http.Get http.Post methods directly
The second method is to use the http.Client object resp, _ := http.Get("http://localhost:8080/test")

Actually, the first method, http.Get, also uses the http.Client object in the background.
Let's take a brief look:

In other words, both methods do the same thing in the background.

Now let's look at the fields that we can customize when creating the http.Client object.

http.Client provides all connection establishment processes with RoundTripper interface. And it uses the Transport object as a RoundTripper implementation in itself.

If we want, we can create this Transport object ourselves when creating http.Client.

The Transport object we created has a method called DialContext. This method is the method that enables connection opening in the background, it returns a value of type net.Conn as a variable.

In a moment, we will combine this information and the information that the server.Serve method we just saw is listening for connections with net.Conn using the accept method of the net.Listener variable that takes as a parameter.

Little by little the pieces are falling into place, aren't they?

net.Pipe

Finally, there is another method I would like to mention: net.Pipe(). It returns two net.Conn type variables as return value.

One of these variables can be used as a writing point and the other as a reading point. So what does this mean? How can we use this?

If you remember, the Listen() method of the Listener object returned the variable net.Conn. The Dial() method of the HTTP client Transport object also returns the variable net.Conn.

Starting from here, when we give one end of the connections returned by the net.Pipe() method to the Dial method of our HTTP client and the other end to the Listen method of our Listener object, the request we made with the HTTP client will create the connection expected by the Listener.

Testing in Different Packages

Now, let's create an API using several different web application libraries common in the Go language and write tests for them.

Fiber

First, let's take a look at the Fiber library, which we often use in our projects and to which I contribute.

Writing a test with Fiber is quite simple because it contains a direct Test method.

Now let's look at a sample Fiber application and how it was tested:

Here we need to pay attention to the line app.Test(r, -1).

We're sending our HTTP request here so we can test it.
So how did this HTTP request go without calling app.Listen(":8080") which we normally do to run a Fiber application and make it listen on a port?

Let's take a look at what's behind the app.Test() method (you can access all the code here):

When we examine the code, we see that Fiber actually creates its own connection objects by conn := new(testConn) in order to test the application, as we explained in the net.Pipe() section.
It writes the HTTP request to this conn object and sends this conn object to the server with the app.server.ServeConn(conn) call.

So what happens when we say app.Listen(":8080") in Fiber app normally? Let's take a quick look at this:

Here we see that Fiber does the net.Listen() operation within itself and gives the listener object it created to the server by saying app.server.Serve(ln). The server listens to the connections from this listener object and passes it as a parameter to the serveConn() method.

In other words, it implements an approach that brings together the Listener and net.Conn objects that we explained at the beginning, allowing us to test it without physically listening to the port.

gofiber / fiber

⚡️ Express inspired web framework written in Go

Fiber is an Express inspired web framework built on top of Fasthttp, the fastest HTTP engine for Go. Designed to ease things up for fast development with zero memory allocation and performance in mind.

⚠️ Attention

Fiber v3 is currently in beta and under active development. While it offers exciting new features, please note that it may not be stable for production use. We recommend sticking to the latest stable release (v2.x) for mission-critical applications. If you choose to use v3, be prepared for potential bugs and breaking changes. Always check the official documentation and release notes for updates and proceed with caution. Happy coding! 🚀

⚙️ Installation

Fiber requires Go version 1.22 or higher to run. If you need to install or upgrade Go, visit the official Go download page. To start setting up your project. Create a new directory for your project and navigate into…

View on GitHub

fasthttp

We said that Fiber is using the Fasthttp package in the background. If we want, we can develop high-performance API applications directly using the Fasthttp package.
Now let's see how we can test an application we developed with Fasthttp.

In the code above, we created a simple server and defined a handler, and when a GET request was made to the /test endpoint, we returned the "OK" response.

Here, unlike the previous code, we see a definition like ln := fasthttputil.NewInmemoryListener().

Next, we define a http.Client and give the Dial method of the ln variable we created to the DailContext method of the Transport object.

So does this remind you of anything? Yes, we mentioned above that we can give the net.Listener object to our http.Client variable and test it in-memory.
Fasthttp does exactly this approach with a utility package in the background.
Those who are curious about the details of the code can read it here.

valyala / fasthttp

Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http

fasthttp

Fast HTTP implementation for Go.

fasthttp might not be for you!

fasthttp was designed for some high performance edge cases. Unless your server/client needs to handle thousands of small to medium requests per second and needs a consistent low millisecond response time fasthttp might not be for you. For most cases net/http is much better as it's easier to use and can handle more cases. For most cases you won't even notice the performance difference.

General info and links

Currently fasthttp is successfully used by VertaMedia in a production serving up to 200K rps from more than 1.5M concurrent keep-alive connections per physical server.

TechEmpower Benchmark round 19 results

Awesome fasthttp tools

Switching from net/http to fasthttp

Fasthttp best practices

Tricks with byte buffers

Related projects

FAQ

HTTP server performance comparison with net/http

In short, fasthttp server is…

View on GitHub

Go HTTP

Now, let's write a test for an API that we developed with Go's own packages without using any external libraries.

Let's say we developed an API like above. So how do we test this?
We can write our tests by using the techniques we learned above here as well. Go language has a package called httptest which contains some utility methods. In this package, a method named NewUnstartedServer() returns us an unstarted test server object.

Let's take a look at the source code:

When we create a Server with the httptest.NewUnstartedServer() method, a Listener is created in the background that listens to a random port with the newLocalListener() internal method.

When we say start, it listens for connections from the listener in a goroutine with the s.goServe() method.
What comes to mind when we see Listener? As in Fasthttp, if we create an InMemoryListener and give it to our server, we can perform our test without listening a physical port.

In the above code, we run our server by actually listening to a physical port in the test named TestHttpServer. I put the time.Sleep to wait for the server to start. Using the srv.Listener.Addr().String() method, we can obtain the address (including the port randomly assigned by the OS) to which we need to request.

In the test called TestHttpServerInMemory, we performed the http server-client communication via an in-memory listener we created without listening to a physical port.
Here I used the fasthttputil.NewInMemoryListener method for convenience. If we want, we can do the same thing by creating a struct and implementing the Listener interface.
Let's take a quick look at it:

As we saw above, we can provide in-memory server-client communication by implementing the Listener interface ourselves and using the net.Pipe() method. The reason why I gave connection objects to our InMemListener struct over a channel is that the server calls the Accept() method in a loop and waits for a new connection. Thus, only if we give a connection to our Listener object, the Server side will receive a new connection.

So why isn't this in-memory server approach provided by default by Go?

In fact, there is a timely issue about it and there have been some people working on it, but the process has not been finalized yet.

net/http/httptest: optional faster test server #14200

bradfitz posted on Feb 02, 2016

Look into making httptest.Server have a way to use in-memory networking (i.e. net.Pipe) instead of localhost TCP. Might be faster/cheaper and not hit ephemeral port issues under load?

View on GitHub

Our Test Performance Improvements

Now let's see how we improved our own test performance using the methods I described above.
We developed our application with fasthttp and this application was acting as a basic reverse proxy. We also use some functionality provided by fasthttp for reverse proxy operation.

For our tests, we need a reverse proxy (our application) and a fake API to represent the server we are redirecting in the background.

We used httptest.NewUnstartedServer for this and we used fasthttptest.NewInmemoryListener.

Previously, we were actually listening on a port to run test servers in our tests, and this slowed down our testing process quite a bit. When I looked from our CI pipeline, our testing process was taking about 10 seconds.

As a result of our development, after using the in-memory listener approach, all our tests were completed in ~0.3 seconds.

In the image below, we can see all our tests and total uptime.

Conclusion

When we want to test an HTTP server, we can perform our test faster without listening to a physical PORT.
Our tests run much faster with this technique
We can simulate in-memory net operations using net.Pipe(), net.Listener, net.Conn
We can use the app.Test() method when testing our applications with fiber.
We can create in-memory Listener using fasthttputil.InMemoryListener()

I hope this was a fun and informative article for you. See you in the next post :)

Let's Connect:
Twitter
Github

You can support me on Github: Support mstrYoda on Github

My Contribution to a Popular Open-Source Package Caused a Panic in Golang Projects

Emre Savcı — Sat, 27 Aug 2022 13:11:21 +0000

Hello, today I will tell you how my development to a popular open-source project caused errors to people using the project around the world and what I learned from this process.

Content
— Swaggo Project
— Our Requirement and My Contribution
— The Problem
— Reason of The Problem
— Function Definition Withtout Body (go:linkname)
— The Solution
— Learned Lessons

SWAGGO Project

Creating swagger documents for your API projects developed in Go language is a little different from languages such as Java, C#. Generally, you can create it by adding Swagger directives in the comment lines of the code you write, and you don't have many alternatives for this.

The swaggo/swag project is one of the most popular and widely used solutions.

The basic process of this project is creating the swagger document by browsing the Go files and processing the comment lines in the definitions. It does this by creating an Abstract Syntax Tree (AST) and navigating it.

Our Requirements and My Contribution

While trying to create a swagger document for a project with my teammate, we realized that we could not make the definition we wanted using the swaggo/swag tool.

What we wanted to do was to define the request and response objects inside the handler function we created and ensure that they can only be accessed from within the function. But when we did this, the swaggo/swag parser package could not access the related request and response objects.

I wanted to make a contribution to the swaggo/swag library in order to overcome this problem.

First of all, I opened an issue to the library and asked if we needed such a thing and that I wanted to contribute by making this development.
In response, they said that it is not possible to parse private definitions and when they asked how I would do it, they were curious about my proposal and were waiting to see it.

I rolled up my sleeves. I did the development at the first opportunity and opened PR for the project.

As a result of the development I made, request/response objects became definable within the function. 💪

add functio scoped struct parse, tests #1283

mstrYoda posted on Aug 07, 2022

Describe the PR I added the functionality of parsing function scoped structs.

With this development we will be able to define our request response types in the same function scope as our handlers:

package main

// @Param request body main.Fun.request true "query params" 
// @Success 200 {object} main.Fun.response
// @Router /test [post]
func Fun()  {
    type request struct {
        Name string
    }
    
    type response struct {
        Name string
        Child string
    }
}

Relation issue https://github.com/swaggo/swag/issues/1274

Additional context All tests passed in parser_tests.go. Changes look backward compatible.

View on GitHub

The Problem

It's a Friday evening... The work is over, there are minutes... I'm looking forward to the weekend... And... I got a message from my teammate that one of our projects got an error 😞

I actually want to share the full conversation:

Translation:

Vural: bro I will tell you something
Me: yes bro
Vural: one of our services getting error while generating swagger do, and guess what on which part :D
Me: is that because of my development asdfasdf
Vural: yes, I guess asdfsadfs
Me: you can't be serious asdfasdf

We immediately went to the project's Github page and looked at the issues. And what we saw made me a little uneasy.

Many swaggo/swag package users were experiencing errors and they were all getting errors in the CI/CD pipelines or while manually generating swagger due to my development.

Segmentation fault running `swag init` after 1.8.5 release #1309

mackrorysd posted on Aug 26, 2022

Describe the bug Since 1.8.5 was released, when running swag init --parseDependency true on my code I encounter a segfault. After switching back to 1.8.4 (or other versions in the 1.8 line) the command completes successfully.

To Reproduce I'm unable to share the entire code this happens on. I will see if I can isolate this to a small code sample I can share.

Stack trace

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x85c8a8]

goroutine 1 [running]:
github.com/swaggo/swag.(*PackagesDefinitions).parseFunctionScopedTypesFromFile(0xc00000e7c8, 0xc00035b980, {0xc0003e15c0, 0x18}, 0xc000cff680)
    /home/sean/go/pkg/mod/github.com/swaggo/swag@v1.8.5/packages.go:168 +0xa8
github.com/swaggo/swag.(*PackagesDefinitions).ParseTypes(0xc00000e7c8)
    /home/sean/go/pkg/mod/github.com/swaggo/swag@v1.8.5/packages.go:110 +0xc9
github.com/swaggo/swag.(*Parser).ParseAPIMultiSearchDir(0xc0001e22a0, {0xc0001d7fb0?, 0x1?, 0x0?}, {0x9587e8?, 0x7?}, 0x64)
    /home/sean/go/pkg/mod/github.com/swaggo/swag@v1.8.5/parser.go:362 +0x3bf
github.com/swaggo/swag/gen.(*Gen).Build(0xc0001db980, 0xc0001e1790)
    /home/sean/go/pkg/mod/github.com/swaggo/swag@v1.8.5/gen/gen.go:177 +0x5c9
main.initAction(0xc0001af680?)
    /home/sean/go/pkg/mod/github.com/swaggo/swag@v1.8.5/cmd/swag/main.go:151 +0x757
github.com/urfave/cli/v2.(*Command).Run(0xc0001ad560, 0xc0001ec580)
    /home/sean/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/command.go:163 +0x5bb
github.com/urfave/cli/v2.(*App).RunContext(0xc000204000, {0xa25860?, 0xc0000240f0}, {0xc000020080, 0x4, 0x4})
    /home/sean/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:313 +0xb48
github.com/urfave/cli/v2.(*App).Run(...)
    /home/sean/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:224
main.main()
    /home/sean/go/pkg/mod/github.com/swaggo/swag@v1.8.5/cmd/swag/main.go:221 +0x55d

Your swag version 1.8.5 (1.8.4 and earlier worked)

Go version: 1.18

Desktop (please complete the following information):

OS: Pop! OS 22.04 (essentially Ubuntu)

View on GitHub

Frankly, I was surprised by this situation because I wrote unit tests for my development and the code was working correctly. In fact, the successful passing of the current unit tests proved that it did not break any previous development.

So what could be the problem? 🤔

Reason of The Problem

I'm attaching a screenshot from my development for swaggo/swag.

As we can see in the error messages, the error is received in the parseFunctionScopedTypesFromFile method. In fact, the exact line number of the error is 168.

Now let's take a closer look at this method I wrote.

In line 168, AST is visited and if there is FunctionDecleration among the definitions, what written inside the defined function is iterated with a for loop.

So if we are sure that a function definition exists, why do we get an error when trying to iterate this function definition with a for loop?
What causes the nil pointer error?

When we look at the functionDeclaration.Body variable, we see that it is a pointer type.

I didn't do a nil check here because I assumed that if a function was defined, that function would have a body definition.

But in reality it may not be so.👇

Function Definition Without Body

Does this term sound foreign to you? Can we define a function without writing a body?

Consider the code below, do you think this code is a valid definition in Go?

Will we get an error if we try to run it?

Yes, as you guessed, we cannot run or even compile this code because it is not a valid definition.

Can we run the code below?

Here, unlike the previous one, the answer will be yes, we can run it.

A little-known feature of Go is that we can link a function definition to the function definitions of other packages with the go:linkname directive.

Thus, we can define a function without defining a body.

And if you have noticed, this feature allowed me to use a private function defined under the Go runtime package.

External dependencies that we use in our projects can have such function definitions. And when we tried to create a swagger document with the swag --parseDependency command, functions with this definition caused a panic in my development because it also parsed dependencies.

The Solution

I was going to do a development to fix the problem right away, but the good thing about the open-source community is that another developer wrote the code that fixed the bug before me and created a PR.

fix: funcDeclaration body check #1310

rytsh posted on Aug 26, 2022

Describe the PR Nil pointer check in function scoped types.

Relation issue #1309

This is fixes problem on running swag build.

View on GitHub

Learned Lessons

With this problem, I learned some lessons for myself.

✅ Never bypass the nil pointer check hypothetically
✅ Unit tests may not always be enough. Do the end to end test
✅ We can write functions without body definition using go:linkname with Go, we can import functions from private packages
✅ Never make the package dependencies the latest version, specify the version you are sure is working

I hope it was a fun and useful content for you. Take care of yourself, I wish you bug-free codes 🙏

Let's Connect

Twitter
Github

You can support me on Github: Support mstrYoda on Github

Application Architecture for Microservices: Sidecar Pattern

Emre Savcı — Tue, 02 Aug 2022 08:05:00 +0000

What should our microservice application architectures be like?

Let's examine together how we use sidecar to solve cross-cutting concerns such as authorization, caching, configuration secret management, and observability.

Cross-Cutting Concerns

Let's start with explaining cross-cutting concerns.

Some of the requirements outside of the application's business code are needed in the different layers of an application.

For example logging, configuration, caching, authorization, observability…

Traditional Development

The traditional way is to code and use all this tooling within our development service. If we have different services written in the same language, we can share those codes between services and use them as packages/libraries between services.

So what is a bad scenario?

In an environment where services are written in more than one language, these requirements had to be reimplemented with each language.

Imagine that you (or your teams) are writing same functionality over and over with different languages...

Is there a solution in this case instead of writing the same code?

The first thing that comes to mind is to write separate API services for those requirements.

So is this a correct solution?

What problems can we encounter?

How to scale these shared services when the number of consumer services increase?

Moving the application code to another service that accessing over the network will cause the network latency(delay) to our application. It will slow down our application.

So, what would you do if you do not want to rewrite the same stories in different languages, or to face with network latencies?

Answer: We will use to Sidecar pattern.

Sidecar Pattern

So what is this Sidecar?

We can run multiple containers inside our pods running on Kubernetes. We can call the containers running beside our main application container as sidecars.

These additional features can develop in a single language and then we can "inject" it to our applications. We develop our sidecars with Go language and inject them into microservices.

This "inject" operation is performed by Kubernetes on the basis of a concept which name is Dynamic Admission Webhook. I will leave the source link at the end of the article for details.

When a pod scales up, the additional sidecars also go with it.

In addition, with the network interface created specifically for the pod shared across sidecars, network latency can be ignored because it is over localhost.

What else can be done with the sidecars?

Pod network configurations
Pod network requests/responses can be interrupted/manipulated
Service mesh can be made
Microservice runtime implementation can be done

For example, Istio Service Mesh can work as sidecar. The pod's entire network control is taken over by the incoming istio-proxy as a sidecar and can manipulate incoming and outgoing requests/responses.

You can write various network rules. When the whole network passes through the istio-proxy sidecar, there is metric data for all network processes and we can visualize microservice inter-service calls.

I want to complete my post here. We take a quick look into the sidecar approach with you. See you in the next posts.

Resources

https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/

https://betterprogramming.pub/understanding-kubernetes-multi-container-pod-patterns-577f74690aee

https://istio.io/

Let's Connect

Twitter
Github

You can support me on Github: Support mstrYoda on Github

Configure RBAC in Kubernetes Like a Boss

Emre Savcı — Wed, 27 Jul 2022 21:47:38 +0000

In this post I will explain how to configure RBAC in kubernetes. We will configure RBAC both with kubectl and yaml definitions.

What is RBAC

In kubernetes there are several authorization mechanism like RBAC, ABAC.

With RBAC we can add constraints to access kubernetes resources. For example we can give permission to listing pods for specific namespace to a ServiceAccount and prevent it to delete any resource. We can do it for a specific namespace or cluster wide as well.

Key points of RBAC

There are 3 important concept in RBAC.

Subject : Users, groups or service accounts.
Resources : Kubernetes API objects which we will operate on.
Verbs : The operations which we want to do with our resources.

Role & ClusterRole

Role and ClusterRole contains set of rules to access & modify kubernetes resources. Difference between them is Role works in particular namespace while ClusterRole is cluster wide which is obvious from it’s name.

If you need define permissions inside a namespace use Role, if you need that cluster wide use ClusterRole.

Creating Role via kubectl is simple like I showed you below:

kubectl create role my-custom-role --verb=list --resource=pods --namespace k8boss

We can also specify multiple verbs and resources as well:

kubectl create role my-custom-role --verb=list --verb=get --resource=pods --resource=services --namespace k8boss

If we want to make it via yaml:

kubectl apply -f role.yaml

If we did will not specify namespace in our role.yaml it will considered as default namespace.

ClusterRole is not much differs:

It is almost same with Role definition only without a namespace.

ServiceAccount

Service accounts are used by processes inside pods to interact with the Kubernetes API. If you write custom application to work with kubernetes you will probably need to create a custom ServiceAccount with some specialized roles assigned to it.

For example when you deploy kubernetes dashboard you can give additional roles to kubernetes-dashboard service account or create a custom ServiceAccount with a cluster-admin role bindings.

Creating a ServiceAccount is so simple with kubectl:

kubectl create serviceaccount custom-sa -n k8boss

Or yaml:

After creating ServiceAccount lets show it’s yaml content:

kubectl get sa custom-sa -n k8boss -o yaml

As you can see there is an additional field “secrets” which we did not specify while creating ServiceAccount. A secret has been created which related to our ServiceAccount by kubernetes.

Lets take a look our secret:

kubectl get secret custom-sa-token-jcq7h -n k8boss -o yaml

We can see a token key value here. We can use these secret to access kubernetes-dashboard. But we need to decode it because kubernetes stores secret values in base64 encoded format. Simply we can describe secret and take the token value:

kubectl describe secret custom-sa -n k8boss

RoleBinding & ClusterRoleBinding

Role bindings as indicates from names makes us bind roles to subjects (user or set of users or service accounts). Again the only difference is in usage, we use RoleBinding to bind Roles while ClusterRoleBinding to bind ClusterRoles. But as an addition we can use RoleBinding to bind a ClusterRole to a Role within a namespace.

Before creating role binding we can ask our service account’s capabilities with:

kubectl auth can-i list pods --as=system:serviceaccount:k8boss:custom-sa -n k8boss

You will see “no” as output of that command.

Lets create a role binding with kubectl:

kubectl create rolebinding my-custom-role-binding --role=my-custom-role --serviceaccount=k8boss:custom-sa --namespace k8boss

Equivalent yaml file:

ClusterRoleBinding is not much differs:

kubectl create clusterrolebinding my-custom-clusterrole-binding --clusterrole=my-custom-cluster-role --serviceaccount=k8boss:custom-sa

Now again lets ask our capabilities:

kubectl auth can-i list pods --as=system:serviceaccount:k8boss:custom-sa -n k8boss

Now you can see “yes” as an answer.

Aggregated ClusterRoles

Last thing we will mention is Aggregated ClusterRoles. Sometimes we might want to combine several cluster role into one.

To do this we can use aggregationRule field of ClusterRole. We can select roles using label selectors. For example lets assume we have two ClusterRole one for listing pods one for listing services. So we need another ClusterRole to list pods and services. We can now use aggregationRule to combine our two ClusterRoles.

There are other useful articles about RBAC I suggest you to read those too:

Using RBAC Authorization | KubernetesUsing RBAC Authorization | Kubernetes

Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within your organization. RBAC authorization uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing you to dynamically configure policies through the Kubernetes API. To enable RBAC, start the API server with the --authorization-mode flag set to a comma-separated list that includes RBAC; for example: kube-apiserver --authorization-mode=Example,RBAC --other-options --more-options API objects The RBAC API declares four kinds of Kubernetes object: Role, ClusterRole, RoleBinding and ClusterRoleBinding.

kubernetes.io

Demystifying RBAC in Kubernetes | CNCF

Today’s post is written by Javier Salmeron, Engineer at Bitnami Many experienced Kubernetes users may remember the Kubernetes 1.6 release, where the Role-Based Access Control (RBAC) authorizer was…

cncf.io

Configuring permissions in Kubernetes with RBAC | by Containerum | Containerum | Medium

Containerum ・ Aug 1, 2018 ・
Medium

Thank you for reading so far. See you on the next article.

You can follow me on:

Github

Twitter

How To Become an Open Source Contributor

Emre Savcı — Tue, 26 Jul 2022 11:17:00 +0000

As a software developer, I feel happy when I contribute to the community. It is an opportunity to improve your development skills and learn new things. It is both an educative and entertaining process. There are few things that satisfy me as much as an accepted pull request :)

Becoming an open source contributor is a dream for some developers. If you are a new developer or never made a contribution before and want to become a contributor, in this post, I have several suggestions for you.

Where To Start

Well, let’s assume you found a project on Github and you want to contribute to that project. Which steps to follow to become a contributor? What might be helpful to the contribution process? I will list those baby steps that I followed and made the contribution process easy for me.

1- Use that project in production as much as possible

This might be the most important part of becoming a contributor. Using a tool in production or in daily life helps you to understand it better. While using the tool, you will probably face errors that are actually bugs that need to be fixed.

At this point, you can start contributing by creating an issue that describes the steps to reproduce that error.

After that, you can get your hands dirty to get dive into the source code to investigate the root cause of that bug.

When you find the problem and fix it, you can create a pull request.

As an example, about 2.5 years ago, when I was writing C#, we used to use RestSharp as http client like most of the other developers. At that time, we had trouble comparing two almost identical DateTime; we investigated it and found that RestSharp had missed a DateTime format while deserializing json string. And I decided to fix that issue by opening a pull request. After that, my pr merged, and I became a contributor to RestSharp.

2- Take a look at the source code

When I am curious about a project, I will clone it and take a look at the source code of it. It is not always a good starting point to contribute but it might help you to find some code to refactor.

As an example, when I started my career at Trendyol, I forked a project which was an internal framework used for legacy projects. When I looked at the source code, I found a piece of code to refactor.

Another example of looking at source code is while we were using Consul in our project, I looked at the source code, and I saw an array declaration without specified length (declaring length while initialization is a performance improvement for arrays in Golang ). So I made one line contribution by specifying the length of that array. Even that was satisfying for me :)

There is one more thing to do in the source code: “Measuring test coverage.” Writing unit tests to increase test coverage is another way to contribute to the project. For example, I am preparing a pr to Kubernetes that contains unit tests which increase test coverage.

3- Read documentations and apply examples

Reading documentation helps you to understand how to use the tool. There are two things related to reading docs. One is that you might need information about a feature and you can not find it in docs. Here you can contribute by writing documentation about that part. The second one is that sometimes documentation might not be up to date or have typos. When you are faced with that, you can contribute by fixing docs.

I remember I was following an example on Istio documentation, applied it, and got an error. I realized that the example has a typo on yaml configuration. I created a pr and fixed that documentation to save other developers time.

4- Use the client of that project (if exists)

Diving right into the source code of an open source project might be hard, especially if the project has a huge source code. At this point, it might be helpful to start using the client code of that project. This also helps to understand the internal structures of that project.

Let’s take Kubernetes as an example, it has quite a huge source code, and it is not straightforward to dive into source code as a stranger. But, you can use Kubernetes client and make some example projects which interact with Kubernetes. You will see that it helps you to understand the internals of Kubernetes better.

5- Search for issues

Searching for issues to resolve is another alternative to start contributing, but it might be hard for a beginner. You can always start searching for issues labeled as GoodFirstIssue.

When I wrote Java, we used to use OpenFeign within our Spring Boot projects. I decided to search for issues to find if there is an easy one to get involved in the project. And I found it!

I start to resolve that issue and opened a pr for it. It was accepted by authors, and I got my name in contributors.

There are sites that help you to find projects to contribute to. You can filter projects by tags or languages.

codetriage.com

Up For Grabs

Want to contribute to open source, but not sure where to start?

up-for-grabs.net

You can also use Github directly to find projects by filtering:

github.com

6- Create an open source project

Creating an open source project might sound very hard at first glance. But in time you will realize that open sourcing your codes, even basic automated tasks, might help others too.

So following this thought, I created a repository called docker-shell, which does auto-completion and suggestions for docker commands. It grew further than I expected. It helps developers who use docker in daily life.

You can contribute to the docker-shell project as well:

Trendyol / docker-shell

A simple interactive prompt for docker

docker-shell

A simple interactive prompt for Docker. Inspired from kube-prompt uses go-prompt.

Features

Suggest docker commands
List container ids&names after docker exec/start/stop commands
Suggest command parameters based on typed command
List images from docker hub after docker pull command v1.2.0
Suggest port mappings after docker run command v1.3.0
Suggest available images after docker run command v1.3.0

Installation

Homebrew

You can install by using homebrew:

brew tap trendyol/trendyol-tap

brew install docker-shell

Build From Source Code

You can build the command from source code by following the steps below:

git clone git@github.com:Trendyol/docker-shell.git

cd docker-shell

sudo go build -o /usr/local/bin/docker-shell .

docker-shell

How To Use

After installation, you can type docker-shell and run the interactive shell.

Image suggestion from docker hub:

Port mapping suggestion:

How To Contribute

Contributions are welcome and will be fully credited…

View on GitHub

Another open source project created by me is "kubernetes-kitap" project, which is a book written in Turkish for Kubernetes to help Turkish developers to understand Kubernetes:

mstrYoda / kubernetes-kitap

Hoş geldiniz

Open Source severler olarak bir araya gelip, Kubernetes için Türkçe kaynak olması açısından bu kitabı hazırladık ve siz değerli okuyuculara sunduk.

Kitabın adresi: https://mstryoda.github.io/kubernetes-kitap/#/

Konu Başlıkları

Giriş
Kubernetes Node Elemanları
- Control Plane Elemanları
- Worker Node Elemanları
  - kubelet
  - kube-proxy
Development Ortamı Kurulumu
Kubectl Kullanımı
- kubeconfig
- Kubectl İşlemleri
Controllerlar
Pod Genel Bakış
- Pod Yaşam Döngüsü
- Pod Preset
- Static Pod
- Ephemeral Container
Podların Çalışacağı Nodeları Belirleme - Scheduling
Uygulama Kaynaklarının Konfigürasyonu
- ResourceQuata Objesi
- LimitRange Objesi
Health Check İşlemleri
HorizontalPodAutoscaler ile Scaling
- Ram & Cpu Bazlı Uygulama Scale Etme
- …

View on GitHub

Before finishing the post, I want to say that contributing to other projects has its own benefits. The first and obvious one is that it improves your development skills. Your code is reviewed by other developers, and this gives you a new perspective. Also, you are making new networks and friendships with other developers.

Bonus: Here is another site that might be helpful to start open source:

Open Source Guides | Learn how to launch and grow your project.

Learn how to launch and grow your project.

opensource.guide

You can follow me on:

mstrYoda (Emre Savcı) · GitHub

Tech. Lead @Trendyol | K8S <3 | Loves Golang | Contributions: [docker-shell, awesome-istio, awesome-dapr, @gofiber, @swaggo, kustomize, gocb] - mstrYoda

github.com

Twitter

Thank you for reading so far. Take care of yourself until next post.

How to Become Cross Functional Polyglot Developer

Emre Savcı — Mon, 25 Jul 2022 10:03:00 +0000

How to become a production ready developer?

In these days sticking to a single technology / programming language / tool is not enough for a developer. There are so many things to consider, especially in the microservice’s world.

This might make developers freak.

As a software developer in my daily job I am handling different stuffs like designin architecture, writing code, testing it, writing CI/CD pipelines, creating alerts according to logs or events, monitoring the app etc..

In this article I am aiming to enlighten the new developers or the ones who does not involved in multi-tech stack.

What is the requirements of developing applications and deploying to production?

What are some concepts to follow?

Unit Tests (Coding)

We need to develop applications in a testable matter. All the business requirements and features must be testable. With unit testing, we prove that our code works respecting to what business said.

2. Domain Driven Design (Architecture)

To talking the same language in business persons we need to develop a language. Also to build extensible, testable, understandable application we need some abstractions&rules to follow. In DDD we can provide a clean solution to complex business domains.

I recommend the red book of Vaughn Vernon: “Implementing Domain Driven Design”.

3. Clean Code Practices (Coding)

We need to follow clean code practices to write clean, understandable, readable code.

4. Integration&Automation Tests (Testing)

We need to prove that our application can work without trouble with 3rd party applications. Also we want to test our application from outside point of view. And we want to avoid doing that manually every time for any change occurs. So we write integration&automation tests to automate that process.

Consumer Driven Contract Tests (Testing)

How do we sure about our consumers will not affected when we made a change to an endpoint in our rest api?

What if we change a field’s name which is used by a consumer by accidentally?

CDC tests to rescue. We and our consumers writes CDC tests and run those tests in pipeline to prevent deploying any breaking change.

Take a look to project pact: https://docs.pact.io

6. Hexagonal Architecture & Ports Adapters & CQRS (Architecture)

We need to provide extensibility and a well abstraction to our project. Abstracting application/domain/infrastructure/api levels of our project is much more important compared to what framework we used. Here is hexagonal architecture and port&adapters comes into view.

Also we might want to scale our application’s read and write parts separately. Or we want to divide/abstract that responsibility in the code. Here is what CQRS tells us.

Take a look for more information: https://herbertograca.com/2017/11/16/explicit-architecture-01-ddd-hexagonal-onion-clean-cqrs-how-i-put-it-all-together/

7. Containers & Docker (Devops)

We can dive into containers from a higher level with Docker. Mostly used de-facto standart in container tech.

Also there are some other tools to build containers like buildah & podman.

8. CI/CD Pipelines (Devops)

We need to automate our test / build / deploy steps of our applications.

With using CI/CD tools (Jenkins, Gitlab etc.) we can achieve this.

9. Kubernetes (Devops)

Well K8S is the de-facto standart for container orchestration. We can run/scale/discover our services with kubernetes.

As a developer we need to be familiar with it to understand the environment which our application works on.

10. Logging (Monitoring)

Logs are the one of the way to help us to understand our application’s behavior. With a good logging system we can catch unexpected behaviors/errors and take actions to them.

There are many different logging techs like ELK stack.

For example, we are trying to follow 12 factor app strategy. Our applications does not aware of the underlying tech about logging. All our microservices print their logs to stdout and we collect it with fluentbit/fluentd in kubernetes. With this way we are abstracting the logging technology from applications. We can switch the log storage without affecting applications.

11. Monitoring

Softwares are alive systems. They need to be watched all the time. To determine any anomaly (like high resource usage, total requests, response times) and optimize those we need to keep track it.

There are tools like NewRelic, Jaeger, Elastic APM which helps us to monitor applications.

12. IaC (Devops)

Infrastructure as code help us to automate any machine provision & configuration process. Tools like terraform, ansible, chef, puppet, vagrant, docker help us to achieve IaC. We can write our environment in a terraform/ansible as code and we automatically run those in a pipeline to bootstrap the environment.

13. Message Brokers (Architecture)

We might want to design our service communications in an async way. Mostly we do it as a performance optimization. For example, sending information mail end of a process can be async. So we can separate those services and provide communication between separated services with a message broker.

There are some popular tools like RabbitMQ, ActiveMQ, Kafka, Aws Sqs etc.

14. Databases & Cache (Storage)

We use storage systems. So we need to know how to use them. Underlying storage systems might affect our data structure. Wrongly designed schemas affects performance of the application.

How to store and retrieve data? Is there a faster way to access data? Do I need to do db calls all the time? We need to be able to answer these questions.

When we using a database (sql/nosql), we need to take a look how to index data, how to apply right schema, how to retrieve it etc.

We also might need to take the overhead from db and use a cache solution. We can cache frequently accessed data.

I am stopping here and wishing you to enjoy while reading.

This list contains the first things that comes into my mind. There are so many other parts to consider while developing an application. Like eventually consistency, event sourcing, circuit breaker, retries, distributed locks, saga, state machines etc..

If you have any suggestion/addition to list write it on comments.

You can follow me on:

https://twitter.com/mstrYoda_

https://github.com/mstrYoda

Extending Envoy Proxy - WASM Filter with Golang

Emre Savcı — Sun, 24 Jul 2022 17:50:00 +0000

Envoy is an open source service proxy especially designed for cloud native applications. It has a wide variety of features like connection pooling, retry mechanism, TLS management, compression, health checking, fault injection, rate limiting, authorization etc. Those are achieved with built-in http filters. And today I will talk about a special filter which name is WASM Filter.

This article is not meant for explaining what is WASM, so I am skipping explaining WASM instead I will add resources for that at the end of the article.

Why We Use WASM Filter

In Trendyol Tech. We are using Istio as a service mesh. And our team’s (DevX) responsibility is improving developer experience by developing applications which meets common requirements of microservice’s like caching, authorization, rate limiting, cross cluster service discovery etc.

Since we already use Istio why not take advantages of the power of Envoy Proxy’s extensibility.

Our use case is acquiring JWT token for microservice’s which identifies that microservice application. When we want to avoid every team to write the same code in different languages, we can create a WASM Filter and inject it into Envoy Proxies.

Advantages of WASM Filters:

It allows to write code in any language which has WASM support
Dynamically load code to Envoy
WASM code is isolated from Envoy so crashes in WASM not affect Envoy

In Envoy Proxy there are worker threads that handles incoming requests. Every worker thread has its own WASM VM. So if you write time based operational code it works separately for every thread.

In Envoy Proxy every worker thread isolated from each other and has one or multiple WASM VM. There is also a concept called WASM Service for inter thread communications and data sharing (we are not cover this).

Writing WASM in Go

We are going to use tetratelabs/proxy-wasm-go-sdk to write WASM in Go. We also need TinyGo to build our Go code as WASM.

Our use case is very simple so we write a code which sends request to JWT Api for every 15 second. It extracts authorization header and sets it’s value to a global variable and puts that value to response header of every incoming request. We also set “hello from wasm” value to another header called “x-wasm-filter“.

In the OnTick function we are making http call to a service known by Envoy as cluster.

Let’s build our go code as WASM:



tinygo build -o optimized.wasm -scheduler=none -target=wasi ./main.go

Now we need to configure Envoy Proxy to use WASM Filter for incoming requests. We will define a routing rule and a WASM filter for our WASM code, also we define a cluster which represents our service.

I put all of the files into same directory. Now let’s run Envoy Proxy in Docker:



docker run -it — rm -v "$PWD"/envoy.yaml:/etc/envoy/envoy.yaml -v "$PWD"/optimized.wasm:/etc/envoy/optimized.wasm -p 9901:9901 -p 10000:10000 envoyproxy/envoy:v1.17.0

As we can see from logs our WASM Filter started to work and sending request to JWT Api in every 15 secon

Now let’s send a request to Envoy Proxy. We configure Envoy to listen incoming requests from 1000 port and we start our container with port mapping. So we can send request to localhost:10000:

In the response headers we can see “x-wasm-filter: hello from wasm” and “x-auth” values.

Thank you for reading so far. I hope it will give you a perspective about how and why use WASM in Envoy Proxy.

You can see full example in github:

mstrYoda / envoy-proxy-wasm-filter-golang

A WASM Filter for Envoy Proxy written in Golang

envoy-proxy-wasm-filter-golang

A WASM Filter for Envoy Proxy written in Golang

Build

tinygo build -o optimized.wasm -scheduler=none -target=wasi ./main.go

Run Envoy Proxy in Docker with WASM Filter

docker run -it --rm -v "$PWD"/envoy.yaml:/etc/envoy/envoy.yaml -v "$PWD"/optimized.wasm:/etc/envoy/optimized.wasm -p 9901:9901 -p 10000:10000 envoyproxy/envoy:v1.17.0

View on GitHub

Resources

webassembly.org

WebAssembly | MDN

WebAssembly is a type of code that can be run in modern web browsers — it is a low-level assembly-like language with a compact binary format that runs with near-native performance and provides languages such as C/C++, C# and Rust with a compilation target so that they can run on the web. It is also designed to run alongside JavaScript, allowing both to work together.

developer.mozilla.org

Wasm Modules and Envoy Extensibility Explained, Part 1 - The New Stack

If you've ever wondered what WebAssembly (Wasm) is and how it fits into the service mesh ecosystem, this is the article you want to read.

thenewstack.io

tetratelabs / proxy-wasm-go-sdk

WebAssembly for Proxies (Go SDK)

Warning

We are no longer recommending this SDK or Wasm in general for anyone due to the fundamental memory issue of TinyGo (See the detailed explanation by a long-time community member) as well as the project state of Proxy-Wasm in general If you are not in a position where you have to run untrusted binaries (like for example, you run Envoy proxies while your client gives you the binaries to run), we recommend using other extension mechanism such as Lua or External Processing which should be comparable or better or worse depending on the use case.

If you are already using this SDK, but still want to continue using Wasm for some reason instead of Lua or External Processing we strongly recommend migrating to the Rust or C++ SDK due to the memory issue of TinyGo described in the link above.

We keep this repository open and not archived…

View on GitHub

You can follow me on:
Twitter
Linkedin
Github

Software Interview Processes Recommendations

Emre Savcı — Thu, 21 Jul 2022 06:56:00 +0000

Greetings, in this article, I will share with you some of my views that occurred over a period of time so that both parties can have a good experience in software interviews.

First of all, regardless of the outcome of the interview, I think that the interview should contribute to both parties. The candidate should be able to leave the interview by learning new things. These things can be a technical topic, process, product, or approach.

The interviewer, on the other hand, should be able to learn to analyze people with different characters, measure different levels of knowledge, direct the subject, and create a healthy communication environment.

In this process, there are important responsibilities that fall on the parties. Now let’s go over the topics that we will examine these responsibilities.

Preliminary

The first issue I would like to mention is that the parties should have information about each other before the interview.

Having information about the company, team, and project you are going to interview contains important clues to prepare for an interview, you can predict what you will encounter. It allows you to prepare in advance some questions that you can ask during the interview.

Having information about the candidate allows the interviewers to have an interview that is suitable for the level and experience of the candidate, to skip the boring interview questions, and measure the hands-on experience of the candidate directly through the tasks he took in his previous projects and the work he has done.

This preliminary preparation also saves time to be lost by explaining the resume or the team during the interview.

Ability to Problem Solving

Rather than trying to measure the mere knowledge of the candidate you are interviewing, asking questions that will reveal the candidate’s problem-solving ability will help you see the abilities of the person you are interviewing. Specific information, a piece of code, and the behavior of a framework may not come to mind at that moment, it may be forgotten. This does not mean that the person in front of you is ignorant.

One thing that the interviewees should not forget is that they must demonstrate their problem-solving skills in the interview.

Not Personalizing Processes

What is meant by personalizing the process? What processes can be personalized in an interview?

For the candidates who are interviewed, you should consider that the questions posed to you are not so that you cannot answer, but to measure how much you know about a subject. These opposing thoughts will make you tense in the interview and feel cornered.

Interviewers, on the other hand, should not have an ego-satisfaction motive on candidates. Without focusing on understanding what the candidate knows, interviewers should not try to get pre-prepared answers specific to the subject that already knows. Be aware that you are not in a race for supremacy. There is a very important ethical value here, at that moment your company gives you the responsibility to take the deserved person to the position they deserve. You must not abuse this responsibility.

Another thing that should not be personalized, regardless of the interview process, is that you should describe your problem (for example, a person with whom you have problems in your working life) without personalizing it, through a cause-effect relationship. So how can this be done? Let’s say that when you are asked about something that you are uncomfortable with, instead of saying, “I am uncomfortable with person A, I do not work with it, he is so and so”, instead of saying “exhibiting x behaviors causes y results, these negative results disturb the motivation/peace of the employees”, it shows that you do not personalize the issue, on the contrary, you are in favor of improving processes.

Being realistic

Yes, another important issue is to be realistic. Candidates who are interviewed should first determine their expectations from the company, their working environment, and the team they will work with (salary, fringe benefits, technical responsibility, culture) and should also be able to transparently state that they can add them to the team. This is an important situation in terms of maintaining the motivation of both yourself and the other party. He should also be aware that pretending to do things he hasn’t already done will not make a good impression as it will weaken the current team dynamics.

The interviewer, on the other hand, should not raise the candidate’s expectations in such a way that cannot meet them. The image above sums up what I’m about to say. A developer who joins the team thinking that he will deal with different technical challenges every day will lose his motivation and enthusiasm quite quickly when he spends his day only with crud transactions.

Pieces of Advice

Yes, we’ve gone over the general topics so far. Now, I would like to briefly share a few recommendations with you.

Developers: Before starting to solve the problems posed to you, ask questions to find out the extent of the problem. Set limits. How many users will use the system Can the process be managed asynchronously HA, what are the consistency priorities…

Interviewers: You should not leave the candidate alone in the interview. The moment you feel that the interviewee is lost, you must take on the role of guiding.

Developers: Problems do not have a single correct solution. There are different approaches. Don’t be fooled into thinking that the other party is waiting for a single clear answer from you. Interpret the problem.

Interviewers: You cannot measure any knowledge or ability with stereotyped questions. Do not use rote interview questions and techniques.

Developers: When you come across a question you don’t know, don’t just say you don’t know and throw it away. In the worst case, try to find direction by asking questions to find out what you don’t know so that you can come up with a solution. Otherwise, your ability to produce solutions cannot be measured.

Interviewers: The candidates are not the prisoners you question. They do not have to memorize a topic, tool, or code that you have learned specifically at your workplace. Being aware of this, measure the interviewee’s approach specific to a subject. Do not try to measure mere knowledge.

Developers: Practice topics like design patterns, microservices, data structures, DDD, unit testing, architecture, caching, scaling, cap, message brokers, CI/CD. These topics will come up in most interviews.

Interviewers: Inform the candidate on a question that the candidate does not know or cannot answer correctly. Both you and your company will gain value in the eyes of a candidate who leaves the interview by learning something. Thus, even if the interview is negative, there will be no loss for both parties.

Developers: The team you will work with, the work you will do, the technologies used, the projects developed, the way of doing business, company culture, incident management… Ask questions about anything that comes to your mind. If you do not work in a happy and productive environment, you cannot be beneficial to yourself or the company.

Interviewers: Asking questions about the problems you are dealing with to get the right person to your team allows you to recruit people with the same perspective and interests. If the person you ask utopian questions to writes “just crud”, they will be disappointed.

Developers: On something actually you don’t know anything about, do not pretend like you have knowledge about it. It is very easily understood and certainly does not leave a good impression.

Interviewers: Choose your co-workers from people who fit the current team culture. Someone will not be included in the team just because they have technical knowledge, nor will they be eliminated because you find their technical knowledge insufficient. Technical knowledge can be gained later.

Developers: Do not disparage your old company, or your colleagues. If you’ve had a problem, talk about the “problematic behavior” without personalizing it.

Interviewers: Get up-to-date information about the candidate’s past projects and interests. Don’t ask funny questions like what is SOLID, what is Factory Pattern, to an experienced candidate who has worked on good projects.

Developers: Questions will be asked in order to measure how experienced you are on a subject. Don’t feel cornered or questioned. Remember that you can ask counter-questions as part of the process and you don’t have to know everything.

Thank you for reading so far. See you in another article.

Want to Connect?

Twitter
Linkedin
Github

Software Developers, You Needn’t Be Perfect

Emre Savcı — Thu, 21 Jul 2022 06:48:00 +0000

Lately, I have observed that developers, especially the ones who’re starting out — are under the influence of different factors and get lost in the concept of “perfectionism”.

We, as developers, want the best for ourselves and what we produce. As we build from scratch it’s natural to be committed to what we produce. Hoping it’s flawless. This is one of the factors that force us to be “perfect”.

So, is being perfect a realistic goal?

Do we have to be perfect? Or are we unintentionally forced to be perfect?

You don’t have to keep up with everything

We know that the software community has a multidisciplinary structure. The emergence of a working code is the result of the collaboration of scientists, mathematicians, and programmers.

You will meet people with different knowledge and experience in such a field. In such a situation, you should not feel inadequate.

Backend, frontend, DevOps, security, AI/ML, design, infrastructure, cloud…

Go, Java, C#, Python, Kotlin, Scala, Haskell, f#, Elixir, C/C++, JavaScript.

Different areas of expertise, different tools, different approaches. If you try to learn all of them, at the end of the day you will remain a tired and exhausted “coder” who has not learned any of them completely.

You don’t have to know everything. No one expects such a thing from you. Which is not an attainable goal in finite human life anyway. You should not expect yourself to know everything.

“But people on the Internet know many things, they are very good and successful in everything?”

If you say so, let’s move on to the next topic.

Social media fallacy

The world is like a stage. Everyone plays their part in this scene, and when their role is over, they leave this scene forever.

Shakespeare

You may be deceived by what you see on Twitter and LinkedIn and get the impression that people are always highly motivated, hard-working, never making mistakes, and know everything. But the truth of the matter is not at all what it seems. Most people do not share the facts, they only share the parts they want you to see. This, in general, reveals the misconception that those persons are “constantly successful and good at everything”.

People share their best moments on social media because they don’t want their failures to be seen. However, experience is gained as a result of failures and mistakes. It is hidden that this is a natural process and the perception is created that they are flawless and perfect. This is an illusion. Do not lose your motivation and self-confidence when you make a mistake or fail to achieve the desired result after a few tries. You make as many mistakes as anyone else. Treat yourself fairly.

Having lots of basic-level posts on different topics does not necessarily make a person very knowledgeable.

As Morpheus said, “There is a difference between knowing the way and walking on it”. Since even specializing in a single subject is something that requires many years of effort, you should not get the thought of “how much people know, I miss everything” after every post you see.

In the same way, do not view people you see on social media as those who live everyday with a plan. Who do not miss anything. Who continue to write code with one hand while rubbing an avocado on toasted bread in the morning. Who never make mistakes and know the best of everything.

It wouldn’t hurt to play games for a few hours, spend time with your friends, or even take time for yourself to do nothing sometimes. On the contrary, sometimes you need to do these things to clear your mind and relax.

So, don’t get caught up in thoughts like “I wish I had written code instead of playing games”, “I wish I had read a few more articles instead of watching movies”, “I should have looked at projects instead of going out with my friends”. There’s nothing wrong with living your life. The activities you will do to distract yourself a little will not hinder your self-development.

Another concern is the effort people put to be justified — simply by hiding behind the masses. Even when something you think could be wrong, your sense of truth can suffer when too many people take the opposite view.

Popularity, combined with fanaticism leads to the veiling of reality and the perception that what is being advocated is true. However, every truth can be defended, but not all defended truths.

“There is no doubt that our age… prefers the description to the object, the copy to the original, the representational to the reality, the outer appearance to the essence… The only thing sacred for our age is illusion, the only thing that is not sacred is truth.”

Perfection has no limits

Chasing the impossible is insane.

Marcus Aurelius

What is perfect? To be faultless? To be the best?

What is the definition of good? How good is the best?

How to gain experience if no mistakes are made?

Does knowing everything perfection? Can everything be known?

Go deep into something? How deep do you go?

Knowing all the libraries of a programming language by heart?

How many of these do we need? Is it necessary to be perfect if what is needed is enough?

The answers to the above questions are things that can change from person to person and whose boundaries cannot be determined. Trying to reach something immaterial that cannot be defined in the same way by everyone will be nothing more than a futile effort.

In the past, perfection for me was knowing at the source code level what the code I wrote and the product I used was doing in the background in each process. But this is not an achievable and sustainable goal.

Knowing the language, orm tool, http package, database, operating system, kernel/CPU calls, and network operations at the source code level, it is impossible to try to learn more than one in this list. This is not perfection, it is madness. There is no need for such a thing.

You too may have different definitions of perfection. It doesn’t feel right to generalize such a variable, ambiguous and abstract concept for everyone and to put it in front of people as if it is a goal that needs to be reached.

Most of the time, it will be “sufficient” to know the tool you use enough to reveal something, solve your problem, to be able to build on it by researching and learning.

You need a lot to be perfect, and the more your needs, the more restricted your freedom. Freedom is not needed — to be enough, to be self-sufficient…

So does happiness come from having something or not wanting anything to have?

Freedom is not the right or ability to do whatever you want.

Epictetus

Perfect is the enemy of the good

“We are what we constantly do, excellence is a habit, not an act.”

Aristotle

How many projects do you have that you strive to be perfect but can’t finish?

How many codes do you have that you are afraid to share because you think it is not perfect?

How many attempts have you failed to do because you couldn’t find the perfect idea?

How many blog posts are there that you didn’t write because there wasn’t great content?

Do you realize how much trying to be perfect causes us to miss out? In fact, what we need is to come up with a working project, write and share the right code, and have the courage to try an idea. To try, to do, to complete. These are far more important concepts than perfection.

Don’t let others define how you should be. Keep in mind that striving to be perfect deprives you of how many things.

In seeking the perfect, we lose the good enough.

Not to compete with anyone

“The wise man does not compete with anyone, so no one can win against him.”

Lao Tzu

Competing with others is another occupation that will consume you. Comparing yourself to others every day and trying to be like them or better will wear you out both mentally and physically.

The only thing you need to compare yourself is how you were the day before. Trying to ensure your development by focusing entirely on yourself will eliminate the distraction and demotivating situations that will arise from external factors.

Your progress is up to you, not perfection

There is only one path to happiness; and not worrying about events that you can do nothing about.

Epictetus

We want to improve ourselves to feel better, achieve our goals, and be happy people.

You don’t have to try to be perfect while doing this.

Setting achievable goals and working towards those goals will bring you to the level you want. Development is achieved not by trying to be perfect, but by working hard in a disciplined way.

Setting achievable goals during this process will help you stay motivated. If you constantly set big goals and cannot reach them, you start to feel that you are running out over time and that your old enthusiasm is no longer there.

What you really need is not to be perfect, but to have discipline and habit.

As long as you keep yourself in shape by regularly writing code, reading articles, and doing research, the development will be inevitable.

It doesn’t matter how slow you go as long as you don’t stop.

Andy Warhol

Now, without waiting any longer, share your first code, realize your first idea, write your first article, and complete your first project.

It doesn’t have to be perfect. Being yours makes it special.

Want to Connect?

Twitter
Linkedin
Github

DEV Community: Emre Savcı

Troubleshooting Kubernetes Application - Istio Service Mesh Upstream Connect Error

Service Mesh in Trendyol 🚀

Error Case ❌

Determining Troubleshooting Paths 📝

Reproducing the Error 🚨

Sniffing the Pod Network 🕵️

Inspecting Code 🔍

Problem and Solution ✅

Conclusion 📚

Golang -What is Broken Pipe Error? Tcp Http Connections and Pools

Connection Pools are Everywhere

HTTP Client - Server Communication with Golang

HTTP Keep Alive

Idle Connection Settings

Creating a TCP Connection in Golang

Broken Pipe Error

Lets Create Our Own TCP Connection Pool

No-Code Tools For Your Next SaaS & Side-Project

AirTable

BaseRow

Notion

AwesomeTable

ParseHub

GlideApps

WebFlow

Pory

Golang Blazing Fast Unit Tests - Fiber/fasthttp/http Internals and Optimizing HTTP Server Tests

Contents

Go HTTP Internals

Testing in Different Packages

Fiber

fasthttp

Go HTTP

Our Test Performance Improvement

Conclusion

Go HTTP Internals

http.ListenAndServe

net.Listen & net.Conn

http.Client Dial Function

net.Pipe

Testing in Different Packages

gofiber / fiber

⚡️ Express inspired web framework written in Go

⚠️ Attention

⚙️ Installation

valyala / fasthttp

Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http

fasthttp

fasthttp might not be for you!

General info and links

HTTP server performance comparison with net/http

Go HTTP

Our Test Performance Improvements

Conclusion

My Contribution to a Popular Open-Source Package Caused a Panic in Golang Projects

SWAGGO Project

Our Requirements and My Contribution

The Problem

Reason of The Problem

Function Definition Without Body

The Solution

Learned Lessons

Application Architecture for Microservices: Sidecar Pattern

Cross-Cutting Concerns

Traditional Development

Sidecar Pattern

Configure RBAC in Kubernetes Like a Boss

What is RBAC

Key points of RBAC

Role & ClusterRole

ServiceAccount

RoleBinding & ClusterRoleBinding

Aggregated ClusterRoles

Configuring permissions in Kubernetes with RBAC | by Containerum | Containerum | Medium

Containerum ・ Aug 1, 2018 ・ Medium

How To Become an Open Source Contributor

Where To Start

Trendyol / docker-shell

A simple interactive prompt for docker

Containerum ・ Aug 1, 2018 ・
Medium