DEV Community: CodeFuture

A simple configmanager program for assigning values to variables from a custom config file | Golang

CodeFuture — Mon, 29 Mar 2021 15:10:35 +0000

This program configmanager reads a configuration file (.conf) and assigns values to a struct defined by the user. This comes in handy when you want to assign some variables in golang program from a custom configuration file. The process is simple: import configmanager, create your struct with **exported* fields, create a *.conf file or a default.conf file in PWD, define the variables in the config file, pass the struct reference to AssignConfiguration method in configmanager; you are good to go.

Configuration File Format

Comments shall start with #

  # this is a comment
  Val = 5 # this is a comment

All variables in the configuration file must start with Caps Letter
Variable must be defined with a field name and a value separated with =
If a variable is defined multiple times in a single configuration file, the bottom-most value of the corresponding variable is assigned

Struct Definition

Each field of the user-defined struct shall be exported (First letter must be in caps), so that the same can be accessed by configmanager program to assign values to struct fields from config file.

Following types are supported for struct fields:

int64

float64

string

bool

type config struct {
    Val1 int64
    Val2 string
    Val3 bool
    Val4 float64
}

configmanager usage

let's first create a file main.go . Import configmanager from GitHub. We define a struct config with four fields: Val1, Val2, Val3, Val4.

package main

import (
    "fmt"
    "github.com/rrrcode9-golang-util/configmanager"
)

type config struct {
    Val1 int64
    Val2 string
    Val3 bool
    val4 float64
}

func main() {
    c := config{}
    configmanager.AssignConfiguration(&c)

    fmt.Printf("%+v\n", c)

}

AssignConfiguration method takes the reference of struct variable, and then the fields will be assigned with the values from the config file.

Now create a config file default.conf in the present directory.

Note: Alternatively, *.conf file can be created anywhere in the system and the same can be specified while running your program passing arguments as mentioned in Notes below.

# file: ./default.conf

Val1 = 40
Val2 = hello
Val3 = true
Val4 = 80.7

Now, run the following command:

$ go run main.go

Output: {Val1:40 Val2:hello Val3:true Val4:80.7}

Notes:

Alternatively, the config file location can be specified using arguments with "--config-file-path" or "-f" and with path of config file.

$ go run main.go --config-file-path <path of config file>
or
$ go run main.go -f <path of config file>

In case no config-file is specified, the configmanager program will search for default.conf file in the present working directory.

Thanks. I would appreciate your feedback/suggestions/comments.

Follow Me :

YouTube CodeFuture

golang

Flutter+Dart Frontend <-->Golang Backend API

CodeFuture — Wed, 24 Feb 2021 19:10:45 +0000

In Part-1/2 of this series, we are going to implement a Golang API server that caters to HTTP GET requests from clients. We will create a hypothetical store with products carrying attributes: Name, Price, and Count. A valid HTTP GET request gets a response with a list of products.

In Part-2/2 of this series, we are going to develop a Flutter App that will execute API calls to the server. The fetched data in JSON format from the server will be displayed using ListView.

I hope you enjoyed this post. I would appreciate your feedback/suggestions/comments in the comment section below.

Thanks.

Follow Me :

YouTube CodeFuture , Twitter

Link to GitHub :

Source Code

A step-by-step guide to set up an SFTP file transfer server in Linux

CodeFuture — Wed, 24 Feb 2021 19:05:51 +0000

In this tutorial, we are going to learn how to set up a remote server for secure file transfer between the user's local machine and remote server over SFTP. SFTP is known as Secure File Transfer Protocol or SSH File Transfer Protocol.

Before we start, please ensure the following:

You have access to a Linux remote machine (this tutorial used Ubuntu server distro 20.xx)
You have login access to a remote machine through SSH
The User access you have is listed under sudo group
The local machine can be of any OS - Mac, Linux, Windows, etc. - on which you can have access to a remote machine through SSH.

Let's first set up a user account typing the following commands in a terminal of the remote machine.

1. Create a user group

sudo groupadd sftpgroup

2. Create user

sudo useradd -m vyndour

3. Assign a password to the user

sudo passwd vyndour

4. Add user to our sftpgroup

sudo usermod -a -G sftpgroup vyndour

6. Make user the owner of it's own directory

sudo chown vyndour /home/vyndour

5. Give read(r),write(w) and execute(x) access of user's directory to only user

sudo chmod 700 /home/vyndour

In case we need to add more users, we can repeat steps 2-6.

Now, let's install the openssh server and set up SFTP settings.

7. First, let's update the existing packages

sudo apt update
sudo apt upgrade

8. Install openssh-sever

sudo apt install openssh-server

9. Open SSHD_config file

sudo nano /etc/ssh/sshd_config

10. Copy the following lines at the end of the _sshd_config_file

# FOR SSH key authentication

PubkeyAuthentication yes
AuthorizedKeysFile    .ssh/authorized_keys

# FOR password authentication

PasswordAuthentication yes

#  SFTP configuration

Match group sftpgroup
ChrootDirectory /home
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp

ctrl+s : save the sshd_config file and ctr+x: exit from sshd_config file

If we want SSH key authentication for SFTP file transfer, we need to set PubkeyAuthentication yes and also we need to specify the file name holding SSH public key AuthorizedKeysFile .ssh/authorized_keys. We will create SSH key in step - 12 below.

Similarly, if we want password authentication for SFTP file transfer, we need to set PasswordAuthentication yes. The password is the user's password that we have set earlier in step-3.

11. For SSH public key authetication , follow the following steps to create and install SSH keys

Open a terminal in our local machine, and create a pair of SSH private and public keys by running the following command.

ssh-keygen -f sftp_rsa -t rsa

Once we run the above command, two files will be generated - one private key sftp_rsa and the public key sftp_rsa.pub. Let's Keep the private key securely with read(r) access to only the user in the user's local system.

We can always change the type of key. Let's stick to the default RSA type key in this tutorial.

# On  user's local machine
sudo chmod 400 <path to the private key in user's local machine>

Then, in our remote PC create a file in /home/vyndour/.ssh and name it as authorized_keys. Copy the content of public key sftp_rsa.pub from local machine to authorized_keys file in a remote machine.

create a .ssh directory in the user's directory in the remote machine.

sudo mkdir /home/vyndour/.ssh

Open a new file with the name authorized_keys in the remote machine

    sudo nano /home/vyndour/.ssh/authorized_keys

Manual task: copy and paste the content manually from sftp_rsa.pub (local machine) key to the authorized_keys (remote machine)

Deny write(w) and execute(x) of authorized_keys by the user with the following command in the remote machine.

sudo chmod 644 /home/vyndour/.ssh/authorized_keys

As an alternative to the above method, you can transfer the public-key file sftp_rsa to the remote server using SCP. In our local machine, use SCP to transfer sftp_rsa.pub file to root directory of remote machine with the following command.

sudo scp -i <ssh key that gives access to remote machine> <path to public key in user's local machine> <sudo user>@<ip address of remote machine>:/

create a .ssh directory in the user's directory in the remote machine.

sudo mkdir /home/vyndour/.ssh

In the remote machine, create a file in /home/vyndour/.ssh and name it as authorized_keys, and append the key from sftp_rsa.pub file located in root directory/.

sudo touch /home/vyndour/authorized_keys
sudo cat /sftp_rsa.pub >> /home/vyndour/.ssh/authorized_keys

Deny delete of authorized_keys by the user with the following command in the remote machine.

sudo chmod 644 /home/vyndour/.ssh/authorized_keys

Don't forget to remove the public key from the root directory.

sudo rm /sftp_rsa.pub

12. Restart SSH

Let's now restart the ssh server with the following command:

sudo systemctl restart sshd

13. Monitor the logs (Optional for debugging)

In case we need to debug the login through SSH, we can open the /var/log/auth.log file to live monitor the logs.

sudo tail -f /var/log/auth.log

14. Browse User's directory from the local machine

We can now use software like FileZilla, Cyberduck, or similar to browse the user directory in the remote machine over SFTP with the following typical entries in our local machine:

Host: sftp://vyndour@
Port : 22 # Default SSH port
Password: User's password # if PasswordAuthentication yes in sshd_config file
Private key path: # if PubkeyAuthentication yes in sshd_config file

We are done! :)
I hope you enjoyed this post. I would appreciate your feedback/suggestions/comments in the comment section below.

Thanks.

Follow Me :

YouTube CodeFuture , Twitter

#SFTP #Linux #Ubuntu #SSH #File Transfer

CHAT-SERVER using gRPC Bidirectional Streaming

CodeFuture — Wed, 24 Feb 2021 18:22:53 +0000

In this post, we will cover how to create a basic chat server for one-to-one chatting using gRPC bidirectional streaming. We are going to use Go language for both gRPC server and gRPC client programs.

Protocol-buffer or protobuf is an efficient format of data transfer that is preferably used with gRPC. First, we will create a protobuf file with messages that define the format of communication between gRPC server and client and, we also define Remote Procedure Call (RPC) services in the same protobuf file.

Create a protobuf file




Let's first create a **chat.proto** file in the project root directory and copy the following lines.



```protobuf
// path: ./chat.proto

syntax = "proto3";

package chatserver;

message FromClient {
    string name = 1;
    string body = 2;
}

message FromServer {
    string name = 1;
    string body = 2;
}

service Services {

    rpc ChatService(stream FromClient) returns (stream FromServer){};

}

The syntax = 'proto3' specifies that we are using proto3 version syntax (the default version while writing this post is proto2). We have also specified a package name as chatserver. Then, we have defined two messages, one for clients to server communication


 and one for the server to client communication

 ```FromServer```

.  The fields in the message are identical in this case though; the `name` is for the user's name and the body is for the user's message.

To generate the service interface, we need to define **RPC** methods in the `chat.proto` file. In this example, we have defined only one RPC method i.e.`ChatService`. The `ChatService` method will be called by clients to set up bidirectional gRPC streams between client and server.

### Compile protobuf file `chat.proto` 

Now that we have our `chat.proto` file is ready. Next step is to compile the `chat.proto` file to generate the compiled `*.pb.go` file with **Interfaces** that we will be using to send and receive messages.

Before you proceed, please ensure that you have installed the following in your system, and both **protoc** and **protoc-gen-go** are in system $PATH:

> - [protobuf compiler](https://github.com/protocolbuffers/protobuf/releases) 
> - Golang Protobuf plugin:

 ```go install google.golang.org/protobuf/cmd/protoc-gen-go```



Now, run the command below in your preferred shell with PWD as your project root directory to generate golang code referring to `chat.proto` file.



```bash
$ protoc --go_out=plugins=grpc:chatserver chat.proto

The above command will generate a file chatserver.pb.go inside a directory ./chatserver/.

Create a file


 under `package main`

Let's create our project as a **Go module** for dependency management by executing the following command in our project root directory. This will create a `go.mod` file that would identify our project as a module.

$ go mod init grpcChatServer




Now, create a file named as `server.go` in our project root directory. We define a listener associated with a port which can be referred from environment variables. In case `PORT` is not set as a system environment, then let's assign 5000 (some port) as the default Port for our gRPC server.



```go
func main() {
  // assign port
  Port := os.Getenv("PORT")
    if Port == "" {
        Port = "5000"// port default : 5000 if in env port is not set
    } 

  // initiate listener
    listen, err := net.Listen("tcp", Port)
    if err != nil {
        log.Fatalf("Could not listen @ %v :: %v", Port, err)
    }
  log.Println("Listening @ "+Port)  

    // ...
}

Create a gRPC server instance that would listen and server through assigned Port.

grpcServer := grpc.NewServer()
err = grpcServer.Serve(listen)
    if err != nil {
        log.Fatalf("Failed to start gRPC server :: %v", err)
    }

Though we have created and also can run the gRPC server, it will not do anything impressive. Let's create a chatserver.go file that invokes interfaces defined in chatserver.pb.go file which will help to establish bidirectional stream between server and client.

Create chatserver.go file under

package chatserver

Let's create a new file named aschatserver.go inside chatserver directory (this is the same directory that has chatserver.pb.go file generated earlier through the proto compiler).

Now, define a struct messageUnit for handling the message in the server. We also create another struct messageQue that basically a slice MQue of type messageUnit and variable mu of type sync.Muex . Go sync.Mutex implements memory Lock() and UnLock() which are used to avoid race conditions while accessing global variables asynchronously through multiple threads.

type messageUnit struct {
    ClientName        string
    MessageBody       string
    MessageUniqueCode int
    ClientUniqueCode  int
}

type messageQue struct {
    MQue []messageUnit
    mu sync.Mutex
}

var messageQueObject = messageQue{}

In addition to the above, lets define a type struct with name ChatServer which implemets a method ChatService as defined in chat.proto file. ChatService method takes an argument of type Services_ChatServiceServer (defined in chatserver.pb.go file) and returns error.

type ChatServer struct {
}

//ChatService 
func (is *ChatServer) ChatService(csi Services_ChatServiceServer) error {

// ...

}

Now, inside our chatservice method we need to call two methods - one for receiving messages from stream and one for sending messages to stream. With recieveFromStream method, we parse the message from the client and append it to our messageQue object.

// recieve from stream
func recieveFromStream(csi_ Services_ChatServiceServer, clientUniqueCode_ int) {

    for {
        req, err := csi_.Recv()
        if err != nil {
            log.Printf("Error reciving request from client :: %v", err)
            break

        } else {
            messageQueObject.mu.Lock()
            messageQueObject.MQue = append(messageQueObject.MQue, messageUnit{ClientName: req.Name, MessageBody: req.Body, MessageUniqueCode: rand.Intn(1e8), ClientUniqueCode: clientUniqueCode_})
            messageQueObject.mu.Unlock()
            log.Printf("%v", messageQueObject.MQue[len(messageQueObject.MQue)-1])
        }

    }

}

With sendToStream method, we serialize the message from messageQue and send it to the designated client.

//send to stream
func sendToStream(csi_ Services_ChatServiceServer, clientUniqueCode_ int, errch_ chan error) {

    for {

        for {
            time.Sleep(500 * time.Millisecond)
            messageQueObject.mu.Lock()
            if len(messageQueObject.MQue) == 0 {
                messageQueObject.mu.Unlock()
                break
            }
            senderUniqueCode := messageQueObject.MQue[0].ClientUniqueCode
            senderName4client := messageQueObject.MQue[0].ClientName
            message4client := messageQueObject.MQue[0].MessageBody
            messageQueObject.mu.Unlock()
            if senderUniqueCode != clientUniqueCode_ {
                err := csi_.Send(&FromServer{Name: senderName4client, Body: message4client})

                if err != nil {
                    errch_ <- err
                }
                messageQueObject.mu.Lock()
                if len(messageQueObject.MQue) >= 2 {
                    messageQueObject.MQue = messageQueObject.MQue[1:] // if send success > delete message
                } else {
                    messageQueObject.MQue = []messageUnit{}
                }
                messageQueObject.mu.Unlock()

            }

        }

        time.Sleep(1 * time.Second)

    }

}

Let's call both recieveFromStream and sendToStream method from ChatService method through go routines. We have created one channel of type error which is used to pass error, if any, to the client during sending message to stream; the same error also would lead to termination of the connection from server-side. Every time a client connects to the server, two go routines would be spawned for recieving and sending messages. Whenever multiple Go-routines asynchronously access a variable (memory location), mutex blocking is used above to avoid Race condition.

// ChatService  
func (is *ChatServer) ChatService(csi Services_ChatServiceServer) error {

    clientUniqueCode := rand.Intn(1e3)

    // recieve request <<< client
    go recieveFromStream(csi, clientUniqueCode)
    //stream >>> client
    errch := make(chan error)
    go sendToStream(csi, clientUniqueCode, errch)

    return <-errch
}

Now, we need to register ChatService by adding the following two lines in server.go file.

// register ChatService
    cs := chatserver.ChatServer{}
    chatserver.RegisterServicesServer(grpcServer, &cs)

with this, we have finished our serverside code for gRPC bidirectional streaming. We can choose any language that supports gRPC for creating client stub. For simplicity, in this post let's create our client-side program in Golang. The chatserver.pb.go file also includes interfaces for the client.

Create `client.go` for client side stubs

let's create a client.go file in our project root directory. The main intention now is to have two clients that would first establish bidirectional streaming connections with the server and then chat with each other. In this post, we will be using the console in the terminal for posting and receiving messages.

For the sake of simplicity, in this post, we have two clients who communicate through one-to-one message. Clients shall have a name and would be able to send and receive messages. Let's define a struct clientHandle which basically has two fields one is clientName and the second one is the stream. clientHandle implements two methods one for sending message sendMessage and one for receiving message receiveMessage.

type clientHandle struct {
    stream     chatserver.Services_ChatServiceClient
    clientName string
}

// Assign name
func (ch *clientHandle) clientConfig() {

    reader := bufio.NewReader(os.Stdin)
    fmt.Printf("Your Name : ")
    msg, err := reader.ReadString('\n')
    if err != nil {
        log.Fatalf("Failed to read from console :: %v", err)

    }
    ch.clientName = strings.TrimRight(msg, "\r\n")

}

// send Message
func (ch *clientHandle) sendMessage() {

    for {
        reader := bufio.NewReader(os.Stdin)
        clientMessage, err := reader.ReadString('\n')
        clientMessage = strings.TrimRight(clientMessage, "\r\n")
        if err != nil {
            log.Printf("Failed to read from console :: %v", err)
            continue
        }

        clientMessageBox := &chatserver.FromClient{
            Name: ch.clientName,
            Body: clientMessage,
        }

        err = ch.stream.Send(clientMessageBox)

        if err != nil {
            log.Printf("Error while sending to server :: %v", err)
        }

    }

}

// receive message
func (ch *clientHandle) receiveMessage() {

    for {
        resp, err := ch.stream.Recv()
        if err != nil {
            log.Fatalf("can not receive %v", err)
        }
        log.Printf("%s : %s", resp.Name, resp.Body)
    }
}

In the main function, the client dial gRPC server addressing the port that the server is listening to. The client.ChatService method is called with a context to initiate a bidirectional stream between the client and the gRPC server. Next, we are running sendMessage and receiveMessage loops in separate go-routines. The main method is blocked by a dummy chan.

func main() {

    const serverID = "localhost:5000"

    log.Println("Connecting : " + serverID)
    conn, err := grpc.Dial(serverID, grpc.WithInsecure())

    if err != nil {
        log.Fatalf("Failed to connect gRPC server :: %v", err)
    }
    defer conn.Close()

    client := chatserver.NewServicesClient(conn)

    stream, err := client.ChatService(context.Background())
    if err != nil {
        log.Fatalf("Failed to get response from gRPC server :: %v", err)
    }

    ch := clientHandle{stream: stream}
    ch.clientConfig()
    go ch.sendMessage()
    go ch.receiveMessage()

    // block main
    bl := make(chan bool)
    <-bl

}

Now we are done! Let's type the following commands on the terminal to run our gRPC chatserver and clients.

# run server (Termina-1)

$ env PORT=5000 go run server.go

# Run client.go file in two separate terminals (Terminal -2 and Terminal -3)

$ go run server.go

I hope, you enjoyed this post. I would appreciate your feedback/suggestions/comments in the comment section below.

Thanks.

** #gRPC #BidirectionalStreaming #Golang #chatServer **

DEV Community: CodeFuture

A simple configmanager program for assigning values to variables from a custom config file | Golang

Configuration File Format

Struct Definition

configmanager usage

Follow Me :

golang

Flutter+Dart Frontend <-->Golang Backend API

Follow Me :

Link to GitHub :

A step-by-step guide to set up an SFTP file transfer server in Linux

1. Create a user group

2. Create user

3. Assign a password to the user

4. Add user to our sftpgroup

6. Make user the owner of it's own directory

5. Give read(r),write(w) and execute(x) access of user's directory to only user

7. First, let's update the existing packages

8. Install openssh-sever

9. Open SSHD_config file

10. Copy the following lines at the end of the _sshd_config_file

ctrl+s : save the sshd_config file and ctr+x: exit from sshd_config file

11. For SSH public key authetication , follow the following steps to create and install SSH keys

12. Restart SSH

13. Monitor the logs (Optional for debugging)

14. Browse User's directory from the local machine

Follow Me :

CHAT-SERVER using gRPC Bidirectional Streaming

Create a protobuf file

Create a file

Create chatserver.go file under

Create client.go for client side stubs

Follow Me :

Link to GitHub :

Useful Links :

Create `client.go` for client side stubs