DEV Community: Anton Gubarev

Testing cli tool with logging

Anton Gubarev — Sun, 07 Aug 2022 11:01:20 +0000

In the previous article I described how to start and manage processes. The next step is to implement a cli utility that uses the resulting code. The task of implementing such a utility is not very difficult, but this is how to test it - this is already a more interesting. In this article I will describe one of the ways of testing that involves reading the output logs.

Flags

To start the necessary process it must be something passed to the utility. In addition, configuration values are required to start, for example

t Time to live of lock.
k Lock key name
l Logging level

and probably something else will be added in the future. The final kind of launch command would like to see such
djob [options] - command.
Command is the command (with arguments to run). -- should separate the arguments of the utility from the arguments of the program to run.
There are several excellent solutions for working with command-line arguments, such as spf13/cobra or urfave/cli. But they’re good for building an interface out of a lot of commands, and for one (as in my example) they’re redundant. So I used the flag library.
I have defined the following structure with arguments:

type CliParams struct {
    key      *string
    cmdArgs  []string
    logLevel *string
    ttl      *int
}

And made a function that parses arguments:

func parseCliParams(args []string) (*CliParams, *flag.FlagSet, error) {
    flags := flag.NewFlagSet("djob", flag.ContinueOnError)

    params := &CliParams{
        key:      new(string),
        cmdArgs:  []string{},
        logLevel: new(string),
        ttl:      new(int),
    }

    flags.StringVar(params.key, "k", "", "Lock key. Default: filename")
    flags.StringVar(params.logLevel, "l", "error", "Log level: debug|info|warn|error|fatal. Default: error")
    flags.IntVar(params.ttl, "t", 60, "Ttl in seconds. Default: 60")
    flags.Usage = func() {
        fmt.Fprint(os.Stderr, "Usage: djob [OPTIONS] -- [command with arguments and options] \n")
        flags.PrintDefaults()
    }
    if err := flags.Parse(args); err != nil {
        return nil, flags, fmt.Errorf("parse flags: %v", err)
    }

    for k, arg := range args {
        if arg == "--" {
            params.cmdArgs = args[k+1:]
            break
        }
    }

    if len(params.cmdArgs) == 0 {
        return nil, flags, errors.New("command is required")
    }

    if len(*params.key) == 0 {
        params.key = &params.cmdArgs[0]
    }

    return params, flags, nil
}

To work with arguments I need to create flagset. There may be a few of these sets, but I’m good with one now.
I’ve identified the flags I wrote about earlier. Define them the types and pass the address of the variable (the field in the structure) where the parsed value should be placed. In order to make it easier for the user to understand how to use the tool I defined the function Usage. It prints in the output information, as well as descriptions of all flags and their default values.
Then I need to call the flagset method Parse([]string). This method gets the user’s arguments on the input. Next it will be os.Args. I deliberately made this feature independent of os.Args for the convenience of testing. Flags that I have not specified in the configuration will be ignored.
Next I found the flag -- and put everything after it as there is in the cmdArgs parameter. This is the command that the user wants to run. In addition, I made some minimal validation and at the output I got the structure command to run and parameters.

Logging

Before moving to testing I will talk about logging. Logging is another component that matters for testing implementation. In any case, it is required in almost any program, including the utility I describe.
I used the library logrus, because it has the ability to intercept logs with hooks. And I used this feature to read logs during testing. At the same time, the code of the main function itself does not have to adapt to tests.

type testHook struct {
    channel chan *log.Entry
}

func (hook *testHook) Fire(entry *log.Entry) error {
    hook.channel <- entry
    return nil
}

func (hook *testHook) Levels() []log.Level {
    return log.AllLevels
}

func newTestLogger() (*log.Logger, chan *log.Entry) {
    logger := log.New()
    logger.Out = ioutil.Discard
    logger.Level = log.DebugLevel

    entryChan := make(chan *log.Entry, 0)

    logger.Hooks.Add(&testHook{channel: entryChan})

    return logger, entryChan
}

I created a new logger with the output turned off and added a testHook to it. It instantly sends the logger to the channel instead of putting it into the console. Not only the messages themselves, but the whole structure that contains the full information, including the levels of the log entry, time, etc. It is all very convenient.

Main function

Now I can go directly to the main function. In it, I need to call parseCliParams and pass the os.Args there, and then start the process as I described in the previous article.
However, I have two obstacles:

I can’t test main function
I don’t have the ability to pass various arguments in tests other than os.Args (or it will be very ugly and unreliable) So I implemented Exec():

func Exec(osArgs []string, logger *log.Logger) int {

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    params, flags, err := parseCliParams(osArgs)
    if err != nil {
        logger.Errorf("parse arguments: %v \n", err.Error())
        flags.Usage()
        return 2
    }

    lvl, err := logrus.ParseLevel(*params.logLevel)
    if err != nil {
        logger.Fatalf("ivalid log level %s \n", *params.logLevel)
    }
    logger.SetLevel(lvl)

    memLock := internal.NewMemoryLock()

    locked, err := memLock.Lock(ctx, *params.key, *params.clientId)
    if err != nil {
        logger.Errorf("Didn't lock: %v \n", err)
    }
    if !locked {
        logger.Info("Already locked \n")
        return 2
    }

    cmd, err := internal.ExecCmd(ctx, params.cmdArgs)
    if err != nil {
        logger.Errorf("exec command: %v \n", err)
        return 1
    }
    logger.Info("started")

    defer func() {
        unlocked, err := memLock.UnLock(ctx, *params.key, *params.clientId)
        if err != nil {
            logger.Errorf("can't unlock: %v", err)
        }
        if !unlocked {
            logger.Info("Already unlocked")
        }
    }()

    exitCode, err := internal.WaitCmd(ctx, cmd)
    if err != nil {
        logger.Fatalf("wait command: %v", err)
        return 1
    }

    return exitCode
}

memLock := internal.NewMemoryLock() This is a simple implementation of locks in memory. In this example, it is only necessary for the code to work. More locks I will consider in the following articles and now their implementation does not matter.

cmd, err := internal.ExecCmd(ctx, params.cmdArgs) Wrap on the function of starting a new process, which I discussed in the last article. It is not necessary to know the content of this function to understand the material of this article.

exitCode, err := internal.WaitCmd(ctx, cmd) This function waits for the process running with ExecCmd to complete. It returns the response code. In this case too, it is not necessary to know the content of the function.

I will go ahead and say that all the received code of the utility will be pushed on GitHub. But this is after I go through all the parts of the implementation step by step.

func main() {
    logger := log.New()
    os.Exit(Exec(os.Args, logger))
}

Testing

Finally I came to the most important thing for which this article - testing. I did all the necessary preparation.
I needed a script to run in the tests. It has to be a program that can run for a while and I need to be able to set that time.

#!/usr/bin/env bash

trap "echo SIGINT; exit" SIGINT

for (( i=1; i<$1; i++ ))
do
  echo "step $i"
  sleep $2
done

echo "finish"

In the first argument I set the number of iterations, and in the second time the iteration. Everything is simple and this should be enough.

Next is the test itself.

func TestRunJob(t *testing.T) {
    logger, entryChan := newTestLogger()
    testCases := []struct {
        name       string
        args       []string
        logEntries []*struct {
            level   log.Level
            message string
        }
    }{
        {
            name: "first",
            args: []string{"djob", "-k", "key", "-c", "client1", "l", "info", "--", "../tests/script.sh", "2", "2"},
            logEntries: []*struct {
                level   log.Level
                message string
            }{
                {
                    level:   log.InfoLevel,
                    message: "start",
                },
                {
                    level:   log.InfoLevel,
                    message: "finish",
                },
            },
        },
    }

    for _, tt := range testCases {
        t.Run(tt.name, func(t *testing.T) {
            ctx, cancel := context.WithCancel(context.Background())
            defer cancel()

            go func() {
                entryIndex := 0
                for {
                    select {
                    case <-ctx.Done():
                        return
                    case entry := <-entryChan:
                        if entryIndex > len(tt.logEntries) {
                            t.Errorf("got more than expected (%d) log messages", len(tt.logEntries))
                        }
                        if tt.logEntries[entryIndex].level != entry.Level {
                            t.Errorf("Expected level: %s, actual: %s", tt.logEntries[entryIndex].level, entry.Level)
                        }
                        if tt.logEntries[entryIndex].message != entry.Message {
                            t.Errorf("Expected log message: %s, actual: %s", tt.logEntries[entryIndex].message, entry.Message)
                        }
                        entryIndex++
                    }
                }
            }()

            cmd.Exec(tt.args, logger)
        })
    }
}

Let me describe the whole code step by step.

testCases := []struct {
    name       string
    args       []string
    logEntries []*struct {
        level   log.Level
        message string
    }
}

in logEntries I specified expected entries in logs. I am only interested in message and level.

go func() {
                entryIndex := 0
                for {
                    select {
                    case <-ctx.Done():
                        return
                    case entry := <-entryChan:
                        if entryIndex > len(tt.logEntries) {
                            t.Errorf("got more than expected (%d) log messages", len(tt.logEntries))
                        }
                        if tt.logEntries[entryIndex].level != entry.Level {
                            t.Errorf("Expected level: %s, actual: %s", tt.logEntries[entryIndex].level, entry.Level)
                        }
                        if tt.logEntries[entryIndex].message != entry.Message {
                            t.Errorf("Expected log message: %s, actual: %s", tt.logEntries[entryIndex].message, entry.Message)
                        }
                        entryIndex++
                    }
                }
            }()

In the test I run a go routine. It listens to the channel with logs. And then I just make a check whether the entry is in the log, which I expect or not.

cmd. Exec(tt.args, logger) I run the main function with the necessary arguments and configured logger.

Conclusion

I described how it is possible to organize the testing of the cli utility with the help of logger. Next you only need to make all the necessary test cases. It was the second article in the Distributed Locks series. Next, I’ll move on to the implementation of the locks themselves and I’ll put the result on GitHub.

RBAC with OPA

Anton Gubarev — Sun, 24 Jul 2022 09:27:52 +0000

Open Policy Agent (OPA) is a tool aimed at unifying the application of policies in different technologies and systems. It avoids the introduction of multiple authentication approaches separately for Kubernetes or Kafka or even for micro-services. The tool is part of CNCF and is used by companies such as Netflix, Cloudflare and Pinterest.
Let me briefly tell you how it works. The OPA is written in the golang language and can be used as an agent/daemon or as a golang library with extensibility. The Rego language is provided to create access policies.
Imagine that your company has several hundred micro-services and you need to restrict access to those that contain sensitive data, such as personal data of users. You can of course implement authorization on the handles of the service: verify token, store access policies in the service itself. But you may need to repeat the implementation of this mechanism several tens or hundreds of times for each such service, which of course will be very expensive and difficult.
It is much easier to implement the authorization mechanism once and let other services access it as needed. And it seems simple and obvious, too. But everything is complicated if we need to give access not only to the service handles, but also to the database (because the developers to search for problems can not do without access to the production database), also probably need access to namespace in the cluster of Kuberenets and everything else. Imagine how many times you have to go to different systems and give access.
OPA is a universal tool that can be integrated with other tools such as ssh, Kubernetes, Envoy and many others.
The picture below shows how it works.

The possibilities of using OPA are very wide. In this article I will describe one of the ways of use. Let me tell you how you can execute access policies using data in a SQL database using the RBAC approach. I will store access policies in the database because it is very flexible and easy to implement APIs and frontends to manage them.

Scheme

What are the main entities to be represented in the mechanism.

User. Unique login. Policy subject.
Group. Users may belong to a single group. Policy subject.
Desired action.
Role. A set of actions that are allowed to everyone who has this role. This will simplify the work with permissions because you do not have to issue the same set of permissions many times and give the role at once.
Service. This is an authorization system object. The rights are given to work with the service and its resources (databases, S3 repositories, secrets in vault and others).

--- groups
CREATE TABLE IF NOT EXISTS public.groups
(
    id serial,
    name character varying(250) NOT NULL,
    CONSTRAINT groups_pkey PRIMARY KEY (id),
    CONSTRAINT groups_name_key UNIQUE (name)
)

I made the name unique so there are no random takes.

--- users
CREATE TABLE IF NOT EXISTS public.users(
    id serial,
    login varchar(250) NOT NULL,
    name varchar(250) NOT NULL,
    group_id integer references groups(id),
    CONSTRAINT users_pkey PRIMARY KEY (id)
);

The user is unique in his login and referenced to one group. If you want, you can extend the example in this article to more than one group.

--- roles
CREATE TABLE IF NOT EXISTS public.roles
(
    id serial,
    name text NOT NULL,
    actions character varying(100)[] NOT NULL,
    CONSTRAINT roles_pkey PRIMARY KEY (id)
)

A role is a simple entity that stores an array of available permissions. I will bind subjects to a role.

CREATE TABLE IF NOT EXISTS public.service
(
    id serial,
    name character varying NOT NULL,
    owner_id integer,
    CONSTRAINT service_pkey PRIMARY KEY (service_id),
    CONSTRAINT service_owner_id_fkey FOREIGN KEY (owner_id)
        REFERENCES public.users (id) MATCH SIMPLE
        ON UPDATE NO ACTION
        ON DELETE RESTRICT
)

The service name is unique and should not be repeated, as well as each service has a user-owner, a chief responsible, in fact an admin.

CREATE TABLE IF NOT EXISTS public.rules
(
    id serial NOT NULL,
    user_id integer,
    group_id integer,
    role_id integer NOT NULL,
    service_id integer NOT NULL,
    CONSTRAINT rules_pkey PRIMARY KEY (id),
    CONSTRAINT rules_group_id_fkey FOREIGN KEY (group_id)
        REFERENCES public.groups (id) MATCH SIMPLE
        ON UPDATE NO ACTION
        ON DELETE CASCADE,
    CONSTRAINT rules_role_id_fkey FOREIGN KEY (role_id)
        REFERENCES public.roles (id) MATCH SIMPLE
        ON UPDATE NO ACTION
        ON DELETE RESTRICT,
    CONSTRAINT rules_service_id_fkey FOREIGN KEY (service_id)
        REFERENCES public.service (id) MATCH SIMPLE
        ON UPDATE NO ACTION
        ON DELETE CASCADE,
    CONSTRAINT rules_user_id_fkey FOREIGN KEY (user_id)
        REFERENCES public.users (id) MATCH SIMPLE
        ON UPDATE NO ACTION
        ON DELETE CASCADE
)

I implemented binding to the object through separate fields user_id, group_id. That is, the rule can be either user or group. First, it is done to simplify the example and understand how the work with the OPA. Secondly, to maintain consistency of the data in the database. In real examples, it is best to implement as a pair of fields subject_type/subject_id, this can be useful for extending the types of subjects (organization, command, other service, etc.)

Rego rule

Now I need to implement a rule that succeeds if one of the conditions is met:

User is the owner of the service. Owners are allowed any actions.
This user has permission for the requested action with the service.
User's group has permission for requested action with service.

There are two ways:

Upload to OPA all data from the database by user, group and rule and update this data every few minutes.
Make queries into database directly from rego rules.

The second way looks much preferable because

The data will always be up to date and will not have to be updated.
Do not need to store in memory a large amount of data, which in some cases can become huge.

So our rule will look like this

package auth

default is_allowed = false

is_allowed {
    is_user_service_owner(input.userId, input.serviceName)
}

is_allowed {
    has_user_permission(input.userId, input.serviceName, input.action)
}

is_allowed {
    has_user_group_permission(input.userId, input.serviceName, input.action)
}

I use id in some arguments and in some unique name, as in real life it is not always possible to engage in what you want and this makes the example more close to reality. I hope you understand what I mean)
By default is_allowed is false. This means that if none of the conditions below are met, the total result will be false too. Let’s move to the implementation of custom functions

Rego custom functions

The OPA allows us to extend the set of functions. I will show you the implementation of this feature on the example of has_user_permission. Other functions will differ only by SQL query inside and quite simply implemented by analogy.

type RegoHasUserPermission struct {
    userPermission HasPermission
}

func NewRegoHasUserPermission(userPermission storage.HasPermission) *RegoHasUserPermission {
    return &RegoHasUserPermission{
        userPermission: userPermission,
    }
}

func (r *RegoHasUserPermission) GetFunction() func(r *rego.Rego) {
    return rego.Function3(&rego.Function{
        Name:    "has_user_permission",
        Decl:    types.NewFunction(types.Args(types.N, types.S, types.S), types.A),
        Memoize: false,
    }, func(bctx rego.BuiltinContext, userId, serviceName, action *ast.Term) (*ast.Term, error) {
        userIDInt, err := termToInt(userId)
        if err != nil {
            return nil, fmt.Errorf("OPA function RegoHasUserPermission, arg `userId`: %v", err)
        }

        serviceNameStr, err := termToString(serviceName)
        if err != nil {
            return nil, fmt.Errorf("OPA function RegoHasUserPermission, arg `serviceName`: %v", err)
        }

        actionStr, err := termToString(action)
        if err != nil {
            return nil, fmt.Errorf("OPA function RegoHasUserPermission, arg `action`: %s", err)
        }

        result, err := r.userPermission.HasUserPermission(r.ctx, *userIDInt, *serviceNameStr, *actionStr)
        if err != nil {
            return ast.BooleanTerm(false), fmt.Errorf("OPA function RegoHasUserPermission: %v", err)
        }
        return ast.BooleanTerm(result), nil
    })
}

I created a structure into which I transmit some sort of storage as an interface.

type HasPermission interface {
    HasUserPermission(userId int, serviceName, action string) (bool, error)
}

Below I will show its implementation. I have defined the has_user_permission function with three arguments

First type.N. Argument type number. The user id will be passed here
Other type.S. String type argument. For service name and action name. To convert the arguments values I will receive during the rego function execution, I have implemented two functions termToString and termToInt, which convert an argument with a string into string type and a numeric into int type.

func termToString(arg *ast.Term) (*string, error) {
    astStringVal, ok := arg.Value.(ast.String)
    if !ok {
        return nil, fmt.Errorf("cannot convert term to string: %s", arg.String())
    }
    stringVal := string(astStringVal)

    return &stringVal, nil
}

func termToInt(arg *ast.Term) (*int, error) {
    number, ok := arg.Value.(ast.Number)
    if !ok {
        return nil, fmt.Errorf("cannot convert term to number: %s", arg.String())
    }
    intval, ok := number.Int()
    if !ok {
        return nil, fmt.Errorf("cannot convert term to int: %s", arg.String())
    }

    return &intval, nil
}

After that the arguments are received and there are no errors remaining only to execute the query to the database. Above in custom function I used the HasPermission interface. I implemented it as follows.

func (pg *UserPermissionControllerPg) HasUserPermission(userId int, serviceName, action string) (bool, error) {
    rule := struct {
        Id int `json:"id"`
    }{}
    query := `SELECT r.id FROM rules AS r
        LEFT JOIN service AS s ON r.service_id=s.service_id
        LEFT JOIN roles AS rr ON r.role_id=rr.id
        WHERE r.user_id=$1 
            AND s.service_name=$2 
            AND $3=ANY(rr.actions)`

    if err := pg.db.GetContext(context.Background(), &rule, query, userId, serviceName, action); err != nil {
        if err == sql.ErrNoRows {
            return false, nil
        }
        return false, fmt.Errorf("HasUserPermission: %v", err)
    }

    return true, nil
}

In this method, I verify that the required permissions are present in the user roles. As you remember, there are a set of rules that are tied to the user and service. And each such rule is tied to a role. The query looks for at least one record in the rules table, where there is a binding to the desired role, user, and service. And if it’s there, then the user has a rule.
By analogy it is possible to implement the functions rego is_user_service_owner and has_user_group_permission with the only difference that the rule search will go in a different way.

Integration

It remains only to integrate the received mechanism into the application.

type RegoAuthController struct {
    rego *rego.Rego
}

func NewRegoController(
    policiesPath []string,
    ruleName string,
    funcs ...func(*rego.Rego)) *RegoAuthController {
    opts := []func(*rego.Rego){
        rego.Query(ruleName),
        rego.Load(policiesPath),
    }
    if len(opts) > 0 {
        opts = append(opts, funcs...)
    }

    return &RegoAuthController{
        rego: rego.New(opts...),
    }
}

I created a structure in which there is a link to the object Rego. And in the constructor execute its initial configuration. In particular, you need to skip the paths in which the files with the rules are stored. In our example it is one, but you can create as much as you like. For example, divide them by domains. In addition, I have provided the opportunity to transfer to the designer additional functions that can perform the OPA configuration. In particular, I need to transfer the additional custom function I described above.

func (p *RegoAuthController) IsAllowed(ctx context.Context, input map[string]interface{}) (bool, error) {
    query, err := p.rego.PrepareForEval(ctx)
    if err != nil {
        return false, fmt.Errorf("IsAllowed PrepareForEval: %v", err)
    }
    resultSet, err := query.Eval(ctx, rego.EvalInput(input))
    if err != nil {
        return false, fmt.Errorf("eval rego query: %v", err)
    }

    if len(resultSet) == 0 {
        return false, errors.New("undefined result")
    }

    return resultSet.Allowed(), nil
}

This method starts the rule and transfers input-mapped arguments to it. In the rule that I showed you earlier, we’re just looking at this structure, which is also called.
Next on the code I think everything is clear. Prepared expression is executed. Since the executed rule can have several results in the response, the result set is returned. If, in addition to is_allowed, which we have now, we add some more function, for example is_owner, then the result of its execution will also be presented in the answer. In this case, we don’t need this, so check the validity of all the conditions (i.e., one) in the call to resultSet.Allowed().

Now RegoAuthController can be used either in the handler on which will come the rule call or somewhere else. It depends on the specific application. Below I’ll show you an example of what it might look like.

func (h *Handler) Handle(...) error {
    regoHasUserPermissionFunction := policy.NewRegoHasUserPermission(hasPermission, ctx)
    authController := policy.NewRegoController(
        []string{h.policyDir + "/auth.rego"},
        "data.auth.is_allowed",
        regoHasUserPermissionFunction.GetFunction(),
    )

    isAllowed, err := authController.IsAllowed(ctx, map[string]interface{}{
        "userId":      in.UserId,
        "serviceName": in.ServiceName,
        "action":      in.Action,
            }, nil)
    if err != nil {
        return fmt.Errorf("handler: %v", err)
    }
    out.IsAllowed = isAllowed

    return nil //nolint:govet
}

In this example, I created a new RegoController structure object and configured:

Passed the file in which are the rules (file contents I gave above in the article). If necessary, you can transfer several files.
Specifies which rule to work with. data.auth.is_allowed Consists of the package name and function name.
Passed a custom function as a pointer to an object structure. I configured it beforehand, transferring the ready connection to the database (so as not to install it again on each request). And then it remains only to call the method IsAllowed and pass the parameters from the request. I omit some implementation details such as establishing a database connection and working with the http framework as it would be superfluous information that is not related to the topic and can vary greatly from project to project.

Conclusion

I showed the whole mechanism. It can be extended further by adding the necessary functionality. For example, it is possible to grant rights to other entities such as an organization or other services. I hope that the presented approach will help you realize your tasks. If there are any questions, ask them in the comments or to me in the еmail.

Extending NeoVim with Lua

Anton Gubarev — Sun, 10 Jul 2022 13:39:23 +0000

I spend a lot of time with NeoVim because it is my main tool for development. It suits me more than others primarily because it has lua and the ability to write your plugins and automation in a convenient and uncomplicated way. Why do I need this when there are already many ready-made solutions for the most common problems? Because sooner or later, there are cases that are not among the ready. And there are also project-specific cases that will never be among the ready. In this article I will show in two examples how it is relatively easy to expand NeoVim and make your work more productive. All examples below can be viewed live in my dotfile.

Curl

None of the query UIs I know will give me a Vim-like environment. For example, I need to be able to search and navigate the answer from the API using the same hotkeys that I already have configured in my Vim. Previously, I always had to switch to Postman, write a query there and copy the answer to my editor to be able to analyze it. And all this with a mouse, which is very unusual for someone who used to use a lot of hot keys.

local utils = {}

function utils.getCurrentParagraph()
    local row, col = unpack(vim.api.nvim_win_get_cursor(0))

    -- search forward 
    local rowi = row
    while true do
        local lastLine = vim.api.nvim_buf_get_lines(0, rowi, rowi+1, false) or {""}
        if lastLine[1] == "" then break end
        if lastLine[1] == nil then break end
        rowi = rowi + 1
    end

    -- search back 
    local rowj = row
    while true do
        local lastLine = vim.api.nvim_buf_get_lines(0, rowj, rowj+1, false) or {""}
        if lastLine[1] == "" then break end
        if lastLine[1] == nil then break end
        rowj = rowj - 1
        if rowj < 1 then break end
    end

    local lines = vim.api.nvim_buf_get_lines(0, rowj+1, rowi, false)
    local result = table.concat(lines, " ")
    result = result:gsub('[%c]', '')

    return result
end

return utils

This function searches for the first empty line after the cursor position and then the same line until the cursor position. The empty line at the beginning and at the end I consider the border of the paragraph. I will take a step-by-step look at what happens in the function above.

local row, col = unpack(vim.api.nvim_win_get_cursor(0))

So I get the current cursor position in the buffer 0. Buffer 0 is always the current buffer. This API method returns a row and a column, but I’m only interested in a row.


local rowi = row
while true do
    local lastLine = vim.api.nvim_buf_get_lines(0, rowi, rowi+1, false) or {""}
    if lastLine[1] == "" then break end
    if lastLine[1] == nil then break end
    rowi = rowi + 1
end

By getting one line and increasing the count by 1 I check the row that it is empty and that it exists at all (for example if the end of the file is reached). All I need to do is memorize the row number. False in function arguments means that you do not want to throw an error if there is no index.
Then I look in the opposite direction and I find the top of the paragraph.
Then all I have to do is get the lines along the lines and merge them into one.

local lines = vim.api.nvim_buf_get_lines(0, rowj+1, rowi, false)
local result = table.concat(lines, " ")
result = result:gsub('[%c]', '')

Here I additionally delete the template [%c] The fact is that the buffer will have a large number of different special characters, which when placed in the console breed a lot of errors. Such as

line 1 v curl -X POST https://jsonplaceholder.typicode.com/comments
line 2 v ^I-H 'Content-Type: application/json'
line 3 v ^I-d '{
line 4 v ^I^I"postId": 1
line 5 v ^I}'

Okay, now I have the current paragraph.
To allow this function to be imported into other lua scripts I made it as a module and return it when the file is imported. Below in this article in another example I again use this module in another script. And so far I send the command to the terminal. As a terminal emulator I like toggleterm because it is flexible enough and has a convenient API.

local utils = require("apg.utils")

function execCurl()
    local command = utils.getCurrentParagraph()

    local Terminal = require('toggleterm.terminal').Terminal
    local run = Terminal:new({
        cmd = command,
        hidden = true,
        direction = "float",
        close_on_exit = false,
       on_open = function(term)
          vim.api.nvim_buf_set_keymap(term.bufnr, "t", "q", "<cmd>close<CR>", {noremap = true, silent = true})
        end,
    })

    run:toggle()

end

And now let me show you how it works.

SQL

To work with SQL databases already there are a huge number of excellent tools. Every IDE I’m familiar with has database plugins. Vim is included. LSP server already exists. But sometimes I just need to run a simple query, without getting distracted by going to other tools, looking for the right connection, or the prepared SQL query. Therefore, using the getCurrentParagraph function discussed above, I implemented a simple and fast start of SQL queries for PostgreSQL

local utils = require("apg.utils")

function execPgSql()
    local query = utils.getCurrentParagraph()
    local cfg = vim.api.nvim_buf_get_lines(0, 0, 1, false)
    local command = 'psql '..cfg[1]..' -c "'..query..'"'

    local Terminal = require('toggleterm.terminal').Terminal
    local run = Terminal:new({
        cmd = command,
        hidden = true,
        direction = "float",
        close_on_exit = false,
       on_open = function(term)
          vim.api.nvim_buf_set_keymap(term.bufnr, "t", "q", "<cmd>close<CR>", {noremap = true, silent = true})
        end,
    })

    run:toggle()
end

Now I have in the project files with prepared SQL queries, which I can reach and run unrealistically fast. Here is an example of such a file

-h localhost -p 5432 -U postgres -d demo -W

SELECT   a.aircraft_code,
         a.model,
         s.seat_no,
         s.fare_conditions
FROM     aircrafts a
         JOIN seats s ON a.aircraft_code = s.aircraft_code
WHERE    a.model = 'Cessna 208 Caravan'
ORDER BY s.seat_no;

SELECT   s2.aircraft_code,
         string_agg (s2.fare_conditions || '(' || s2.num::text || ')',
                     ', ') as fare_conditions
FROM     (
          SELECT   s.aircraft_code, s.fare_conditions, count(*) as num
          FROM     seats s
          GROUP BY s.aircraft_code, s.fare_conditions
          ORDER BY s.aircraft_code, s.fare_conditions
         ) s2
GROUP BY s2.aircraft_code
ORDER BY s2.aircraft_code;

Here you can see the additional first line. It includes the parameters of connection to the database server and is substituted before query

local cfg = vim.api.nvim_buf_get_lines(0, 0, 1, false)
local command = 'psql '..cfg[1]..' -c "'..query..'"'

I would still like to have a separate file for the configuration of the connection to the base and select the desired one before executing the request. But so far, it’s working fine for me. Maybe in the future I’ll work on this script. I’ll show you how it works.

And now let me show you how it works.

Conclusion

With these scripts, I speed up my work and make it more pleasant. Because I don’t get distracted by unnecessary actions, like switching to other tools and memorizing their features. I hope my examples have helped you understand the general principle of writing extensions for NeoVim and maybe find new opportunities for yourself.

Autocomplete with Fzf

Anton Gubarev — Thu, 07 Jul 2022 19:45:13 +0000

Since I spend a lot of time in the console, I constantly improve the configuration of my console tools. My experience is that the costs of tools are often paid back many times over. Increasing the speed and pleasure of automating routine operations, which makes me a much happier engineer. To date, I have grown up with enough different tools and scripts to share with some of them.

One such of my any tools is fzf. It is a tool for fuzzy search on a given set of elements that is passed to it. It can also be integrated with other tools (and there are already many ready-made integrations). It is a very simple yet powerful tool with which to facilitate your work. In the examples below I will show how I use fzf and automate part of my routine. All the examples below are personally convenient for me, but I hope it will give you ideas for applying it to yourself.

Docker
It was very tiring for me to stop and remove containers with these commands

docker ps -a
copy container id
docker stop {id}
docker rm {id}

(Yes then I found lazydocker, but it was after)

function ds() {
  cid=$(docker ps | sed 1d | fzf -q "$1" | awk '{print $1}')
    [ -n "$cid" ] && docker stop "$cid"
}

function drm() {
  cid=$(docker ps -a | sed 1d | fzf -q "$1" | awk '{print $1}')
  [ -n "$cid" ] && docker rm "$cid"
}

I mean, sure, it could be combined into one feature and stop and delete, but it was more convenient for me to do that.

Another example is removing old images that take up space. I prefer to delete by hand, because I know exactly what else I will need and I would not like to reload them.

function ds() {
  cid=$(docker ps | sed 1d | fzf -q "$1" | awk '{print $1}')
    [ -n "$cid" ] && docker stop "$cid"
}

function drm() {
  cid=$(docker ps -a | sed 1d | fzf -q "$1" | awk '{print $1}')
  [ -n "$cid" ] && docker rm "$cid"
}

With these two simple examples, you can understand how fzf works. It is transmitted strings that fzf shows and allows you to search for different templates. The result of the user’s choice is then passed on to the user to perform some actions with him. This way I organize myself into a autocomplete.

Git

More realistic and useful automation that I have implemented for myself

Checkout to a branch

gcor() {
  local branches branch
  branches=$(git branch --all | grep -v HEAD) &&
  branch=$(echo "$branches" |
           fzf-tmux -d $(( 2 + $(wc -l <<< "$branches") )) +m) &&
  git checkout $(echo "$branch" | sed "s/.* //" | sed "s#remotes/[^/]*/##")
}

First I got a list of all the branches, and then I passed this list to fzf. But the truth sometimes has to be enchanted and cut out the extra information. The thing is that fzf transmits the entire string that the user has chosen, including numbers and other redundant information that makes further action impossible.

Delete a branch
Periodically cleaning old local branches that distract attention

gbd() {
    local branche_to_delete
    branch_to_delete=$(git branch | fzf | xargs)

    if [ -n "$branch_to_delete" ]; then
         git branch --delete --force $branch_to_delete
    fi
}

Navigation

I am a big fan of Lf file manager and can move between directories with high speed. Especially using bookmarks in Lf. And because I’m working on a project that uses microservices architecture, you have to move between services quite often. But the thing is that in addition to directly moving to the desired directory I still perform some routine operations after being in the project folder:

make sure I’m in the main branch, not some old branch that’s already actually
make sure there are no uncommitted changes
pull the last changes from the repository
Remembering and keeping track of this prevents focusing on the current problem. So I solved this problem in the following way

function cdp() {
    local dirPathes
    IFS=';' 
    read -A PD <<< "$PROJECT_PATHES"
    for i in "${PD[@]}"; 
    do
        local dd=$(ls -d $HOME/$i/*)
        dirPathes=("${dirPathes} ${dd}")
    done

    local targerDir=$(echo $dirPathes | fzf)
    cd $targerDir

    local changes=$(git status --porcelain)
    if [ -z "$changes" ]
    then
        git checkout master
        git pull origin master
    else
        echo "There are uncommited changes. Can't checkout and update master"
    fi
}

I get a list of directories from the project’s root directory. I pass it to fzf for selection. I make the selected directory the current one. If there are no uncommitted changes, I upload to the wizard and update it. And if there is, I do not do anything, as it is necessary to deal with them manually so as not to accidentally lose the work.

Now I am one team switching between projects. It saved me a lot of time in total.

Kubernetes

I could not avoid one of the most important tools that is found in most projects. I am a big fan of k9s. This is a great tool that allows me to easily and quickly navigate the cluster and learn what happens in the pods and quickly see the statuses. However, k9s is not always appropriate. For example, when I only need to look at the logs of the container, or perform some command inside the container. Running k9s for this seems redundant. So I wrote a small script that allows me to do frequent operations no less quickly.

kexe() {
    local namespace
    namespace=$(kubectl get ns --no-headers -o custom-columns=":metadata.name" | fzf )

    local pod
    pod=$(kubectl get po -n "$namespace" --no-headers -o custom-columns=":metadata.name" | fzf)
    echo "$pod"

    local container
    container=$(kubectl get pods "$pod" -n "$namespace" -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}' | fzf )


    kubectl exec -it "$pod" -n "$namespace" -c "$container" -- /bin/bash
}

First I pick up the namespace from the list, then I pick up all the files from this namespace, and at the end of the container of the selected pod. Next I run kubectl exec and voila, I can work in the desired container.

Conclusion

This is just a small set of all the improvements I have accumulated. I gradually publish these scripts in this repository. Once they are adapted for publication, they are removed and sharpened for specific work projects.

Also a huge number of ready-made scripts can be found in the official fzf wiki. In addition, I often subscribe to other public repositories where authors publish their ideas. It is very interesting to observe the commits and the evolution of these projects. Thank you to all who make this work open!

Distributed locks in go without changing the app

Anton Gubarev — Sun, 29 May 2022 12:13:59 +0000

Intro

For processes that must always exist in only one instance in a distributed system, we implement locks. There are many suitable tools for this, mainly the key-value stores. And this is implemented as the project grows and most likely directly in the code of the application.And the application starts to know something about the infrastructure around it, but whether it’s true. Of course not. This makes it more difficult for developers to run the application in a dev or test environment. And if the infrastructure changes again, this again will require work on the service.
I’ve been studying the web for ready-made solutions to this problem, but I haven’t found anything suitable. In this article series, I’ll tell you how to implement distributed locks without changing the code of the application itself, and without wasting the time of the developers (especially if you have dozens or hundreds of services)
This is the first part where I will introduce the reader into linux processes and we will see how to run and control them programmatically.

Processes

In order to achieve the main goal, you need to understand how the processes in linux work. In the kernel, the processes are presented simply as a structure. The process has attributes and states. There are a lot of attributes in the structure, but a few are important to us now:

PID. Unique process identifier in system
PPID. Unique process identifier of parent
Process command. Which process command runs and in which directory
Return code. Appears when the process completes

In Linux, no process appears out of nowhere. Everyone has a process that spawned it - the parent process. The exception is the init process (pid 1), which starts when the kernel starts (in some distributions, systemd or others can replace init).
When a new process starts, the parent fork first, after which we get a nearly identical process, but with a new PID and PPID which equals the parent PID. In this case, all memory, file descriptors, current working directory and so on are copied.
The exec of the new command is then executed in the resulting fork.
All processes can be in several states of existence:

Birth
Ready. Ready to work and waiting for the Linux scheduler to start running it and allocate CPU resources.
Execution. The process is in progress.
Waiting. The process is pending or blocked by other reasons.
Completion (death). Completes its work and frees up resources. The parent waits for the child to complete the process and reads the result of its completion, the response code. On the basis of this data, he can make a decision, either to put the information in the stdout, or to try to restart the process, or write something in the logs. For example, a well-known nginx web server creates a new process on each connection, and logs information about all errors in the child processes. By the way, the child can also write to the same logs, as they inherit file descriptors from the parent. But what if the parent dies before the child? His child processes will not die with him. They will be transferred to another process, you can say adopted. Usually this is PID 1, but depending on the distribution, this may differ. This knowledge is enough to continue the conversation. We can write a util that will run our main application and at the same time implement locks. Then you do not have to describe the nuances of the infrastructure inside the application. I will delve into the details and add the capabilities of our util step by step, gradually, for a better understanding. In Go there is an os/exec package that allows us to do this. Let’s try to start some process.

func main() {
    var cmd *exec.Cmd
    if len(os.Args) == 1 {
        cmd = exec.Command(os.Args[1])
    } else {
        cmd = exec.Command(os.Args[1], os.Args[2:]...)
    }

    fmt.Printf("I`am process %d \n", os.Getpid())
    println("Let`s start a new process")
    var outb, errb bytes.Buffer
    cmd.Stdout = &outb
    cmd.Stderr = &errb
    if err := cmd.Run(); err != nil {
        panic(err)
    }
    fmt.Printf("New process finished. Pid: %d \n", cmd.Process.Pid)
    fmt.Println("out:", outb.String(), "err:", errb.String())
}

Result:

go run main.go ls
I`am process 13405
Let`s start a new process
New process finished. Pid: 13406
out: go.mod
main.go
 err:

Let’s analyze this code in more detail:

var cmd *exec.Cmd
if len(os.Args) == 1 {
        cmd = exec.Command(os.Args[1])
    } else {
        cmd = exec.Command(os.Args[1], os.Args[2:]...)
    }
}

I describe the command as a structure. I take the data from the arguments. Since the null argument is a compiled binary, then I subtract arguments starting with index 1, which will be the name of the file to be run, then there may be arguments (or maybe not)

var outb, errb bytes.Buffer
cmd.Stdout = &outb
cmd.Stderr = &errb

I will need to read the standard output of a running command to see what was displayed. This creates two buffers for standard output and error output. I will return to it after the util is finished.

if err := cmd.Run(); err != nil {
    panic(err)
}

This is where the command was triggered. First, the fork of the current process (the compiled binary file of our application) is executed, and then what we passed in the arguments is started. This is a child process and the parent is waiting for it to complete. As soon as the child completes, the parent continues. And at the end I print the child process output.

fmt.Println("out:", outb.String(), "err:", errb.String())

Manage processes

At the moment I could try to insert the logic of the locks before starting the process. But what to do if our util dies, the child process will remain running and there will be nobody to unlock and no more process will start. So when the parent completes, all the child must be completed first.
Linux has the ability to manage processes. Signals are designed to do this. Signals are some events in the system that are sent to the process. Signals may come from the system kernel (for example, a hardware failure) or from the user (for example, a key is pressed).
There are many and there is no need to consider them all. If you wish, you can view them in kernel. I will bring those that may be important to us further in the topic under consideration.

SIGINT. Interrupt. Sent when the user sends a shutdown signal to the process. Ctrl-C
SIGKILL. Immediately terminate the process. The program cannot process or ignore this signal and will be terminated by the kernel.
SIGTERM. Terminate. Polite finishing. The program must complete its actions and finish correctly.

This set of signals must be processed by the parent and completed by the child. Except for SIGKILL, but there will be a separate conversation about it much later. So, let’s see an example of how this can be implemented:

func main() {
    var cmd *exec.Cmd
    if len(os.Args) == 1 {
        cmd = exec.Command(os.Args[1])
    } else {
        cmd = exec.Command(os.Args[1], os.Args[2:]...)
    }

    fmt.Printf("I`am process %d \n", os.Getpid())
    println("Let`s start a new process")

    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr
    if err := cmd.Start(); err != nil {
        panic(err)
    }
    fmt.Printf("Process started. Pid: %d \n", cmd.Process.Pid)

    sigs := make(chan os.Signal, 1)
    signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM)

    ctx, cancel := context.WithCancel(context.Background())

    go func() {
        select {
        case sig := <-sigs:
            fmt.Printf("got signal: %s \n", sig.String())
            if err := cmd.Process.Signal(sig); err != nil {
                log.Fatalf("error sending signal to process: %v", err)
            }
            return
        case <-ctx.Done():
            println("finished go routin")
            return
        }

    }()

    if err := cmd.Wait(); err != nil {
        if _, ok := err.(*exec.ExitError); ok {
            log.Fatalf("Child process failed: %v", err)
        }
        log.Fatalf("Parent failed, wait command: %v", err)
    }

    println("process finished")
    cancel()
}

Function has changed a lot. Let’s look at it in detail.

cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Start(); err != nil {
    panic(err)
}

First, it is better to redirect process output streams to standard system outputs. Then you do not have to wait for the completion of the process to see its output. Especially if the process can work for a long time.
Second, I used Start() instead of Run(). This method does the same, starts a new process, but does not wait for its completion and continues the program.

sigs := make(chan os.Signal, 1)
signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM)

Subscribe in the parent process to receive the interrupt signals mentioned above. For this purpose a buffered channel of size 1 is created.

go func() {
    select {
    case sig := <-sigs:
        fmt.Printf("got signal: %s \n", sig.String())
        if err := cmd.Process.Signal(sig); err != nil {
            log.Fatalf("error sending signal to process: %v", err)
        }
        return
    case <-ctx.Done():
        println("finished go routin")
        return
    }
}()

In this goroutine I listen to signals from the kernel. And as soon as they come in immediately duplicate their child process. It is also planned to complete the goroutine if the parent process is completed before any signal is received. It may not be accepted at all if the child properly works out and ends on her own.

if err := cmd.Wait(); err != nil {
    if _, ok := err.(*exec.ExitError); ok {
        log.Fatalf("Child process failed: %v", err)
    }
    log.Fatalf("Parent failed, wait command: %v", err)
}

Now the parent is waiting for the child to complete the process. The execution will continue when the child sends the completion code.

Try the option when the process ends on the Sigint signal.

go run main.go ping 8.8.8.8
I`am process 18060
Let`s start a new process
Process started. Pid: 18061
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=56 time=37.743 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=45.169 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=44.006 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 37.743/42.306/45.169/3.261 ms
got signal: interrupt
process finished

And the option where the process ended before any signal was received:

go run main.go ls
I`am process 18095
Let`s start a new process
Process started. Pid: 18096
go.mod  main.go
process finished

Conclusion

In the first part of the article series, I explained how the processes are arranged, how to manage them and showed an example of implementation on Go. In the next part I will add lock here. Of course, there were not considered many other cases, such as how to shoot a lock if the process got SIGKILL. I’ll write about it later, gradually getting into the topic.

The Story of a Startup

Anton Gubarev — Sat, 14 May 2022 11:49:50 +0000

In this article I will tell the story of how I tried to create and launch a startup, having only a little free time and a specific goal. I’m sure many developers would like to work on own project and have thought about it more than once. I hope that the description of my mistakes will help others not to do the same.

Idea

So, it was around 2016. I was working on custom development and support, that is, outsourcing. I had my own small team (3-5 developers), office and regular customers. This small business was successful, bringing income, feeding my family, and giving pleasure to projects. However, I wanted more freedom in product selection, technical solutions, financial independence. The desire to self-realization also did not allow me to sleep at night. The soul of an engineer demanded its own product, with the strategy and the roadmap as it seems right and interesting to me. And of course, the yield is a little higher than it was at the time. And it seemed that my project could give all this and make me happy, because the recruited critical mass of paid customers can provide the desired level of profitability constantly, while the outsourcer client may leave at the most inopportune moment and not always his needs will coincide with my wishes.

The idea of the project was not born out of nowhere. I had a living and constant problem in the work. This is financial accounting. Since customers paid for the working hours, there were hundreds of hours to be counted each month, reports to be issued, accounts to be reconciled with customers, income to expenses, loss locations to be identified. Tax reports and salaries were also time-consuming. Since the business was very small, it was not possible to maintain an accountant. A suitable automated system did not exist at that time. A system that will be flexible and sensitive to my country’s tax laws.
My work with finance started out simple and gradually evolved and became more complex. The table below shows how I tried to plan my income and expenses, calculate the minimum profitable cost of an hour and calculate taxes.

It was enough for a while, but there came a point when I realized I didn’t see the important details. For example, how effective each person in the team, he pays for his work or brings a loss, how justified the cost of software, courier, etc. I needed details. And gradually the table was like this:

Of course, the figures are all made up. And this is only a small part of the tables that I used, the rest unfortunately did not survive. At some point I started to miss the features of the tables. Rather, to say that the cost of their development has become very high. I started looking for some more specialized and box-based solutions that would solve most of my problems immediately. We don’t really do CRMs in tables) Although it’s possible, there are solutions.
So all my searches ended in failure. Those solutions that could be found were directed to accounting, which is different from financial. It provides very little planning and analysis. In these programs there was a recording of facts and tax calculation, it is of course also necessary and useful, but not at all what is missing and does not solve the problem.

First try

From that moment I decided that this is the idea! I am a developer and I can code it! There was a huge amount of enthusiasm, ideas and hazy vision of the final product. But I really wanted to get the first prototype as soon as possible. Although at the time I wasn’t thinking about the prototype, I was thinking about a finished product with lots of possibilities. On my first try, I took the tools I used every day. At that time it was a framework yii2 On the frontend I did not use anything except jQuery and this was enough. VueJS was released later.
I started work. I immediately ordered a logo and landing page from a familiar designer. The familiar maker quickly turned the layout landing page into HTML. And the name came up immediately) Of course bought a domain for it.

Landing page unfortunately did not survive, as well as the source code of the first version, but there is still nothing in the source code that could be of interest.
It’s been less than a month, and I’ve already started to lose enthusiasm. As time went on, I worked in my free time, and the result was very small. One more or less working section that was nothing like the multi-functional product I had imagined. As he began to solve one problem, he realized that it was blocked by two more etc. On the horizon appeared a long work for many months before the launch. All this was very demotivated and gradually the first attempt was abandoned. I was unprepared for the work and underestimated them.

Second try

However, the desire for my project still did not let go. It came to me every day and in the end six months I decided to try again. I tried to take into account the mistakes of the past and started with the prototype. In Axure I drew the end result, even worked out well. Fortunately, there were also developments from the first attempt: source code base, landing page, experience. I didn’t have to start from scratch, so I started coding again.
This time, things were going much better thanks to the prototypes. I understood more what I was doing and why I didn’t have to redo the working code because the result doesn’t look like it should from the point of view of UI/UX.
In this mode, about 3 months have passed. I continued to work in my free time. And here happens an offer, very interesting and necessary. At that time I was ready to change the format of work from outsourcing to the company. Revenues were frozen in place and stopped growing. It seems that I have reached the ceiling when to further develop from an engineer it was necessary to become a manager and an entrepreneur. And I didn’t want it at all, because coding for me was and remains more interesting. So I accepted this offer and my life changed dramatically. The income grew, but the free time ceased to be. As well as the energy. It was a period of reformatting to a new lifestyle. And of course the project was abandoned again in such conditions. This time I was not ready for the changed circumstances.

One year later

I was working at a new company, and I was happy and interested. But one day I happen to find out that there’s a startup that’s exactly like my idea. It was like a lightning strike.

I’m sorry I can’t find a screenshot in English. Here’s a section with a list of income and expenses, and a graph and a prediction of cash breaks. Yes, it was all far from the multi-functional combine. The author of the project launched MVP and immediately it was paid and immediately he had sales. The author did not develop a bunch of features and instead made a product that could be used even now in the minimum mode.
Of course I was upset. My idea was alive and needed, and I couldn’t realize it and someone else took it.
And almost immediately, I discover a second project like this. It’s a little bit different than the first, it’s a little worse, it’s better, it’s better, but it works and it’s profitable for its founders.

It was the last straw and I decided to make a third attempt. I could not just throw away an idea that ended up being so popular.

Third try

In past attempts, every time there were serious problems in planning. It was primarily about planning the time I had and where to invest it first. I found a time for myself when I can do the project, so that there is an opportunity to pay attention to family and children too.

Mon through Sun from 21 to 0 hours. Time when children go to bed.
All full-time. It was decided to donate one holiday.
2-3 hours in the afternoon when the children were asleep. Of all this, I deducted 10% for periods when you need to rest, and such periods will definitely be. Now I have a picture of what time resource I have at my disposal. This could already allow for a more objective assessment of what I can do and in what time frame. Given the first two abandoned attempts I did not rule out that the third could go wrong. And I would not like to waste a lot of time. So I needed plan B. And I prepared it. First, the source code of the project must be used in any case. For example, put on open source and show as an example code or portfolio to prospective employers. Therefore, the quality of his writing had to be approached as responsibly as possible. The second thing I had to do was choose the technology that I could use to help me find the best jobs. At that point in the trend were Symfony and Angular and it was them I chose. Experience with them had no at all. This backup has helped me a lot since then, but about that later. After a month of Symfony, I didn’t open docs often, because I took care of this framework and just coded. But with Angular everything was more complicated. Since the frontend is not really mine, and the experience I had a little, I decided to take VueJs, which was much easier to work with. It’s been six months. I already have a more or less working prototype and a good design that I built myself and spent a lot of time on it. But, the stock of forces dried up. At competitors appeared more and more new features and to keep up with them I obviously could not and the launch was not everything. I tried looking for partners. A person who will take on some of the work, like marketing or UX or something, so I can focus on the development. But it turned out that there were no enthusiasts like me who were willing to work 60 to 70 hours a week for free. I tried to appeal to investors. But this did not suit me. First, I was not interested in very large shares for a modest infusion and hard working conditions. Second, investors wanted me to do a lot of management work instead of development, which also didn’t fit my purpose for what I was doing. Third, it is always a full time job. I could not afford such an adventure, as I already had a family at that time. The third attempt was drowned after about 7-8 months from the start. According to the report on the time spent, you can see how the enthusiasm burns and fades) In the amount of only a third attempt I spent more than 500 hours of free time. After a few years, I realized that even then it was possible and necessary to lay out the minimum functionality, even if free. The first users would certainly boost enthusiasm and perhaps now things would be different.

Conclusion

What matters in this story is not how successful the project turned out to be. The most important thing is the experience gained and the benefit I was able to derive from all this.
On the one hand, it seems that the startup failed. Yes, but that’s what I got valuable.

Got the experience with Symfony, which eventually helped me get to work with the requirement of this framework and increase my salary almost twice. I could say that I did achieve one goal
Looked at the development through the eyes of the business owner, the product owner and even the project manager. It allows me now to better understand these people and more often find common ground and find really suitable solutions
I got over the idea of startup, understood all the pros and cons and now it hardly bothers me) I have become more appreciative of employers and respect their work.
I learned to plan my work, which helped me to successfully lead my projects for a while, and it was also another useful milestone in my career.
And most importantly, the time spent with family has become more valuable.

I am very glad that I have had this experience in my life, but I cannot recommend it to others to repeat it. Let each one decide for himself, and hope this will help him to make the right decision.

Project sources:

Backend
Frontend

I will be happy if someone could bend the rules and launch their project

Terminal for every day

Anton Gubarev — Fri, 13 May 2022 18:15:23 +0000

The console is one of the main tools of many developers. Someone needs to open a terminal more often, someone less often. But one thing I notice exactly, this tool is greatly underestimated. In an era of convenient and advanced IDE, which are created to the maximum comfortable and working out of the box many engineers forget or do not learn about the features that give us a seemingly scary and "inconvenient console for admins". In this article I will talk about several utilities that I use most often in everyday work. I hope this is a description not only will it help to discover something new, but also to look differently at the use of the console.

Bat

Bat perhaps the easiest thing to use from my list. This is the same cat only with syntax highlighting. Let’s compare the two output options.

The difference is obvious. In my example, I have a screenshot of minimalist nginx config. In reality they are often much larger, especially if it is a main balancer in a large company and highlighting helps a lot. Highlight is not the only feature of the utility. It also has seemingly small, but very useful conveniences.

Scroll Large Files
Diffs
Reading a mans
Display of service elements (spaces, tabs, hyphens, etc.)
Logging.

It is difficult not to agree that here such logs to read much more pleasant and visually tail -n 100 access.log | bat -l log

Color schemes to choose from have several bat --list-themes. I like VSCode stylist. And of course there is a possibility to create your own schemes.
Also bat can be integrated into other tools, which I will talk about below.

Lf

Lf is a very fast and lightweight file manager that only does its job and does not absorb as combines like mc. With it, you can quickly move to the desired directory and stay in the same session as you were. Of course, you can perform all the basic operations that are required every day: copying, deleting, creating character links, moving, etc. Supports group operations. But still the main advantage for me was its programmability. You can create whole commands that run whole scripts on bash or something. In the wiki project a set of some ready-made functions using the ability to create your own commands.
The bookmark function was very useful. I have a set of directories to which I move quite often that cd even with auto-compile can be tiring. But everything changed when implementing a simple command:

cmd bookmark_jump ${{
   res="$(cat $LF_BOOKMARK_PATH/$(ls $LF_BOOKMARK_PATH | fzf))"
   lf -remote "send $id cd \"$res\""
}}

map bj bookmark_jump

cmd bookmark_create ${{
   read ans
   echo "$PWD" > "$LF_BOOKMARK_PATH/$ans"
}}

map bc bookmark_create

Now bj adds the current directory to the bookmarks, and bc lists all the new ones. This example clearly shows how easy it is to implement functionality.
Here is an example of another useful command. Delete to Trash with confirmation.

cmd delete ${{
   set -f
   printf "$fx\n"
   printf "delete?[y/n]"
   read ans
   [ $ans = "y" ] && mv $fx ~/.trash
}}

map d delete

Pressing d will first ask for confirmation of the operation, and then move to the trash. The flight for creativity in creating an instrument for themselves and their needs is unlimited.
Another useful feature is the ability to view files while navigating through directories. But even more importantly, you can attach a bat to the views! This is as simple as bookmarks. First we create scripts that will call when preview.

#!/bin/bash
unset COLORTERM
bat --color=always "$1"

Now add to Lf configuration.

set previewer ~/.config/lf/bat.sh

And now bat is used for browsing with all its advantages. And you can set any settings in the script above, including various options depending on the file type, its size and so on.
For vim users the pleasant news will be the opportunity to use for positioning cursor habits hjkl without moving your fingers on the arrows. This makes the Lf works extremely fast.
I highly recommend to refer to the wiki project where there is a description of many lf possibilities, as well as some ready-made ways of expansion.

Exa

Exa is a modern replacement for the old good ls. Unlike its predecessor

Uses highlight for different file types
Detects symlinks, displays the actual location of files
Supports git
Can display directories as a tree
Supports various color schemes and fonts (including icons)

Example of what an output might look like

fzf

Fuzzy finder is a search utility for a given set of elements that you can embed almost anywhere and anything. I’ll show you a simple example of finding the right process in the system. Most often used to search utilities like grep, which give the opportunity to look not all the sheet and search for keywords or templates what you need. Let’s see how it looks with fzf

ps aux | fzf --height 40% --border

Fzf makes it possible to use live search, which makes it more convenient, fast and visual. Conclusively, Fzf supports templates for fuzzy queries.

'wild Items that include wild
^music Items that start with music
.mp3$ Items that end with .mp3
etc

But the main convenience is the ability to embed fzf in any script or pipeline and make it interactive. Here is an example for stopping a running container.

function ds() {
  local cid
  cid=$(docker ps | sed 1d | fzf -q "$1" | awk '{print $1}')

  [ -n "$cid" ] && docker stop "$cid"
}

I have the source of all projects in the same directory. Since my company has many microservices, there are also a lot of catalogs with projects. Navigating through the file manager every time can be exhausting, as I can do this procedure several times a day. This simple script made my life easier:

function cdp() {
    local project_path
    project_path=$(ls ~/go/src/github.com/mycompany | fzf)

    cd ~/go/src/github.com/mycompany/$project_path

    changes=$(git status --porcelain)
    if [ -z "$changes" ]
    then
        git checkout master
        git pull origin master
    else
        echo "There are uncommited changes. Can't checkout and update master"
    fi
}

As you can see here not only moves to the desired directory, but switches to the wizard and its update, but only if there are no unchecked changes. It’s one of my scripts, in reality there are many more of them, and they are tailored to my particular needs. Automation requires a small investment, but after that it pays off with excess.
Examples where I use it are still:

Create a new branch in the git. The name is the same as the task number in Jira. I take the ticket list of those in the "In progress" status and assigned to me
I run the timer, choosing from the list of previously run timers. I have many similar records of the form "meeting", "duty", "planning" and so on.
Switching between kubernetes cluster. I have several dozen of them in operation As you can see very specific tasks for me and everyone here can create under themselves)

For typical tasks there is a huge number of ready-made solutions for all major areas of the workflow, both plugins and scripts. You can fix the second ones for yourself, not write from scratch.

git Work with git. Switch branches, search history with diffs viewing, interactive rebase and more.
kubernetes Helps to search for pods, namespaces and other frequent features
tmux More convenient and intuitive switching between windows, sessions and more.

In the wiki project there is a set of ready-made functions and scripts. These are the ones that seemed most useful to me

Switching between tmux sessions
Stop/Remove Containers, Clear Images
[A set of utility for vim](https:/github.com/junegunn/fzf/wiki/Examples-(vim). Navigation by buffers, tags, files (But I still prefer telescope)
and even chrome bookmarks By the way, for files there is a possibility to display a preview of their content before selecting it. And of course you can use bat.

Direnv

Direnv is a utility allows you to automate setting environment variables in a directory. Works this way. In a file .envrc contains the required environment variables and they are automatically displayed when moving to the directory. Do not do it manually and remember about it. That’s all) Quite a simple task, which without automation takes time.

An example from one of my projects

export GITHUB_TOKEN=***
export PET_BOT_TOKEN=***
export PET_DOMAIN=https://*.eu.ngrok.io
export PET_WEBHOOK_TOKEN=***

When I move to the project directory I get this result

direnv: loading ~/project/gorelex/.envrc
direnv: export +GITHUB_TOKEN +PET_BOT_TOKEN +PET_DOMAIN +PET_WEBHOOK_TOKEN

Conclusion

I told you about some of my utilities that I use every day. The flexibility of their settings and the possibilities of integration are simply limitless. You create your IDE for everything, gradually improving and complementing the automation you need. Of course, I did not tell about all the tools. There are still NeoVim and Tmux, everyone is worthy of not just a single article, but whole courses. I tried to show the main thing - the console is not difficult. The console is convenient and interesting.