DEV Community: Clivern

Lynx - A Fast, Secure and Reliable Terraform Backend

Clivern — Sat, 22 Jun 2024 19:43:01 +0000

Lynx is a Fast, Secure and Reliable Terraform Backend. It is built in Elixir with Phoenix framework.

Features:

Simplified Setup: Easy installation and maintenance for hassle-free usage.
Team Collaboration: Manage multiple teams and users seamlessly.
User-Friendly Interface: Enjoy a visually appealing dashboard for intuitive navigation.
Project Flexibility: Support for multiple projects within each team.
Environment Management: Create and manage multiple environments per project.
State Versioning: Keep track of Terraform state versions for better control.
Rollback Capability: Easily revert to previous states for efficient infrastructure management.
Terraform Locking Support: The project also supports Terraform locking, ensuring state integrity and preventing concurrent operations that could lead to data corruption
RESTful Endpoints: for seamless teams, users, projects, environments, and snapshots management.
Snapshots Support: for both projects and environments to ensure data integrity and provide recovery options at specific points in time.
Terraform Provider: Automate creation/updates of teams, users, projects, environments and snapshots with terraform.
Single Sign-On (SSO): Support for OAuth2 Providers like Azure AD OAuth, Keycloak, Okta … etc

Clivern / Lynx

🐺 A Fast, Secure and Reliable Terraform Backend, Set up in Minutes.

Lynx

A Fast, Secure and Reliable Terraform Backend, Set up in Minutes.

Lynx is a Fast, Secure and Reliable Terraform Backend. It is built in Elixir with Phoenix framework.

Features:

Simplified Setup: Easy installation and maintenance for hassle-free usage.
Team Collaboration: Manage multiple teams and users seamlessly.
User-Friendly Interface: Enjoy a visually appealing dashboard for intuitive navigation.
Project Flexibility: Support for multiple projects within each team.
Environment Management: Create and manage multiple environments per project.
State Versioning: Keep track of Terraform state versions for better control.
Rollback Capability: Easily revert to previous states for efficient infrastructure management.
Terraform Locking Support: The project also supports Terraform locking, ensuring state integrity and preventing concurrent operations that could lead to data corruption
RESTful Endpoints: for seamless teams, users, projects, environments, and snapshots management.
Snapshots Support: for both projects and environments to ensure data integrity and provide recovery options at specific points in…

View on GitHub

How To Manage Your Applications Go Environment with Goenv

Clivern — Wed, 13 Jul 2022 22:05:42 +0000

Goenv helps you to work with multiple golang versions at the same time whether on mac or linux operating system. It supports both global and per-application version configuration.

Here is a demo of goenv in action

Clivern / Goenv

🐺 Manage Your Applications Go Environment.

Goenv

Manage Your Applications Go Environment

Goenv helps you to work with multiple golang versions at the same time whether on mac or linux operating system. It supports both global and per-application version configuration.

Usage

Download the latest goenv binary. Make it executable from everywhere.

$ export GOENV_LATEST_VERSION=$(curl --silent "https://api.github.com/repos/clivern/Goenv/releases/latest" | jq '.tag_name' | sed -E 's/.*"([^"]+)".*/\1/' | tr -d v)

# For Linux
$ curl -sL https://github.com/clivern/Goenv/releases/download/v{$GOENV_LATEST_VERSION}/goenv_Linux_x86_64.tar.gz | tar xz

# For Mac
$ curl -sL https://github.com/clivern/Goenv/releases/download/v{$GOENV_LATEST_VERSION}/goenv_Darwin_x86_64.tar.gz | tar xz

Configure the goenv using the following command

$ goenv config

Add goenv shims to PATH using the following command. also append it to ~/.profile file to make it permanent.

$ export PATH="$HOME/.goenv/shims:"$PATH

# OR

$ eval "$(goenv init)"

Install a new go version 1.18 and set…

View on GitHub

Usage

Download the latest goenv binary. Make it executable from everywhere.



$ export GOENV_LATEST_VERSION=$(curl --silent "https://api.github.com/repos/spacewalkio/Goenv/releases/latest" | jq '.tag_name' | sed -E 's/.*"([^"]+)".*/\1/' | tr -d v)

# For Linux
$ curl -sL https://github.com/spacewalkio/Goenv/releases/download/v{$GOENV_LATEST_VERSION}/goenv_{$GOENV_LATEST_VERSION}_Linux_x86_64.tar.gz | tar xz

# For Mac
$ curl -sL https://github.com/spacewalkio/Goenv/releases/download/v{$GOENV_LATEST_VERSION}/goenv_{$GOENV_LATEST_VERSION}_Darwin_x86_64.tar.gz | tar xz

Configure the goenv using the following command



$ goenv config

Add goenv shims to PATH using the following command. also append it to ~/.profile file to make it permanent.



$ export PATH="$HOME/.goenv/shims:"$PATH

# OR

$ eval "$(goenv init)"

Install a new go version 1.18 and set as a global



$ goenv install 1.18
$ goenv global 1.18

To configure a local version different from the global



$ goenv local 1.18

To Uninstall a version



$ goenv uninstall 1.18

Show the used version either from current directory or parent directories or the global version.



$ goenv version

To list all installed versions



$ goenv versions

for a list of all available commands



$ goenv --help

🐺 Manage Your Applications Go Environment.

If you have any suggestions, bug reports, or annoyances please report
them to our issue tracker at <https://github.com/spacewalkio/goenv/issues>

Usage:
  goenv [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  config      Configure the goenv application.
  exec        Show the current go version.
  global      Set or show the global go version.
  help        Help about any command
  info        Print the goenv version
  init        Init the import path for goenv shims.
  install     Install a go version.
  license     Print the license
  local       Set or show the local application-specific go version.
  rehash      Refresh binaries under goenv shim directory.
  uninstall   Uninstall a specific go version.
  version     Show the current go version.
  versions    List installed go versions.

Flags:
  -h, --help   help for goenv

Use "goenv [command] --help" for more information about a command.

Under The Hood

Goenv is inspired by and works like rbenv. At a high level, goenv intercepts Go commands using shim executables injected into your PATH, determines which Go version has been specified by your application or globally, and passes your commands to the correct Go installation bin folder.

Understanding PATH

When you run a command like go or gofmt, your operating system searches through a list of directories to find an executable file with that name. This list of directories lives in an environment variable called PATH, with each directory in the list separated by a colon:



/usr/local/bin:/usr/bin:/bin

Directories in PATH are searched from left to right, so a matching executable in a directory at the beginning of the list takes precedence over another one at the end. In this example, the /usr/local/bin directory will be searched first, then /usr/bin, then /bin.

Understanding Shims

goenv works by inserting a directory of shims at the front of your PATH:



~/.goenv/shims:/usr/local/bin:/usr/bin:/bin

Through a process called rehashing, goenv maintains shims in that directory to match every Go command across every installed version of go like gofmt and so on.

shims are lightweight executables that simply pass your command to the right binary under the current go version, your operating system will do the following:

Search your PATH for an executable file named gofmt.
Find the goenv shim named gofmt at the beginning of your PATH
Run the shim named gofmt, which in turn fetch the target go version and use the gofmt inside go/bin directory.

Choosing the Go Version

When you execute a shim, goenv determines which Go version to use by reading it from the following sources, in this order:

The first .go-versionfile found by searching the current working directory and each of its parent directories until reaching the root of your filesystem. You can modify the .go-version file in the current working directory with the goenv local x.x.x command.
The global $HOME/.goenv/.go-version file. You can modify this file using the goenv global x.x.x command.

Getting Started With Apache Kafka and Java

Clivern — Thu, 01 Jul 2021 14:12:46 +0000

Installation

To add a dependency using Maven, use the following:

<dependency>
    <groupId>com.clivern</groupId>
    <artifactId>kafka-sdk</artifactId>
    <version>0.1.0</version>
</dependency>

To add a dependency using Gradle, use the following:

dependencies {
    compile 'com.clivern:kafka-sdk:0.1.0'
}

To add a dependency using Scala SBT, use the following:

libraryDependencies += "com.clivern" % "kafka-sdk" % "0.1.0"

To Create a Kafka Topic:

import java.util.HashMap;
import com.clivern.kafka.Configs;
import com.clivern.kafka.Utils;


HashMap<String, String> map = new HashMap<String, String>();
map.put("bootstrap.servers", "localhost:9092");
Utils.createTopic("clivern", Configs.fromMap(map));

Kafka Producer:

import com.clivern.kafka.Configs;
import com.clivern.kafka.Producer;
import com.clivern.kafka.Kafka;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;


Configs configs = new Configs();
configs.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
configs.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configs.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

Producer producer = (new Kafka()).newProducer(configs);

for (int i = 0; i < 10; i++) {
    ProducerRecord<String, String> record =
            new ProducerRecord<>("clivern", null, "Hello World " + i);

    producer.send(record).flush();
}

producer.close();

Kafka Consumer:

import com.clivern.kafka.Configs;
import com.clivern.kafka.Consumer;
import com.clivern.kafka.Kafka;
import com.clivern.kafka.HandlerCallbackInterface;
import com.clivern.kafka.FailureCallbackInterface;
import com.clivern.kafka.SuccessCallbackInterface;
import com.clivern.kafka.exception.MissingHandler;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.clients.consumer.ConsumerRecord;


Configs configs = new Configs();
configs.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
configs.put(ConsumerConfig.GROUP_ID_CONFIG, "clivern");
configs.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
configs.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
configs.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
configs.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);

Consumer consumer = (new Kafka()).newConsumer(configs);

HandlerCallbackInterface<ConsumerRecord<String, String>> handler =
        (record) -> {
            System.out.println("Message Received: " + record.value());

            // Throw error if message has error
            if (record.value().equals("error")) {
                throw new Exception("Error!");
            }
        };

SuccessCallbackInterface<ConsumerRecord<String, String>> onSuccess =
        (record) -> {
            System.out.println("Message Succeeded: " + record.value());
        };

FailureCallbackInterface<ConsumerRecord<String, String>> onFailure =
        (record, exception) -> {
            System.out.println(
                    "Message " + record.value() + " Failed: " + exception.getMessage());
        };

consumer.subscribe("clivern")
        .handler(handler)
        .onSuccess(onSuccess)
        .onFailure(onFailure)
        .run();

Please don't forget to replace localhost with kafka host.

For starters, don't miss this tutorial https://dev.to/clivern/getting-started-with-kafka-3mbi

Getting Started With Apache Kafka

Clivern — Fri, 18 Jun 2021 23:06:54 +0000

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Let's first run a single kafka node with docker and docker-compose.

Create a docker-compose.yaml file

---
version: '3'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.1.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-server:6.1.1
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'

After installing docker and docker-compose, Start kafka and zookeeper containers with the following command

$ docker-compose up -d

Get kafka container id to run some commands inside the container.

$ docker ps

$ docker exec -it $ID bash

To create a kafka topic clivern

$ cd /bin
$ kafka-topics --create \
  --bootstrap-server localhost:9092 \
  --topic clivern

To run a consumer

$ cd /bin
$ kafka-console-consumer \
    --bootstrap-server localhost:9092 \
    --topic clivern

To run a producer

$ cd /bin
$ kafka-console-producer \
    --bootstrap-server localhost:9092 \
    --topic clivern

To list all topics

$ cd /bin
$ kafka-topics --bootstrap-server localhost:9092 --list

Getting Started with Hashicorp Nomad

Clivern — Thu, 13 May 2021 13:46:56 +0000

Basic Setup (Not Recommended):

Download nomad binary & make it executable:

$ wget https://releases.hashicorp.com/nomad/1.0.5/nomad_1.0.5_linux_amd64.zip
$ unzip nomad_1.0.5_linux_amd64.zip
$ mv nomad /usr/local/bin/

Create configs directory & data directory:

$ sudo mkdir --parents /opt/nomad
$ sudo mkdir --parents /opt/data/redis
$ sudo mkdir --parents /etc/nomad.d
$ sudo chmod 700 /etc/nomad.d
$ sudo chmod 700 /opt/data

$ sudo touch /etc/nomad.d/nomad.hcl
$ sudo touch /etc/nomad.d/server.hcl
$ sudo touch /etc/nomad.d/client.hcl

Add this configuration to the /etc/nomad.d/nomad.hcl configuration file

datacenter = "dc1"
data_dir = "/opt/nomad"

Add this configuration to the /etc/nomad.d/client.hcl configuration file

client {
  enabled = true

  host_volume "redis_data" {
    path      = "/opt/data/redis"
    read_only = false
  }
}

plugin "docker" {
  config {
    volumes {
      enabled      = true
      selinuxlabel = "z"
    }
  }
}

Add this configuration to the /etc/nomad.d/server.hcl configuration file

acl {
  enabled = true
}
server {
  enabled = true
  bootstrap_expect = 1
}

Create a nomad service file

$ sudo touch /etc/systemd/system/nomad.service

Add this configuration to the nomad service file /etc/systemd/system/nomad.service

[Unit]
Description=Nomad
Documentation=https://nomadproject.io/docs/

[Service]
ExecStart=/usr/local/bin/nomad agent -config /etc/nomad.d
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target

Start nomad

sudo systemctl enable nomad
sudo systemctl start nomad
sudo systemctl status nomad

Genertal ACL secrets

$ nomad acl bootstrap
Accessor ID  = a8b724ef-b5eb-21c0-1a68-b0d3616e1e62
Secret ID    = 07fe0ee6-59bc-8ea0-aeeb-9b4e9edcb555
Name         = Bootstrap Token
Type         = management
Global       = true
Policies     = n/a
Create Time  = 2021-05-10 21:05:21.498072284 +0000 UTC
Create Index = 10
Modify Index = 10

$ export NOMAD_TOKEN=07fe0ee6-59bc-8ea0-aeeb-9b4e9edcb555

Create your first job to deploy a stateless container from the server ui http://127.0.0.1:4646/ui

job "api" {
  datacenters = ["dc1"]

  group "example" {
    task "server" {
      driver = "docker"

      config {
        image = "hashicorp/http-echo:0.2.1"

        args = [
          "-listen",
          ":5678",
          "-text",
          "hello world",
        ]
      }

      resources {
        network {
          mbits = 10

          port "http" {
            static = "5678"
          }
        }
      }
    }
  }
}

And another one for stateful container

job "cache" {
  datacenters = ["dc1"]

  group "redis" {

    volume "redis_data" {
      type      = "host"
      read_only = false
      source    = "redis_data"
    }

    task "server" {
      driver = "docker"

      volume_mount {
        volume      = "redis_data"
        destination = "/data"
        read_only   = false
      }

      config {
        image = "redis:4-alpine"

        labels = {
          "sh.hippo.service" = "redis"
          "sh.hippo.service_type" = "cache"
        }

        port_map {
          http = 6379
        }

        command = "redis-server"

        args = [
          "--requirepass",
          "mystery",
        ]
      }

      env = {
        HEALTHY_FOR    = -1,
      }

      resources {
        cpu    = 100
        memory = 256

        network {
          mbits = 10

          port "http" {
            static = "6379"
          }
        }
      }
    }
  }
}

Multi Nodes Setup (Recommended):

To fully understand how nomad cluster should look like, please check this guide!

Assuming we have two servers, one for the nomad server (Leader) and another for the client. Both servers are in DC1. The leader server has public IP $SERVER_PUBLIC_IP and private IP $SERVER_PRIVATE_IP.

First we run the leader or the nomad server by following the next steps:

Download nomad binary & make it executable:

$ wget https://releases.hashicorp.com/nomad/1.0.5/nomad_1.0.5_linux_amd64.zip
$ unzip nomad_1.0.5_linux_amd64.zip
$ mv nomad /usr/local/bin/

Create configs directory & data directory:

$ sudo mkdir --parents /opt/nomad
$ sudo mkdir --parents /opt/data/redis
$ sudo mkdir --parents /etc/nomad.d
$ sudo chmod 700 /etc/nomad.d
$ sudo chmod 700 /opt/data

$ sudo touch /etc/nomad.d/nomad.hcl
$ sudo touch /etc/nomad.d/server.hcl

Add this configuration to the /etc/nomad.d/nomad.hcl configuration file

datacenter = "dc1"
data_dir = "/opt/nomad"

Add this configuration to the /etc/nomad.d/server.hcl configuration file

acl {
  enabled = true
}

server {
  enabled = true
  bootstrap_expect = 1
}

bind_addr = "$SERVER_PUBLIC_IP"

addresses {
  http = "$SERVER_PUBLIC_IP"
  rpc  = "$SERVER_PRIVATE_IP"
  serf = "$SERVER_PRIVATE_IP"
}

advertise {
  http = "$SERVER_PUBLIC_IP:4646"
  rpc  = "$SERVER_PRIVATE_IP:4647"
  serf = "$SERVER_PRIVATE_IP:4648"
}

Create a nomad service file

$ sudo touch /etc/systemd/system/nomad.service

Add this configuration to the nomad service file /etc/systemd/system/nomad.service

[Unit]
Description=Nomad
Documentation=https://nomadproject.io/docs/

[Service]
ExecStart=/usr/local/bin/nomad agent -config /etc/nomad.d
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target

Start nomad

sudo systemctl enable nomad
sudo systemctl start nomad
sudo systemctl status nomad

Genertal ACL secrets

$ nomad acl bootstrap -address=http://$SERVER_PUBLIC_IP:4646
Accessor ID  = a8b724ef-b5eb-21c0-1a68-b0d3616e1e62
Secret ID    = 07fe0ee6-59bc-8ea0-aeeb-9b4e9edcb555
Name         = Bootstrap Token
Type         = management
Global       = true
Policies     = n/a
Create Time  = 2021-05-10 21:05:21.498072284 +0000 UTC
Create Index = 10
Modify Index = 10

$ export NOMAD_TOKEN=07fe0ee6-59bc-8ea0-aeeb-9b4e9edcb555

Then we run the client by following the next steps:

Download nomad binary & make it executable:

$ wget https://releases.hashicorp.com/nomad/1.0.5/nomad_1.0.5_linux_amd64.zip
$ unzip nomad_1.0.5_linux_amd64.zip
$ mv nomad /usr/local/bin/

Create configs directory & data directory:

$ sudo mkdir --parents /opt/nomad
$ sudo mkdir --parents /opt/data/redis
$ sudo mkdir --parents /etc/nomad.d
$ sudo chmod 700 /etc/nomad.d
$ sudo chmod 700 /opt/data

$ sudo touch /etc/nomad.d/nomad.hcl
$ sudo touch /etc/nomad.d/client.hcl

Add this configuration to the /etc/nomad.d/nomad.hcl configuration file

datacenter = "dc1"
data_dir = "/opt/nomad"

Add this configuration to the /etc/nomad.d/client.hcl configuration file

client {
  enabled = true

  servers = ["$SERVER_PRIVATE_IP:4647"]

  host_volume "redis_data" {
    path      = "/opt/data/redis"
    read_only = false
  }
}

plugin "docker" {
  config {
    volumes {
      enabled      = true
      selinuxlabel = "z"
    }
  }
}

Create a nomad service file

$ sudo touch /etc/systemd/system/nomad.service

Add this configuration to the nomad service file /etc/systemd/system/nomad.service

[Unit]
Description=Nomad
Documentation=https://nomadproject.io/docs/

[Service]
ExecStart=/usr/local/bin/nomad agent -config /etc/nomad.d
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target

Start nomad

sudo systemctl enable nomad
sudo systemctl start nomad
sudo systemctl status nomad

CNI Plugins in Nomad

Nomad uses CNI plugins when bridge networking is used. To install CNI plugins:

$ curl -L -o cni-plugins.tgz https://github.com/containernetworking/plugins/releases/download/v0.8.0/cni-plugins-linux-amd64-v0.8.0.tgz
$ mkdir -p /opt/cni/bin
$ tar -C /opt/cni/bin -xzf cni-plugins.tgz

job "clivern" {
  datacenters = ["dc1"]

  group "services" {

    network {

      port "toad0_srv" {
        static = 8080
      }

      port "toad1_srv" {
        static = 8081
      }
    }

    task "toad0" {
      driver = "docker"

      config {
        image = "clivern/toad:release-0.2.4"

        labels = {
          "com.clivern.service" = "toad"
          "com.clivern.service_type" = "web"
        }

        ports = ["toad0_srv"]

        command = "./toad"

        args = [
          "--port",
          "${NOMAD_PORT_toad0_srv}",
        ]
      }

      env = {
        IS_STATEFUL    = "false",
        TOAD0_ADDR     = "${NOMAD_HOST_ADDR_toad0_srv}",
        TOAD1_ADDR     = "${NOMAD_HOST_ADDR_toad1_srv}",
      }

      resources {
        network {
          mbits = 10
        }
      }
    }

    task "toad1" {
      driver = "docker"

      config {
        image = "clivern/toad:release-0.2.3"

        labels = {
          "com.clivern.service" = "toad"
          "com.clivern.service_type" = "web"
        }

        ports = ["toad1_srv"]

        command = "./toad"

        args = [
          "--port",
          "${NOMAD_PORT_toad1_srv}",
        ]
      }

      env = {
        IS_STATEFUL    = "false",
        TOAD0_ADDR     = "${NOMAD_HOST_ADDR_toad0_srv}",
        TOAD1_ADDR     = "${NOMAD_HOST_ADDR_toad1_srv}",
      }

      resources {
        network {
          mbits = 10
        }
      }
    }
  }
}

References:

How to Backup your Applications Data to S3 with Walrus

Clivern — Sat, 13 Feb 2021 13:59:47 +0000

👋 there!

I have been working lately to build a reliable and lightweight system for automated backups. Finally it is released 2 weeks ago 🚀

Clivern / Walrus

🔥 Fast, Secure and Reliable System Backup, Set up in Minutes.

Walrus

Fast, Secure and Reliable System Backup, Set up in Minutes.

Dashboard Screenshots

Walrus is a fast, secure and reliable backup system suitable for modern infrastructure. With walrus, you can backup services like SQLite, MySQL, PostgreSQL, Redis, etcd or a complete directory with a short interval and low overhead. It supports AWS S3, digitalocean spaces and any S3-compatible object storage service.

Documentation

Deployment

Download the latest walrus binary. Make it executable from everywhere.

$ export WALRUS_LATEST_VERSION=$(curl --silent "https://api.github.com/repos/Clivern/Walrus/releases/latest" | jq '.tag_name' | sed -E 's/.*"([^"]+)".*/\1/' | tr -d v)

$ curl -sL https://github.com/Clivern/Walrus/releases/download/v{$WALRUS_LATEST_VERSION}/walrus_{$WALRUS_LATEST_VERSION}_Linux_x86_64.tar.gz | tar xz

Then install etcd cluster or single node, please refer to etcd docs or bin directory inside this repository.

Run Walrus Tower:

Create the tower configs file tower.config.yml from config.dist.yml. Something like the following:

Please make sure to update the…

View on GitHub

Deploy with Docker

Install docker & docker-compose

$ apt-get update
$ sudo apt install docker.io
$ sudo systemctl enable docker
$ sudo apt install docker-compose

Clone Walrus for docker-compose.yml file

$ git clone https://github.com/Clivern/Walrus.git walrus
$ cd walrus/deployment/docker-compose

Feel free to update walrus tower port, api.key and api.encryptionKey. make sure you also use these values in walrus agent config file since agents require the tower URL, tower API key and encryptionKey to be able to reach and communicate with walrus tower.

Run the tower and etcd. It is also recommended to run etcd anywhere where data loss is mitigated.

$ docker-compose up -d

Now tower should be running. User your server public IP and tower port configured before to open the dashboard and setup the admin account.

# To get the public IP
$ curl https://ipinfo.io/ip

In the host where backups have to take place, download walrus binary.

$ curl -sL https://github.com/Clivern/Walrus/releases/download/v0.1.6/walrus_0.1.6_Linux_x86_64.tar.gz | tar xz

Create agent config file. Don't forget to replace agent.tower configs with the tower URL, apiKey and encryptionKey, you can get these values from tower configs you created earlier.

# Agent configs
agent:
    # Env mode (dev or prod)
    mode: ${WALRUS_APP_MODE:-prod}
    # HTTP port
    port: ${WALRUS_API_PORT:-8001}
    # URL
    url: ${WALRUS_API_URL:-http://127.0.0.1:8001}
    # TLS configs
    tls:
        status: ${WALRUS_API_TLS_STATUS:-off}
        pemPath: ${WALRUS_API_TLS_PEMPATH:-cert/server.pem}
        keyPath: ${WALRUS_API_TLS_KEYPATH:-cert/server.key}

    # API Configs
    api:
        key: ${WALRUS_API_KEY:-56e1a911-cc64-44af-9c5d-8c7e72ec96a1}

    # Async Workers
    workers:
        # Queue max capacity
        buffer: ${WALRUS_WORKERS_CHAN_CAPACITY:-5000}
        # Number of concurrent workers
        count: ${WALRUS_WORKERS_COUNT:-4}

    # Tower Configs
    tower:
        url: ${WALRUS_TOWER_URL:-http://127.0.0.1:8000}
        # This must match the one defined in tower config file
        apiKey: ${WALRUS_TOWER_API_KEY:-6c68b836-6f8e-465e-b59f-89c1db53afca}
        # This must match the one defined in tower config file
        encryptionKey: ${WALRUS_ENCRYPTION_KEY:-B?E(H+Mb}
        # Time interval between agent ping checks
        pingInterval: ${WALRUS_CHECK_INTERVAL:-60}

    # Backup settings
    backup:
        tmpDir: ${WALRUS_BACKUP_TMP_DIR:-/tmp}

    # Log configs
    log:
        # Log level, it can be debug, info, warn, error, panic, fatal
        level: ${WALRUS_LOG_LEVEL:-info}
        # output can be stdout or abs path to log file /var/logs/walrus.log
        output: ${WALRUS_LOG_OUTPUT:-stdout}
        # Format can be json
        format: ${WALRUS_LOG_FORMAT:-json}

Then run the host agent.

$ walrus agent -c /path/to/agent.config.yml

If everything is right, you should be able to see the host shown in the tower dashboard with one active agent. You can create backup crons under that host and update s3 configs in settings tab.

How to Mock a RESTful Microservice

Clivern — Wed, 30 Dec 2020 23:33:42 +0000

👋 there!

Recently I created a simple API mocking service called Rhino. That was because sometimes i have to integrate with a RESTful web service which is not ready yet or requires a lot of effort to setup and fill with data or Need a backend for a single page application & backend not ready yet.

Clivern / Rhino

❄️ HTTP Mocking & Debugging Service.

Rhino

HTTP Mocking & Debugging Service

Rhino is an HTTP Mocking & Debugging Service. It enables easy mocking of any HTTP web service for testing and debugging purposes. Also it can simulate high latencies and failures to make sure your services have the capability to withstand and recover from failures. It supports cross-origin resource sharing (CORS) so it can be used as a backend for single page applications.

Documentation

Usage

Get the latest binary.

$ curl -sL https://github.com/Clivern/Rhino/releases/download/x.x.x/Rhino_x.x.x_OS_x86_64.tar.gz | tar xz

Create the config file config.prod.json

{
    "app": {
        "mode": "prod or dev"
        "port": "8080",
        "domain": "http://127.0.0.1:8080",
        "tls": {
            "status": "off",
            "pemPath": "/cert/server.pem",
            "keyPath": "/cert/server.key"
        }
    },
    "mock": [
        {
            "path": "/api/v2/service1/mock/:id",
            "request": {
                "method": "get"

…

View on GitHub

Rhino enables easy mocking of any HTTP web service for testing and debugging purposes. Also it can simulate high latencies and failures to make sure your services have the capability to withstand and recover from failures. It supports cross-origin resource sharing (CORS) so it can be used as a backend for single page applications.

In order to run Rhino locally:

You will have to download one of the binaries from this page https://github.com/Clivern/Rhino/releases and make it globally executable.
Create a config file

{
    "app": {
        "mode": "prod",
        "port": "8080",
        "domain": "http://127.0.0.1:8080",
        "tls": {
            "status": "off",
            "pemPath": "/cert/server.pem",
            "keyPath": "/cert/server.key"
        }
    },
    "mock": [
        {
            "path": "/api/v2/service1/item",
            "request": {
                "method": "get"
            },
            "response": {
                "statusCode": 200,
                "headers": [
                    {"key": "Content-Type", "value": "application/json"}
                ],
                "body": "{\"action\": \"list\"}"
            },
            "chaos": {
                "latency": "0s",
                "failRate": "0%"
            }
        },
        {
            "path": "/api/v2/service1/item",
            "request": {
                "method": "post"
            },
            "response": {
                "statusCode": 200,
                "headers": [
                    {"key": "Content-Type", "value": "application/json"}
                ],
                "body": "{\"action\": \"create\"}"
            },
            "chaos": {
                "latency": "0s",
                "failRate": "0%"
            }
        },
        {
            "path": "/api/v2/service1/item/:id",
            "request": {
                "method": "get"
            },
            "response": {
                "statusCode": 200,
                "headers": [
                    {"key": "Content-Type", "value": "application/json"}
                ],
                "body": "{\"id\": \":id\", \"action\": \"get one\"}"
            },
            "chaos": {
                "latency": "0s",
                "failRate": "0%"
            }
        },
        {
            "path": "/api/v2/service1/item/:id",
            "request": {
                "method": "put"
            },
            "response": {
                "statusCode": 200,
                "headers": [
                    {"key": "Content-Type", "value": "application/json"}
                ],
                "body": "{\"id\": \":id\", \"action\": \"update\"}"
            },
            "chaos": {
                "latency": "0s",
                "failRate": "0%"
            }
        },
        {
            "path": "/api/v2/service1/item/:id",
            "request": {
                "method": "delete"
            },
            "response": {
                "statusCode": 200,
                "headers": [
                    {"key": "Content-Type", "value": "application/json"}
                ],
                "body": "{\"id\": \":id\", \"action\": \"delete\"}"
            },
            "chaos": {
                "latency": "0s",
                "failRate": "0%"
            }
        }
    ],
    "debug": [
        {
            "path": "/api/debug",
            "chaos": {
                "latency": "0s",
                "failRate": "0%"
            }
        }
    ],
    "log": {
        "level": "info",
        "output": "stdout",
        "format": "json"
    }
}

The run rhino with that config file

$ rhino serve -c config.json

On a different terminal, let's call these mocked endpoints

➜  ~ curl -X POST "http://127.0.0.1:8080/api/v2/service1/item"
{"action": "create"}

➜  ~ curl -X GET "http://127.0.0.1:8080/api/v2/service1/item"
{"action": "list"}

➜  ~ curl -X GET "http://127.0.0.1:8080/api/v2/service1/item/1"
{"id": "1", "action": "get one"}

➜  ~ curl -X PUT "http://127.0.0.1:8080/api/v2/service1/item/1"
{"id": "1", "action": "update"}

➜  ~ curl -X DELETE "http://127.0.0.1:8080/api/v2/service1/item/1"
{"id": "1", "action": "delete"}

To run rhino with docker and docker compose:

Just close the repository and use either the simple or the advanced setup.

# Simple setup
$ git clone https://github.com/Clivern/Rhino.git
$ cd Rhino/deployment/basic/docker-compose
$ docker-compose up -d

# In case you want to visualize incoming requests with grafana
$ git clone https://github.com/Clivern/Rhino.git
$ cd Rhino/deployment/advanced/docker-compose
$ docker-compose up -d

Rhino supports a bunch of fake data flags that can be used in the endpoint definition

AnyOf: @fake(:anyof[A||B||C||D])
Latitude: @fake(:lat)
Longitude: @fake(:long)
CreditCardNumber: @fake(:cc_number)
CreditCardType: @fake(:cc_type)
Email: @fake(:email)
DomainName: @fake(:domain_name)
IPV4: @fake(:ipv4)
IPV6: @fake(:ipv6)
Password: @fake(:password)
PhoneNumber: @fake(:phone_number)
MacAddress: @fake(:mac_address)
URL: @fake(:url)
UserName: @fake(:username)
TollFreeNumber: @fake(:toll_free_number)
E164PhoneNumber: @fake(:e_164_phone_number)
TitleMale: @fake(:title_male)
TitleFemale: @fake(:title_female)
FirstName: @fake(:first_name)
FirstNameMale: @fake(:first_name_male)
FirstNameFemale: @fake(:first_name_female)
LastName: @fake(:last_name)
Name: @fake(:name)
UnixTime: @fake(:unix_time)
Date: @fake(:date)
Time: @fake(:time)
MonthName: @fake(:month_name)
Year: @fake(:year)
DayOfWeek: @fake(:day_of_week)
DayOfMonth: @fake(:day_of_month)
Timestamp: @fake(:timestamp)
Century: @fake(:century)
TimeZone: @fake(:timezone)
TimePeriod: @fake(:time_period)
Word: @fake(:word)
Sentence: @fake(:sentence)
Paragraph: @fake(:paragraph)
Currency: @fake(:currency)
Amount: @fake(:amount)
AmountWithCurrency: @fake(:amount_with_currency)
UUIDHypenated: @fake(:uuid_hyphenated)
UUID: @fake(:uuid_digit)

Happy coding everyone! ❤️

New Contributors Welcome / Open Source Project (Silverback)

Clivern — Sat, 28 Sep 2019 17:14:31 +0000

👋 there!

Silverback 🔥 is a free and open source project for web infrastructure, developer APIs, services and SaaS companies to set up their very own status page in minutes.

The project started a while ago and quite stable even the beta release will be on october 2019 🤘.

silverbackhq / silverback

🧬 A Status and Incident Communication Tool (Beta Version).

Silverback

A Status and Incident Communication Tool.

Silverback is a free and open source project for web infrastructure, developer APIs, services and SaaS companies to set up their very own status page in minutes.

Overview

Auto Installer.
User management with different roles.
Services and service groups statuses.
Uptime charts for services and service groups.
Incidents and incident updates management.
Metrics with third party integrations (NewRelic and more ...).
Subscriber notifications via email, SMS, customer endpoint, RSS feed and Atom Feed.
Asynchronous workers for subscriber notifications with ability to run any number to scale.
Realtime administration dashboard.
Multilingual.
Easy Theme System! Easily give silverback a complete makeover.
and much more ...

Installation

Versioning

For transparency into our release cycle and in striving to maintain backward compatibility, Silverback is maintained under the Semantic Versioning guidelines and release…

View on GitHub

🚀 Some of the project features quickly:

Auto Installer.
User management with different roles.
Services and service groups statuses.
Uptime charts for services and service groups.
Incidents and incident updates management.
Metrics with third party integrations (NewRelic and more ...).
Subscriber notifications via email, SMS, customer endpoint, RSS feed and Atom Feed.
Asynchronous workers for subscriber notifications with ability to run any number to scale.
Realtime administration dashboard.
Multilingual.
and much more ..

Screenshots or Demo 🤔

Well, You can run silverback on heroku for free!

Architecture

Contributors Wanted 😌

If you're interested in practising your Python (Django), JavaScript, MySQL and Asynchronous Programming (RabbitMQ & Redis) skills, you should jump right now!

Want to know more about a specific area, don't hesitate to ask. More than happy to do whatever I can to help get you ramped up and contributing ❤️

Happy coding everyone! ❤️