DEV Community: Moshe Zada

A (secret) way to bypass Chrome's security/privacy error

Moshe Zada — Fri, 25 Sep 2020 18:34:54 +0000

You may notice that recently chrome adopted a strict approach for SSL certificates errors, and it's probably the right thing. Most internet users are should not get into websites with unverified certificates, but does it's the same for developers?
As developers, we sometimes start local servers with self-signed certificated which can be blocked by Google Chrome.

So how can I bypass the warning page and continue coding?

Simply type thisisunsafe while the browser is focused on that page. That's it 🙂

Elasticsearch Comrade Part #1 - SQL Queries

Moshe Zada — Mon, 30 Dec 2019 09:47:59 +0000

This is the first part of an Elasticsearch Comrade introduction series.
Elasticsearch Comrade is a management UI for common operations within elastic products.
In this post, I will cover the SQL Editor feature.

moshe / elasticsearch-comrade

Elasticsearch admin panel built for ops and monitoring

Elasticsearch Comrade

Elasticsearch Comrade is an open-source Elasticsearch admin and monitoring panel highly inspired by Cerebro. Elasticsearch Comrade built with python3, VueJS, Sanic, Vuetify2 and Cypress

Main Features

Elasticsearch version 5,6 and 7 support (tested against elasticsearch 7.7)
Multi cluster
Rest API with autocompletion, history, templates, and history
SQL editor (version 7 only)
Built for big clusters
Node statistics and monitoring
Manage aliases
Inspect running tasks
Manage index templates
Manage snapshots
And much more ...

Quickstart

Cluster dir definitaions

Comrade discovers clusters using the --clusters-dir param, docs are here, examples are here

Using docker (recommended)

docker run -v $PWD/clusters/:/app/comrade/clusters/ -it -p 8000:8000 mosheza/elasticsearch-comrade

Using the python package

pip install elasticsearch-comrade
comrade --clusters-dir clusters

Installation, configuration and next steps

Here

Roadmap

v1.1.0

Add python package
Reindex screen
Comrade dashboard

v1.2.0

Cluster settings screen
Evacuate node from shards
Add commrade version indicator to footer

v1.3.0

Beats screen
Threadpools screen

Screenshots

View on GitHub

Starting up Comrade server

You can install comrade with pip / docker / source
Follow the installation guide for more details

The SQL UI

Once you configured Comrade and started the server, click on the desired server. Next, open the navbar and click on 🔎SQL nav item

Sending queries and getting results

Now, The SQL editor will show up, and you free to send you queries 🙂

Some things you need to know before running queries:

SQL is only supported in ES version 7 and above
The SQL syntax is quite standard except a few changes
You can refer several indices at once using elasticsearch index expansion syntax, for instance (Notice the second line)

SELECT count(*), source_node.name
FROM ".monitoring-es-*"
WHERE type = 'index_recovery'
GROUP BY source_node.name
LIMIT 100

You can apply Elasticsearch query function like match, query, and score by using them as a function

SELECT source_node.name, SCORE() -- Add score to selected fields
from ".monitoring-es-7-2019.12.22"
where MATCH('type', 'index_recovery') -- Use match query
order by SCORE() -- order by score
LIMIT 100

Found this post useful? Add a ⭐️ to my Github project nor my twitter profile🙂

moshe / elasticsearch-comrade

Elasticsearch admin panel built for ops and monitoring

Elasticsearch Comrade

Elasticsearch Comrade is an open-source Elasticsearch admin and monitoring panel highly inspired by Cerebro. Elasticsearch Comrade built with python3, VueJS, Sanic, Vuetify2 and Cypress

Main Features

Elasticsearch version 5,6 and 7 support (tested against elasticsearch 7.7)
Multi cluster
Rest API with autocompletion, history, templates, and history
SQL editor (version 7 only)
Built for big clusters
Node statistics and monitoring
Manage aliases
Inspect running tasks
Manage index templates
Manage snapshots
And much more ...

Quickstart

Cluster dir definitaions

Comrade discovers clusters using the --clusters-dir param, docs are here, examples are here

Using docker (recommended)

docker run -v $PWD/clusters/:/app/comrade/clusters/ -it -p 8000:8000 mosheza/elasticsearch-comrade

Using the python package

pip install elasticsearch-comrade
comrade --clusters-dir clusters

Installation, configuration and next steps

Here

Roadmap

v1.1.0

Add python package
Reindex screen
Comrade dashboard

v1.2.0

Cluster settings screen
Evacuate node from shards
Add commrade version indicator to footer

v1.3.0

Beats screen
Threadpools screen

Screenshots

View on GitHub

Loop on fail - Small Pytest Trick to Boost Your Productivity

Moshe Zada — Wed, 04 Dec 2019 09:05:36 +0000

While reading pytest-xdist docs I came across a very interesting flag

-f, --looponfail    run tests in subprocess, wait for modified files
                    and re-run failing test set until all pass.

Amazing! especially if you have slow tests and you want faster feedback loop.
All you need to do is to install the pytest-xdist plugin
pip install pytest-xdist
and then run your tests with -f switch.
Now pytest will watch for file changes and run failing tests

Found this post useful? Add a star⭐️ to my Github project🙂

moshe / elasticsearch-comrade

Elasticsearch admin panel built for ops and monitoring

Elasticsearch Comrade

Elasticsearch Comrade is an open-source Elasticsearch admin and monitoring panel highly inspired by Cerebro. Elasticsearch Comrade built with python3, VueJS, Sanic, Vuetify2 and Cypress

Main Features

Elasticsearch version 5,6 and 7 support (tested against elasticsearch 7.7)
Multi cluster
Rest API with autocompletion, history, templates, and history
SQL editor (version 7 only)
Built for big clusters
Node statistics and monitoring
Manage aliases
Inspect running tasks
Manage index templates
Manage snapshots
And much more ...

Quickstart

Cluster dir definitaions

Comrade discovers clusters using the --clusters-dir param, docs are here, examples are here

Using docker (recommended)

docker run -v $PWD/clusters/:/app/comrade/clusters/ -it -p 8000:8000 mosheza/elasticsearch-comrade

Using the python package

pip install elasticsearch-comrade
comrade --clusters-dir clusters

Installation, configuration and next steps

Here

Roadmap

v1.1.0

Add python package
Reindex screen
Comrade dashboard

v1.2.0

Cluster settings screen
Evacuate node from shards
Add commrade version indicator to footer

v1.3.0

Beats screen
Threadpools screen

Screenshots

View on GitHub

Implementing Access Control with Netlify Identity and Netlify Functions

Moshe Zada — Tue, 19 Nov 2019 13:06:14 +0000

Netlify and JAMStack becoming more popular these days. In this post, I will walk through the process of using Netlify Identity and using it in your Netlify functions.

Netlify Identity

From Netlify docs:
Netlify Identity service brings a full suite of authentication functionality, backed by the GoTrue API. This allows you to manage and authenticate users on your site or app, without requiring them to be users of Netlify or any other service. You can use this for gated content, site administration, and more.
So basically, Netlify Identity bring Authentication and user capabilities to your static site using JWT and some authentication providers like GitHub and Google

Enable Netlify Identity in Your Project

To enable Identity, select the identity tab and click Enable Identity

Embed Identity Widget in Your Site

For simplicity, I'm using netlify-identity-widget provided by Netflix in order to add login modal to my site.
Please follow the instruction from the GitHub page, Netlify's post, or from examples

Calling Netlify Functions With the Bearer Header

In order to make the function to know which user called the function, we need to pass the access-token we got from Netlify Identity to the function.
Create a file called netlify.js. This file will include two utility functions for calling your netlify functions - GET and POST.

async function getToken() {
  const currentUser = netlifyIdentity.currentUser()
  if (!currentUser) {
    return ''
  }
  // fetchs new JWT token only if expired
  await currentUser.jwt()
  return currentUser.token.access_token
}

export async function GET(api) {
  const token = await getToken()
  return (await fetch(`/.netlify/functions${api}`, {
    headers: { Authorization: `Bearer ${token}` }
  })).json()
}

export async function POST(api, body) {
  const token = await getToken()
  return (await fetch(`/.netlify/functions${api}`, {
    method: 'POST',
    headers: {
      Accept: 'application/json',
      'Content-Type': 'application/json',
      Authorization: `Bearer ${token}`
    },
    body: JSON.stringify(body)
  })).json()
}

And use it in your frontend app when you need to call one of your functions

import { GET } from '../../netlify'
const data = await GET(`/getData`)

Getting the User Object Within Netlify Functions and Acting Upon

Now, after you passed the Authorization header, netlify will fetch the user object and put it inside the event context.
For example, read the roles of the user:


function getUserRoles(context) {
    const { clientContext } = context
    const userRoles = clientContext.user
      ? clientContext.user.app_metadata.roles
      : ["guest"]
}

export async function handle(event, context) {
    const userRoles = getUserRoles(context)

    // Return with 401 if user is not logged in
    if (userRoles.includes("guest")) {
        return {
            statusCode: 401,
            body: "unauthorized"
        }
    }

    let data;

    // Get data only admins should see
    if (userRoles.includes("admin")) {
        data = getAllStatuses();
    }

    // Get data only owners should see
    else if (userRoles.includes("owner")) {
        data = getStatus();
    }

    return {
        statusCode: 200,
        body: JSON.stringify({
            data,
        })
    }

Found this post useful? Add a star⭐️ to my Github project🙂

moshe / elasticsearch-comrade

Elasticsearch admin panel built for ops and monitoring

Elasticsearch Comrade

Elasticsearch Comrade is an open-source Elasticsearch admin and monitoring panel highly inspired by Cerebro. Elasticsearch Comrade built with python3, VueJS, Sanic, Vuetify2 and Cypress

Main Features

Elasticsearch version 5,6 and 7 support (tested against elasticsearch 7.7)
Multi cluster
Rest API with autocompletion, history, templates, and history
SQL editor (version 7 only)
Built for big clusters
Node statistics and monitoring
Manage aliases
Inspect running tasks
Manage index templates
Manage snapshots
And much more ...

Quickstart

Cluster dir definitaions

Comrade discovers clusters using the --clusters-dir param, docs are here, examples are here

Using docker (recommended)

docker run -v $PWD/clusters/:/app/comrade/clusters/ -it -p 8000:8000 mosheza/elasticsearch-comrade

Using the python package

pip install elasticsearch-comrade
comrade --clusters-dir clusters

Installation, configuration and next steps

Here

Roadmap

v1.1.0

Add python package
Reindex screen
Comrade dashboard

v1.2.0

Cluster settings screen
Evacuate node from shards
Add commrade version indicator to footer

v1.3.0

Beats screen
Threadpools screen

Screenshots

View on GitHub

🥳

Remove accidentally pushed file from a git repository history in 4 simple steps

Moshe Zada — Fri, 15 Nov 2019 12:29:01 +0000

Although git offers gitignore mechanism for ignoring files from being committed to git, you may find yourself stuck with a big file in your repo

Simply deleting the file with rm is not enough, git records all the history of all of your branches forever in the .git directory at the root of your project.

Step #1: Backup your repo!

Simply copy the project directory
cp -r myproject backup

Step #2: Identify the commit that introduced the new file

The easiest way is to look at the output of git log command, assuming you want to delete a file called client/public/favicons/red/hugefile.ova run

$ git log client/public/favicons/red/hugefile.ova
Sat Aug 17 19:16:17 2019 +0300 7ff66fa Add favicons  [Moshe Zada]

As you can see, it seems like commit 7ff66fa introduced the big file, let's rewrite the history!

Step #3: Go back in time

Now that we find the commit that we want to return back to, run

git rebase --interactive 7ff66fa~1

Right after running this command, your editor will open up with the commit history from 7ff66fa to the last commit, for example:

pick 7ff66fa Add favicons
pick 48d9cc0 Readme (#84)
pick b9f23fd GitBook: [master] 17 pages modified
pick 1e0f4f1 move to docs
pick 23adabf GitBook: [master] 4 pages modified
pick 237c792 revert readme
pick 9418698 Continue reload if had exception
pick 3723c3c Add Copy option to index
pick 1ff0ec1 Add filter for manage aliases
pick 8a8770d Better null message
pick 4d8b64c Subscribe to input change
pick 9bc31b6 Save more data for node stats
pick 546e10b Small fixes

# Rebase 5058e82..eafa06e onto 5058e82 (193 commands)
#
# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup <commit> = like "squash", but discard this commit's log message
# x, exec <command> = run command (the rest of the line) using shell
# b, break = stop here (continue rebase later with 'git rebase --continue')
# d, drop <commit> = remove commit
# l, label <label> = label current HEAD with a name
# t, reset <label> = reset HEAD to a label
# m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
# .       create a merge commit using the original merge commit's
# .       message (or the oneline, if no original merge commit was
# .       specified). Use -c <commit> to reword the commit message.
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out

go to the first line, the one that starts with your git commit and change the first word to the letter e - for edit (vimmiers: cwe<ESC>) and exit your editor (:wq)

Step #4: Rewrite the history and push your changes

Now git will start replaying the last commits and give you shell just before the bad commit
delete the file:
rm client/public/favicons/red/hugefile.ova
Add the change to git
git add client/public/favicons/red/hugefile.ova
Commit and continue the rebase
git commit --amend '-S' && git rebase --continue
Verify changes and push your changes

Thank Joe and Eugene for pointing git push --force should be used with caution
git push -f be carful with this. the -f stands for force. Be default git will complain if you try to re-write history, but you can force it to go anyway. If you are on a solo project this should be fine, but if you are working on a team make sure no one has pulled the branch and added commits on top of it You won't be doing them any favors.
Don't be afraid to use it. Just make sure you communicate with your team on what is being done.

git push --force

That's it 🙂

Found this post useful? Add a star⭐️ to my Github project🙂

moshe / elasticsearch-comrade

Elasticsearch admin panel built for ops and monitoring

Elasticsearch Comrade

Elasticsearch Comrade is an open-source Elasticsearch admin and monitoring panel highly inspired by Cerebro. Elasticsearch Comrade built with python3, VueJS, Sanic, Vuetify2 and Cypress

Main Features

Elasticsearch version 5,6 and 7 support (tested against elasticsearch 7.7)
Multi cluster
Rest API with autocompletion, history, templates, and history
SQL editor (version 7 only)
Built for big clusters
Node statistics and monitoring
Manage aliases
Inspect running tasks
Manage index templates
Manage snapshots
And much more ...

Quickstart

Cluster dir definitaions

Comrade discovers clusters using the --clusters-dir param, docs are here, examples are here

Using docker (recommended)

docker run -v $PWD/clusters/:/app/comrade/clusters/ -it -p 8000:8000 mosheza/elasticsearch-comrade

Using the python package

pip install elasticsearch-comrade
comrade --clusters-dir clusters

Installation, configuration and next steps

Here

Roadmap

v1.1.0

Add python package
Reindex screen
Comrade dashboard

v1.2.0

Cluster settings screen
Evacuate node from shards
Add commrade version indicator to footer

v1.3.0

Beats screen
Threadpools screen

Screenshots

View on GitHub

Loading CSVs to elasticsearch like a pro (no Logstash needed)

Moshe Zada — Mon, 09 Sep 2019 13:11:42 +0000

As an Elasticsearch user, I subscribed to elasticsearch groups and channels in many places (Twitter, Reddit, Facebook, StackOverflow) and I keep seeing people struggling to start with elasticsearch, especially loading bunch of CSV files into it.

Hey does anyone know how can i upload multiple CSV files to Kibana?

The first mistake as I see it is to use Logstash in order to load CSV files to elasticsearch.
I see Logstash as a streaming service to elasticsearch rather than a tool to load data to elasticsearch from a static source (like CSV).

Using Elasticsearch Loader to load CSVs to Elasticsearch

I wrote a wrapper on top of Elasticsearch's bulk API in order to reduce the barrier in loading files into Elasticsearch.
The usage simple as running a command from your terminal.

Install

pip install elasticsearch-loader

Plugins

There are several plugins like Parquet, Redis, and S3 as input sources, read more about them in the readme

Usage and examples

Load CSVs into elasticsearch running on localhost
elasticsearch_loader --index incidents csv file1.csv file2.csv

Load CSV from URL and specify id field
elasticsearch_loader --index data --id-field state csv https://raw.githubusercontent.com/jakevdp/data-USstates/master/state-abbrevs.csv

Read CSV data from stdin
generate_data | elasticsearch_loader --index data csv -

And much more, for more examples and documantaion please follow elasticsearch-loader's README file