DEV Community: Loknath Dhar

SSL with Apache Solr

Loknath Dhar — Wed, 24 Aug 2022 09:27:00 +0000

Setting up SSL

As I am highly dependent on official documentation, I found that Apache Solr maintain an awesome documentation from which I learned a lot. Setting up SSL can be found here. I am writing this for easier step by step setup.

Happy Searching!

Generate certificate & key

From the official documentation, you can generate a self-signed certificate to test your setup. But for the trusted authority, generate a certificate and convert it to .pfx format, which is type PKCS12.

And before starting Solr, we need to configure zookeeper.

Configure Zookeeper

We must configure Solr cluster properties in Zookeeper so that Solr nodes know to communicate via SSL. After creating, setting up and starting Zookeeper, we need to setup urlScheme cluster-wide property to https before any Solr nodes start up.

For Unix, go to $solr_home and run,

server/scripts/cloud-scripts/zkcli.sh -zkhost server1:2181,server2:2181,server3:2181 -cmd clusterprop -name urlScheme -val https

For existing collections, we can update cluster properties by using the Collections API (CLUSTERPROP):

http://IP:8983/solr/admin/collections?action=CLUSTERPROP&name=urlScheme&val=https

This command only needs to be run on one node of the cluster, the change will apply to all nodes.

Configure Solr

At this point, I think we all have set up zookeeper properly, have certificates and key at hand and didn't start Solr. Now, let's set up system properties of Solr in $solr_home/bin/solr.in.sh or, /etc/default/solr.in.sh:

SOLR_SSL_ENABLED=true
SOLR_SSL_KEY_STORE=/etc/solr-ssl.keystore.p12    # define your store path
SOLR_SSL_KEY_STORE_PASSWORD=secret    # use your store password
SOLR_SSL_TRUST_STORE=/etc/solr-ssl.keystore.p12    # define your store path
SOLR_SSL_TRUST_STORE_PASSWORD=secret    # use your store password
SOLR_SSL_NEED_CLIENT_AUTH=false
SOLR_SSL_WANT_CLIENT_AUTH=false
SOLR_SSL_CHECK_PEER_NAME=true

Now, start Solr and it's working.

If you use curl, see the official documentation section on how to use curl in case of SSL.

Setting up SolrCloud for Production

Loknath Dhar — Thu, 28 Jul 2022 11:30:00 +0000

I wasn't familiar with SolrCloud when I started working with this. When I started to learn it and worked directly with it - I did so many mistakes. Even setting up SolrCloud for production was a hassle for me (I'm a slow learner). But overtime, I got a hang of it and I think writing an up-to-date guide to set up SolrCloud on production server is a good idea.

The official documentation is amazing enough and you should read it. This post is very straight forward. If you want any explanation and want to know/learn more, please consult official documentation.

Let's get started.

(I'm writing this guide with the experience of Linux servers)

To easily go to the installation directory, I put these on .bashrc:

export solr_home=/opt/solr
PATH=/opt/solr/bin:$PATH

export zookeeper_home=/opt/zookeeper
PATH=/opt/zookeeper/bin:$PATH

$ cd $zookeeper_home    # will take to /opt/zookeeper
$ cd $solr_home    # will take to /opt/solr

Note that, this solr_home and SOLR_HOME variable isn't same.

Zookeeper

To bring the SolrCloud into production, we need external zookeeper (not the embedded one) server to manage our configuration and coordination centrally. We're going to work with 3 servers. We will install zookeeper and Apache Solr into all servers with same configuration.

First, let's configure our Apache Zookeeper. Install desired Java version according to the official documentation. At the time of writing, the latest Zookeeper and Solr - both needs Java 11.

$ sudo apt install openjdk-11-jdk

# check java version
$ java -version

Download the latest stable version from the official website. Notice that we don't want the source (src) bundle, we need the binary (bin) version.

Installation and configuration

It's not recommended to work with them while on root. But I'm going to work as a root user. My installation directory will be under /opt.

$ tar -xvf zookeeper-*.tar.gz -C /opt

$ ln -s /opt/zookeeper-* /opt/zookeeper

We need to create a configuration file for zookeeper under $zookeepr_home/conf/. The file name will be zoo.cfg:

tickTime=2000
dataDir=/var/lib/zookeeper/data
dataLogDir=/var/lib/zookeeper/logs

clientPort=2181
4lw.commands.whitelist=mntr,conf,ruok

initLimit=5
syncLimit=2

# put IP addresses or host on the place of Server1, Server2, Server3
server.1=Server1:2888:3888
server.2=Server2:2888:3888
server.3=Server3:2888:3888

autopurge.snapRetainCount=3
autopurge.purgeInterval=1

We will create a zookeeper environment file in the same place of zoo.cfg, which is under $zookeeper_home/conf. The file name will be zookeeper-env.sh:

JAVA_HOME="/usr/lib/jvm/java-1.11.0-openjdk-amd64"
ZOO_LOG_DIR="/var/lib/zookeeper/logs"
ZOO_LOG4J_PROP="INFO,ROLLINGFILE"

# increaseing the file size limit to 50MiB
JVMFLAGS="$JVMFLAGS -Djute.maxbuffer=50000000"

Create directories defined on the configuration:

mkdir -p /var/lib/zookeeper/data
mkdir -p /var/lib/zookeeper/logs

Create a myid text file under /var/lib/zookeeper/datadirectory. Put the server id in that file with a single line. In case of server 2, the file will contain: 2

echo "2" >/var/lib/zookeeper/data/myid

Now you can start zookeeper whenever you want but it need to be started before Solr.

$ cd $zookeeper_home
$ bin/zkServer.sh start

Apache SolrCloud

Download latest Solr (bin version) on the server, move file to the /opt and extract it.

$ tar xzf solr-*.tgz solr-*/bin/install_solr_service.sh --strip-components=2

Install it under /opt.

$ sudo bash ./install_solr_service.sh solr-*.tgz
$ ln -s solr-*/ solr

Edit bin/solr.in.sh for some configuration. We can also set this system wide by creating a file under /etc/default.

#writing include file

SOLR_PID_DIR=/var/solr
SOLR_HOME=/var/solr/data

#LOG SETTINGS

LOG4J_PROPS=/var/solr/log4j2.xml
SOLR_LOGS_DIR=/var/solr/logs

SOLR_HEAP="1g"

SOLR_JAVA_HOME="/usr/lib/jvm/java-1.11.0-openjdk-amd64"
ZK_HOST="zk-server1:2181,zk-server2:2181,zk-server3:2181"
SOLR_LOG_LEVEL=INFO

# Data backup location for replication environment
SOLR_OPTS="$SOLR_OPTS -Dsolr.allowPaths=/mnt/solr_backup"

# for soft commits
SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=10000"
SOLR_HOST="zk-server-ip" # current server IP address

SOLR_JAVA_MEM="-Xms2g -Xmx2g"
ZK_CLIENT_TIMEOUT="30000"
SOLR_PORT=8983

# To make available on the public internet
SOLR_JETTY_HOST="0.0.0.0"

# set this up in case if you set up authentication. 
# By setting this, the script will run without error
SOLR_AUTH_TYPE="basic"
SOLR_AUTHENTICATION_OPTS="-Dbasicauth=username:password"

Create directories defined in the configuration:

$ mkdir -p /var/solr/data
$ mkdir -p /var/solr/logs

And it's done.

You have to configure all servers like this.

Start SolrCloud by using the script:

$ bin/solr start -c -p 8983 -s /var/solr/data -z zk1:2181,zk2:2181,zk3:2181 -force

To get help:

bin/solr start -help
bin/solr restart -help # you got the point how to get help

I hope you are able to get the SolrCloud running without any errors.

Use bin/solr script to interact with Solr and Zookeeper. To know more, use official documentation.

To create a collection:

$ bin/solr create_collection -c col_name -d _default -shards 1 -replicationFactor 3 -p 8983 -V -force

To delete a collection:

$ bin/solr delete -c col_name -deleteConfig true -p 8983 -V

That's all. I will try to update this post from time to time. I'm also planning to include some useful commands later.

Hope, things are working.

Happy searching!

Setting up Codeblocks with Freeglut on Linux

Loknath Dhar — Sat, 06 Feb 2021 05:17:19 +0000

In my Computer Graphics class, the teacher asked us to setup Codeblocks for the glut. She gave us an amazing tutorial on how to setup Codeblocks on windows but didn't provide any suggestion to set up on Linux. After some researching on google, I was able to setup Codeblocks with Freeglut (which is an alternative to Glut and available on Linux). This is a kind of writeup (or tutorial) on how I was able to setup Codeblocks with Freeglut.

This setup is tested and working on Pop os, as of writing, should be working with Ubuntu/Debian with the latest Codeblocks.

Also, I linked to the posts from where I collected these and setting up Codeblocks to work properly.

Installation Part

To work with all of these, of course, we have to install all the necessary files.

For Freeglut:

sudo apt install g++ freeglut3 freeglut3-dev

sudo apt install libxmu-dev libxi-dev

For Codeblocks:

sudo apt install codeblocks

For OpenGL:

sudo apt install build-essential libgl1-mesa-dev

sudo apt install libglew-dev libsdl2-dev libsdl2-image-dev libglm-dev libfreetype6-dev  # some libraries

Check OpenGL installation:

glxinfo | grep OpenGL

Setup Part

Open Codeblocks, go to settings > Global Variables.
Click new, it should look like this:

type freeglut, press ok and setup the rest like this:

Now close the Codeblocks.

Resting part is collected from here.

Go to /usr/share/codeblocks/templates, create the file freeglut.cbp and put the following code there:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<CodeBlocks_project_file>
    <FileVersion major="1" minor="4" />
    <Project>
        <Option title="freeglut" />
        <Option pch_mode="0" />
        <Option compiler="gcc" />
        <Build>
            <Target title="default">
                <Option output="freeglut.exe" />
                <Option type="0" />
                <Option compiler="gcc" />
                <Option includeInTargetAll="1" />
            </Target>
        </Build>
        <Compiler>
            <Add directory="$(#freeglut.include)" />
        </Compiler>
        <Linker>
            <Add library="freeglut" />
            <Add library="glu32" />
            <Add library="opengl32" />
            <Add library="winmm" />
            <Add library="gdi32" />
            <Add library="user32" />
            <Add library="kernel32" />
            <Add directory="$(#freeglut.lib)" />
        </Linker>
        <Unit filename="main.cpp">
            <Option compilerVar="CPP" />
            <Option target="default" />
        </Unit>
    </Project>
</CodeBlocks_project_file>

Save it.

Create a directory named freeglut on /usr/share/codeblocks/templates/wizard/. You will find another directory named glut there. Copy all the contents and paste it on the newly created freeglut directory.
Now go to the created freeglut directory, edit the wizard.script file and paste the following:

////////////////////////////////////////////////////////////////////////////////
//
// FreeGLUT project wizard
//
////////////////////////////////////////////////////////////////////////////////

// globals
FreeGlutPathDefault    <- _T("$(#freeglut)");
FreeGlutPathDefaultInc <- _T("$(#freeglut.include)");
FreeGlutPathDefaultLib <- _T("$(#freeglut.lib)");
FreeGlutPath <- _T("");

function BeginWizard()
{
    local intro_msg = _T("Welcome to the new FreeGLUT project wizard!\n\n" +
                         "This wizard will guide you to create a new project\n" +
                         "using the FreeGLUT OpenGL extensions.\n\n" +
                         "When you 're ready to proceed, please click \"Next\"...");

    local glutpath_descr = _T("Please select the location of FreeGLUT on your computer.\n" +
                              "This is the top-level folder where FreeGLUT was installed (unpacked).\n" +
                              "To help you, this folder must contain the subfolders\n" +
                              "\"include\" and \"lib\".");

    Wizard.AddInfoPage(_T("GlutIntro"), intro_msg);
    Wizard.AddProjectPathPage();
    if (PLATFORM == PLATFORM_MAC)
    {
        FreeGlutPathDefault="/System/Library/Frameworks/FreeGLUT.framework";
    }
    else
        Wizard.AddGenericSelectPathPage(_T("FreeGlutPath"), glutpath_descr, _T("Please select FreeGLUT's location:"), FreeGlutPathDefault);
    Wizard.AddCompilerPage(_T(""), _T("*"), true, true);
}

////////////////////////////////////////////////////////////////////////////////
// GLUT's path page
////////////////////////////////////////////////////////////////////////////////

function OnLeave_GlutPath(fwd)
{
    if (fwd)
    {
        local dir         = Wizard.GetTextControlValue(_T("txtFolder")); // txtFolder is the text control in GenericSelectPathPage
        local dir_nomacro = VerifyDirectory(dir);

        if (dir_nomacro.IsEmpty())
            return false;

        // verify include dependencies
        local dir_nomacro_inc = GetCompilerIncludeDir(dir, FreeGlutPathDefault, FreeGlutPathDefaultInc);
        if (dir_nomacro_inc.IsEmpty())
            return false;
        if (!VerifyFile(dir_nomacro_inc + wxFILE_SEP_PATH + _T("GL"), _T("freeglut.h"), _T("FreeGLUT's include"))) return false;

        // verify library dependencies
        local dir_nomacro_lib = GetCompilerLibDir(dir, FreeGlutPathDefault, FreeGlutPathDefaultLib);
        if (dir_nomacro_lib.IsEmpty())
            return false;

        if (PLATFORM == PLATFORM_MSW)
        {
            if (!VerifyLibFile(dir_nomacro_lib, _T("freeglut"), _T("FreeGLUT's"))) return false;
        }
        else
        {
            if (!VerifyLibFile(dir_nomacro_lib, _T("freeglut"), _T("FreeGLUT's"))) return false;
        }


        FreeGlutPath = dir; // Remember the original selection.

        local is_macro = _T("");

        // try to resolve the include directory as macro
        is_macro = GetCompilerIncludeMacro(dir, FreeGlutPathDefault, FreeGlutPathDefaultInc);
        if (is_macro.IsEmpty())
        {
            // not possible -> use the real inc path we had computed instead
            FreeGlutPathDefaultInc = dir_nomacro_inc;
        }

        // try to resolve the library directory as macro
        is_macro = GetCompilerLibMacro(dir, FreeGlutPathDefault, FreeGlutPathDefaultLib);
        if (is_macro.IsEmpty())
        {
            // not possible -> use the real lib path we had computed instead
            FreeGlutPathDefaultLib = dir_nomacro_lib;
        }
    }
    return true;
}

// return the files this project contains
function GetFilesDir()
{
    return _T("glut/files");
}

// setup the already created project
function SetupProject(project)
{
    // set project options
    if (PLATFORM != PLATFORM_MAC)
    {
        project.AddIncludeDir(FreeGlutPathDefaultInc);
        project.AddLibDir(FreeGlutPathDefaultLib);
    }

    // add link libraries
    if (PLATFORM == PLATFORM_MSW)
    {
        project.AddLinkLib(_T("freeglut"));
        project.AddLinkLib(_T("opengl32"));
        project.AddLinkLib(_T("glu32"));
        project.AddLinkLib(_T("winmm"));
        project.AddLinkLib(_T("gdi32"));
    }
    else if (PLATFORM == PLATFORM_MAC)
    {
        project.AddLinkerOption(_T("-framework GLUT"));
        project.AddLinkerOption(_T("-framework OpenGL"));

        project.AddLinkerOption(_T("-framework Cocoa")); // GLUT dependency
    }
    else
    {
        project.AddLinkLib(_T("glut"));
        project.AddLinkLib(_T("GL"));
        project.AddLinkLib(_T("GLU"));
        project.AddLinkLib(_T("Xxf86vm"));
    }

    // enable compiler warnings (project-wide)
    WarningsOn(project, Wizard.GetCompilerID());

    // Debug
    local target = project.GetBuildTarget(Wizard.GetDebugName());
    if (!IsNull(target))
    {
        target.SetTargetType(ttConsoleOnly); // ttConsoleOnly: console for debugging
        target.SetOutputFilename(Wizard.GetDebugOutputDir() + Wizard.GetProjectName() + DOT_EXT_EXECUTABLE);
        target.SetWorkingDir(FreeGlutPath + _T("/bin"));
        // enable generation of debugging symbols for target
        DebugSymbolsOn(target, Wizard.GetCompilerID());
    }

    // Release
    target = project.GetBuildTarget(Wizard.GetReleaseName());
    if (!IsNull(target))
    {
        target.SetTargetType(ttExecutable); // ttExecutable: no console
        target.SetOutputFilename(Wizard.GetReleaseOutputDir() + Wizard.GetProjectName() + DOT_EXT_EXECUTABLE);
        target.SetWorkingDir(FreeGlutPath + _T("/bin"));
        // enable optimizations for target
        OptimizationsOn(target, Wizard.GetCompilerID());
    }

    return true;
}

Save it.

Now on /usr/share/codeblocks/templates/wizard directory, there is a script named config.script and add the following line at the end of the file:

RegisterWizard(wizProject,     _T("freeglut"),     _T("FreeGLUT project"),      _T("2D/3D Graphics"));

Now go to this link, this is the Freeglut project's source files, download the latest stable:

Extract it.
Copy the contents of the extracted folder and paste it on the /usr/share/codeblocks/templates/wizard/freeglut directory. At the time of writing, it looks like this:

Codeblocks Setup Part

If everything goes well/correctly, Codeblocks should be open without warning/showing any errors.

Open Codeblocks. Go to File > New > Projects and this dialog should be showing with Freeglut Project:

Give a name of the project as you see fit:

Then, specify the location where the Freeglut is:

Leave the next dialog as it is (don't change anything):

At this moment your Codeblocks should be ready for the Freeglut projects.

Final Part

Just checking if things are working or not.

Go to sources, there should be a main.cpp file for test running:

Build and Run!

That's it.

Okay, that's all. Hope, everything works fine.
See ya later!

Walkthrough: Shodan.io [TryHackMe]

Loknath Dhar — Thu, 08 Oct 2020 08:08:44 +0000

Shodan.io is a search engine for the internet of things. There is a room about Shodan in Tryhackme and this walk-through is about that.

Due to the nature of Shodan and its scanning services, the answer is changing all the time. So keep trying.

Task 01

An autonomous system number (ASN) is a global identifier of a range of IP addresses. Basically, large companies like Google, Microsoft have their own ASN for all of the IP addresses they own.

To find out ASN, first, we can search their IP address: ping google.com

We can put the IP address into an ASN lookup tool such as ultralools/asninfo

[It is mentioned by David Paine (see on the comment) that the above link isn't working. You should check another which he suggested: DNSchecker]

Though, we can put the company name here to find the ASN, using the IP address seems the easiest one to me.

On shodan, we can search using the ASN filter. The filter is: ASN:[number]
here, the number is marked on the picture.

Task 02

What is Google's ASN number?

Ans: You know it now. ;)

When it was allocated?

Ans: Again, look at the details.

Where are most of the machines on this ASN number, physically in the world?

Ans: United States.

What is Google's top service across all their devices on this ASN?

Ans: SSH.

What SSH product does Google use?

Ans: OpenSSH.

What is Google's most used Google product, according to this search? Ignore the word "Google".

Ans: Cloud.

Task 03

Here is a list of filters for shodan:

product: product Name (ex: MySQL)
city
country
Geo (co-ordinates)
Hostname
net (based on IP/CIDR)
os (find operating systems)
port
before/after (time-frames)

Task 04

What is the top operating system for MYSQL servers in Google's ASN?

Ans: 5.6.40–84.0-log

What is the 2nd most popular country for MYSQL servers in Google's ASN?

Ans: Netherlands.

Under Google's ASN, which is more popular for Nginx, Hypertext Transfer Protocol, or Hypertext Transfer Protocol(s)?

Ans: HyperText Transfer Protocol.

Port 80 stands for HTTP; port 443 stands for HTTPs.

Under Google's ASN, what is the most popular city?

Ans: Mountain View. (Answer changes tiem to time)

Under Google's ASN in Los Angeles, what is the top operating system according to Shodan?

Ans: PAN-OS.

Using the top Webcam search from the explore page, does Google's ASN have any webcams? Yay / nay.

Ans: Nay.

Task 05

Shodan has a limit on the free user account. It has an API, use it for more searches.