DEV Community: Mallikarjun H T

Hear is an repo which provides memorey and engineering process to your AI, just clone the repo and install as a plugin. Call the /commands and the app will take care of an end to end process. I have used open source model and lucenedb as JAVA applicatio.

Mallikarjun H T — Fri, 09 Oct 2026 02:33:01 +0000

Mallikarjun H T

Oct 1

Agentic-SDD: Giving Claude Code Agents a Real Engineering Process

#claude #agents #agentskills #ai

6 min read

Hear is an repo which provides memorey and engineering process to your AI, just clone the repo and install as a plugin. Call the /commands and the app will take care of an end to end process. I have used open source model and lucenedb as JAVA applicatio.

Mallikarjun H T — Fri, 09 Oct 2026 02:28:00 +0000

Mallikarjun H T

Oct 1

Agentic-SDD: Giving Claude Code Agents a Real Engineering Process

#claude #agents #agentskills #ai

6 min read

Agentic-SDD: Giving Claude Code Agents a Real Engineering Process

Mallikarjun H T — Thu, 01 Oct 2026 07:02:16 +0000

Agentic-SDD: Giving Claude Code Agents a Real Engineering Process (and a Knowledge Base That Doesn't Depend on Anyone Else)

Ask an AI coding agent to "fix the bug" and it will fix a bug — usually the
one nearest the surface, in whatever file it opens first. It rarely stops to
ask whether the fix addresses the real requirement, whether a sibling caller
has the same problem, or whether a reviewer would actually sign off on it.
That's not a model failure so much as a missing process: humans don't skip
design and review because we're smarter in the moment, we skip it because
nobody built the discipline into the loop.

Agentic-SDD is a Claude Code plugin that tries to build that discipline
in — a six-stage pipeline (require → plan → analyze → implement → verify → fix) gated by a persistent on-disk status file instead of conversation
memory, backed by an 18-agent specialist roster, with its own bundled
knowledge-search engine so it doesn't depend on any other plugin to ground
its decisions in your actual codebase.

Repo: github.com/MallikarjunHt/agentic-sdd

The core idea: six gated stages, not a chat loop

A /sdd.require <feature-id> call doesn't just write code. It walks a real
feature through:

Require. A business-analyst agent turns a raw ticket/request into 1-spec.md — problem, proposed solution, explicit non-goals, risks, and Given/When/Then acceptance criteria. You approve it before anything else happens.
Plan. An architect agent turns the approved spec into 2-plan.md and 3-tasks.md — a concrete Definition of Done, a file map, and a numbered task list, each task tagged with a suggested owner from a 12-role specialist bench (Java, Angular, React, Python, UI, DevOps, QA, BA, DBA, Security, Architect, Senior Dev), scored against trigger keywords. You approve this too.
Analyze. A dedicated gap-analysis agent checks the plan against the spec's acceptance criteria and the real current codebase, before a single line of code is written. CRITICAL findings block progress until resolved or explicitly descoped.
Implement. Each task is implemented by its routed specialist, at a model tier (cheap/mid/expensive) scored from that task's own complexity — a one-line config change doesn't need the same model as a cross-cutting refactor.
Verify. Three separate agents check constitution conformance, quality/security, and test adequacy — plus a fourth, a dedicated devil's advocate, whose only job is to look for what a checklist-shaped review structurally can't catch: concurrency, migration/rollback risk, backward compatibility. A PASS proposes a living-documentation diff, reviewed in the normal PR, never committed silently.
Fix. If verify fails, a fixer agent gets exactly three attempts before it has to stop and recommend a plan change instead of looping forever — a real circuit breaker, not an infinite retry.

Every stage after require reads and writes a status.json per feature, so
a crashed session resumes instead of restarting, and stages genuinely cannot
run out of order. That one design choice — a file as the gate, not the
model's own sense of "did I already do this?" — is doing most of the
reliability work here.

The profile system: one plugin, any repo

Rather than hardcoding one tech stack's conventions, /sdd.init writes a
specs/constitution.md grounded in that specific target repo's real
conventions — build tool, test framework, lint command, forbidden/required
patterns — not a generic textbook standard. A repo's own detected config
always wins over the constitution file when they disagree. One plugin
install, many target repos, each honestly represented.

The part I almost shipped wrong: knowledge grounding

The original design had every stage's "gather context" step call out to an
MCP tool contract — any server exposing a knowledge_search-shaped tool
name could plug in. Clean, decoupled, and wrong for the actual goal: if
someone uninstalls whatever server was providing that tool, the plugin loses
a capability it should always have.

So I vendored the actual search engine — a small Lucene-based Java CLI, BM25
full-text search with an optional ONNX vector-embedding upgrade — directly
into the plugin, under its own renamed package so it carries no trace of
where it came from. A few lessons from doing that for real, not
hypothetically:

I almost committed a 150MB jar. GitHub rejects pushes over 100MB per file. The fix was obvious in retrospect: vendor the source (a few hundred KB), build the jar locally with Maven on first use, and gitignore the build output. The plugin repo stays small; the binary gets built once per machine.
The embedding model is optional by design, not by accident. Full semantic search needs a ~90MB ONNX model I also didn't want to ship by default. The engine already had a clean fallback — no model vendored means BM25-only, reported plainly by its own doctor command — so the default install just... works, lexical search only, with vector search as an opt-in upgrade documented for later.
A relative default path is a trap. The CLI's --index-dir and --models-dir flags default to cwd-relative paths. I'd already seen this exact class of bug once before (a tool silently reporting "no embedding model" because it was invoked from the wrong working directory, not because the model was actually missing) — so every call from inside the plugin passes both as absolute paths. Boring, but it's the difference between "works on my machine" and "works."

Proof, not a pitch

I ran the full pipeline end-to-end against a real Dependabot CVE alert in a
production Java/npm monorepo — a transitive yaml dependency vulnerable to
stack-overflow on deeply nested input. All six stages ran for real: the spec
captured the actual CVE and affected manifest, the plan scoped it to a
one-line npm override (no functional changes, matching the kind of
discipline a dependency-only fix needs), analyze confirmed yaml was
transitive rather than direct (so an override, not a direct bump, was the
right mechanism), implement regenerated the lockfile properly instead of
hand-editing it, and verify confirmed the resolved version actually cleared
the vulnerable range — straight from the regenerated lockfile, since the
environment's own vulnerability-feed proxy wasn't reachable that day. Two
real commits, nothing fabricated, nothing silently skipped.

What's still a known gap

Stated plainly, because a tool that hides its own limitations is worse than
one that states them:

No automated validation of the plugin's own prompts/templates — there's no compiler for a markdown instruction file. A change is proven by running it against a real feature, not by a test suite.
No cross-repo coordination — each install targets one repo via its own config. A feature spanning two repos needs two separate, manually coordinated runs.
The wiki-mirror feature (a page-per-stage sync to Confluence or similar) is the one remaining capability that depends on an external MCP server being connected, and degrades to "local artifact only" when it isn't — deliberately, since a wiki is a genuinely external system, not something a plugin should vendor a copy of.

If you're building something similar — or just tired of watching an agent
confidently fix the wrong thing — the repo's up, MIT licensed, and the
pipeline is designed to be read, not just run:
github.com/MallikarjunHt/agentic-sdd.

No automated validation of the plugin's own prompts/templates — there's no compiler for a markdown instruction file. A change is proven by running it against a real feature, not by a test suite.
No cross-repo coordination — each install targets one repo via its own config. A feature spanning two repos needs two separate, manually coordinated runs.
The wiki-mirror feature (a page-per-stage sync to Confluence or similar) is the one remaining capability that depends on an external MCP server being connected, and degrades to "local artifact only" when it isn't — deliberately, since a wiki is a genuinely external system, not something a plugin should vendor a copy of.

Giving Claude Code a Real Engineering Process

Mallikarjun H T — Thu, 17 Sep 2026 04:44:26 +0000

Wizard: Giving Claude Code a Real Engineering Process

Ask an AI coding agent to "fix the bug" and it will fix a bug — usually the
one nearest the surface, in whatever file it opens first. It rarely stops to
ask whether the fix addresses the actual requirement, whether a sibling
caller has the same problem, or whether the change is something a reviewer
would sign off on before it ships. That's not a model failure so much as a
missing process: humans don't skip design and review because we're smarter
than that in the moment, we skip it because nobody built the discipline into
the loop.

Wizard is an attempt to build that discipline into the loop — a Claude Code
plugin that turns a single free-text request into a small software team's
worth of process: a business analyst who writes the spec, an architect who
turns it into a task-owned plan, a bench of specialists who each implement
their piece, and a QA lead who renders a final verdict against the plan's
own definition of done. All of it running as Claude Code agents, all of it
gated by you at the two points that matter most.

The core idea: spec → plan → code → review, with real stop points

Every plain /wiz <task> invocation runs through four phases:

Spec (Blake). A business-analyst agent reads the task, queries whatever knowledge sources are available about your codebase, and writes a spec.md — problem, proposed solution, explicit non-goals, risks, and Given/When/Then acceptance criteria. No implementation detail yet.
Gate 1. You read the spec and approve, ask for changes, or cancel. The orchestrator does not proceed without an explicit answer — this isn't a rhetorical "does this look right?" the model can talk past, it's a real stop.
Plan (Archie). An architect agent turns the approved spec into plan.md: a concrete Definition of Done, a file map, and a numbered task list where each task carries a suggested owner — one of twelve specialist roles, scored against trigger keywords (.java/spring → the Java developer, CSS/accessibility → the UI developer, CVE/ OWASP → security, and so on). The orchestrator re-scores every task itself afterward, deterministically, so owner assignment never depends on how convincingly the architect argued for a role.
Gate 2. Same approval mechanism, now against the plan's Definition of Done and finalized task list.
Code phase. Each task gets its own specialist, spawned with the full spec, the full plan, and one explicit instruction: if the task contradicts the plan or the actual codebase, don't improvise — report SPEC_DRIFT and let a human decide whether the plan or the spec needs to change. Silent improvisation is exactly the failure mode this whole pipeline exists to prevent.
Final QA (Quinn). Once every task is checked off, a QA agent independently re-verifies the Definition of Done against the actual repository state — not the task list's checkmarks — and renders PASS, CONCERNS, or FAIL. A FAIL reopens exactly the failing tasks; it doesn't restart the whole pipeline.

For a genuine one-liner, /wiz quick <task> skips all of this: one
knowledge query, one complexity estimate, one specialist, done. The
mandatory pipeline is the default because most tasks aren't one-liners, even
when they look like one at 9pm.

Twelve specialists, one routing table

Wizard doesn't have a single "coder" persona — it has an architect, a senior
developer as the generalist fallback, and dedicated specialists for Java,
Angular, React, Python, UI/CSS, DevOps, QA, business analysis, database
work, and security. Routing is keyword-scored against one fixed trigger
table, shared by the quick-mode router and the plan phase's task-owner
assignment — a task never gets routed one way in one mode and a different
way in another:

Role	Triggers
`architect`	design, architecture, how should we, system, trade-off, ADR, strategy
`seniorDev`	implement, refactor, review, PR, pull request, code review
`javaDev`	`.java`, spring, springboot, maven, gradle, hibernate, JPA, `@Bean`, `@Service`
`angularDev`	`.component.ts`, `.module.ts`, angular, `@Component`, `@NgModule`, rxjs
`uiDev`	CSS, SCSS, HTML, template, styling, layout, accessibility, Tailwind
`reactDev`	react, jsx, tsx, `useState`, `useEffect`, Next.js, hook
`pythonDev`	`.py`, django, fastapi, flask, pytest, pandas, pip
`devops`	Dockerfile, k8s, pipeline, CI, CD, Helm, Terraform, kubectl, yaml
`qa`	test, spec, assertion, bug, regression, JUnit, Jest, coverage
`ba`	requirement, user story, acceptance criteria, business rule, stakeholder
`dba`	SQL, schema, migration, index, stored procedure, Flyway, Liquibase
`securityEng`	vulnerability, CVE, OWASP, auth, token, injection, XSS, pentest

Ties resolve architect > seniorDev > any specialist; nothing matching
falls back to seniorDev. It's a small detail, but it's the difference
between "an AI wrote this" and "the person who actually knows Spring Boot
wrote this."

Context that isn't just "paste the file"

Before any routing decision, Wizard queries a layered knowledge pipeline:
a pure-Java unified index (LuceneDb) covering both documentation and code
structure, falling back to a graphify code graph if the index hasn't been
built yet, plus a relational knowledge graph for typed entity relationships,
merged and weighted into a single context block. If none of that is
available, it falls back to a plain grep. The point isn't sophistication for
its own sake — it's that a spec written with zero awareness of the existing
codebase is a spec that will drift the moment implementation starts.

The merge logic itself is a short, readable cascade — each source is tried
in priority order, and only contributes if it actually returns something:

def retrieve(task_summary, working_dir, config=None):
    """Run all enabled knowledge sources and return a merged context string."""
    cfg = config or {}
    wd = Path(working_dir)
    budget = int(cfg.get("maxContextTokens", _DEFAULT_BUDGET))
    weights = {**_DEFAULT_WEIGHTS, **cfg.get("weights", {})}
    parts = []  # (source_name, content, weight)

    # Source 0: LuceneDb — preferred when built, no Node/npm/network needed.
    lucenedb_used = False
    if cfg.get("lucenedb", {}).get("enabled", True) and _lucenedb.available():
        result = _lucenedb.query(task_summary, lucenedb_index_path, limit=10)
        if result:
            parts.append(("lucenedb", result, weights.get("graph", 0.5) + weights.get("qmd", 0.3)))
            lucenedb_used = True

    # Source 1: graphify code graph — automatic fallback if LuceneDb isn't built.
    if not lucenedb_used and _graphify.available(graph_path):
        graph_budget = int(budget * weights["graph"])
        result = _graphify.query(task_summary, graph_path, budget=max(500, graph_budget))
        if result:
            parts.append(("graphify", result, weights["graph"]))

    # Source 2: relational knowledge graph — typed entity links.
    if rel_cfg.get("enabled") and _relational.available(rel_path):
        rel_budget = int(budget * weights["relational"])
        edges = _relational.query(_extract_terms(task_summary), rel_path, limit=10)
        formatted = _relational.format_results(edges, budget_chars=rel_budget)
        if formatted:
            parts.append(("knowledge-graph", formatted, weights["relational"]))

    # Last resort: plain grep, no index required at all.
    if not parts:
        grep_results = _grep_fallback(task_summary, wd, cfg.get("fallbackGlob", "**/*.{md,txt,java,ts,py}"))
        if grep_results:
            parts.append(("grep", grep_results, 1.0))

    if not parts:
        return ""
    return "\n\n".join(
        f'<knowledge_context source="{source}">\n{content}\n</knowledge_context>'
        for source, content, _ in parts
    )

No index built yet on a brand-new checkout? You still get grep results.
LuceneDb built? It wins by default weight. Nothing ever hard-fails just
because one source isn't available.

A case study in eating your own dog food

This plugin was itself repackaged out of an older, looser setup: a skill
file under ~/.claude/skills/wizard, a folder of agent definitions, and a
standalone Python wizard repo that supplied the knowledge-retrieval code.
The repackaging was supposed to make the plugin fully self-contained —
install it, and it runs, no sibling repo required.

It didn't, quite. A routing exercise (asking Wizard itself, in quick mode,
whether the new plugin would work without the old repo present) turned up
three separate breakages, each more interesting than the last:

A stale absolute path. The plugin's knowledge-query step had a path hardcoded that didn't exist on the machine in question at all, because the source repo had since moved:

  - sys.path.insert(0, 'C:/projects/wizard')
  + sys.path.insert(0, r'${CLAUDE_PLUGIN_ROOT}/scripts/wizard')

Wrong in two ways at once: pointing outside the plugin, and pointing at
a location that was already gone. ${CLAUDE_PLUGIN_ROOT} — the same
variable the plugin's other scripts already used correctly — resolves
inside the plugin's own install directory, wherever that happens to be.

An install script referencing the wrong folder name. The skill had been renamed from skills/wizard/ to skills/wiz/ during the repackaging, but install.sh still copied from the old name:

  - cp "$REPO_DIR/skills/wizard/SKILL.md" "$SKILLS_DIR/wizard/SKILL.md"
  + cp "$REPO_DIR/skills/wiz/SKILL.md" "$SKILLS_DIR/wizard/SKILL.md"

It would have failed the moment anyone actually ran it. The same script
also checked for top-level knowledge/, retrieval/, wizard/
directories that no longer existed at that level — they'd been nested
under scripts/wizard/ during the repackaging:

  - if [ -d "$REPO_DIR/knowledge" ] && [ -d "$REPO_DIR/retrieval" ] && [ -d "$REPO_DIR/wizard" ]; then
  -   ln -sfn "$REPO_DIR/knowledge" "$WIZARD_LIB_DIR/knowledge"
  -   ln -sfn "$REPO_DIR/retrieval" "$WIZARD_LIB_DIR/retrieval"
  -   ln -sfn "$REPO_DIR/wizard" "$WIZARD_LIB_DIR/wizard"
  + if [ -d "$REPO_DIR/scripts/wizard/config" ] && [ -d "$REPO_DIR/scripts/wizard/retrieval" ] && [ -d "$REPO_DIR/scripts/wizard/wizard" ]; then
  +   ln -sfn "$REPO_DIR/scripts/wizard/config" "$WIZARD_LIB_DIR/config"
  +   ln -sfn "$REPO_DIR/scripts/wizard/retrieval" "$WIZARD_LIB_DIR/retrieval"
  +   ln -sfn "$REPO_DIR/scripts/wizard/wizard" "$WIZARD_LIB_DIR/wizard"

Silent, not loud: the original condition just evaluated false and skipped
the block, so install.sh never actually crashed — it just quietly never
linked the knowledge modules it advertised, which is arguably worse than
a hard failure.

A missing package, not just a missing path. The deepest issue: retrieval/unified.py imports retrieval/relational.py unconditionally at module load —

  from . import graphify as _graphify
  from . import lucenedb as _lucenedb
  from . import relational as _relational

— and relational.py needs a KnowledgeGraph class:

  try:
      from ..knowledge.graph import KnowledgeGraph
  except (ImportError, ValueError):
      from knowledge.graph import KnowledgeGraph

Both branches assume a knowledge package sits somewhere reachable — as
a parent-package sibling, or as a top-level package on sys.path. Neither
was true: the entire knowledge/ directory (graph.py, extractor.py,
build.py, five files total) had simply never been copied into the
plugin's bundled Python tree during the carve-out. Fixing the path alone
would have traded one ModuleNotFoundError for another the instant
anything tried to import retrieval.unified. The fix required going back
to the original repo, confirming the missing package had no further
hidden dependencies of its own (a quick grep for its imports — only
pathlib, re, json, dataclasses, and its own relative imports came
back), and copying it in as a sibling of the packages that were already
there:

  scripts/wizard/
  ├── config/
  ├── retrieval/
  ├── wizard/
  └── knowledge/   <- added: graph.py, extractor.py, build.py, __init__.py, __main__.py

The lesson generalizes past this one plugin: "it imports correctly" and "the
path string points somewhere plausible" are different claims, and only one
of them is verifiable by reading the text of the code. The other one you
have to actually trace, import by import, or run.

Getting started

claude --plugin-dir /path/to/wizard-plugin
/wizard:wiz setup

setup walks through working directory, model tier preferences, and which
specialist roles you want enabled. After that:

/wizard:wiz <task>          # the full spec -> plan -> code -> review pipeline
/wizard:wiz quick <task>    # skip the gates for a genuine one-liner
/wizard:wiz knowledge status

Runtime configuration lives at ~/.wizard/config.json, outside the plugin
itself, so upgrading the plugin never touches your working-directory or
model-tier preferences.

Why bother with all this ceremony

Because the ceremony is the point. A model that writes code without a spec
will happily write code that solves the wrong problem, quickly and
confidently. A model that writes code without a plan will happily improvise
around a codebase constraint it hasn't actually checked. A model that grades
its own homework will mark it correct. Wizard doesn't make Claude smarter —
it makes the process harder to shortcut, which turns out to matter more.

Lucene V/S KQL

Mallikarjun H T — Thu, 01 May 2025 14:20:10 +0000

A Tour of KQL vs Lucene

Elasticsearch is a search engine, and Kibana can be used to search documents in Elasticsearch.
A search is executed by sending a query to Elasticsearch. A query can answer many different types of questions.

For example:

Who are the customers with the first name Edison?
What are the names of customers in India?
Are there any orders for Men’s Clothing OR Electronics section?

Should I use KQL or Lucene in the Kibana query bar?

It Depends, Both KQL and Lucine have a quite a lot in common.
I recoment to start with KQL then switch to Lucine for Specific Aggrigation use Case.

Learn By Executing.

in order to learn use the Free trial or download the pre prepared data sets to run locally if cloud trial expires.

KQL Features.

once you load your data you can use kibana to explore and learn about the features.

NOTE: KQL is case in sencitive and Lucine operaters are All Caps only

Autocomplete - KQL
Full Testsearch - KQL and Lucine

Exact Match

You can search for documents that contain an exact term in a provided field of data type keyword(opens in a new tab), which will not be analyzed. Not analyzed strings are case sensitive.

Example: Customer_first_name.keyword: Elyssa

Note: keyword is a type that only supports Exact search, if not given while searching then anything close to the word is returned.

Phrase search

To search text fields where the search terms are in the order provided, surround the value in double quotation marks, as follows:

Example: “Elyssa Underwood” - this is all field search
Example: Customer_full_name: “Elyssa Underwood” - this ia field specific search

Fuzzy Search only available in Lucine

arjun~1 - will match any word with 1ch differance, the number determines the no of charecters that can differ
arj*n - whild charecter search one or more charecters can occur in between arj and n
arj?n - single charecter only

NOTE: bad idea to use ** or ?* as this can cause perfomance issues.
to allow change the settings in elastic search.

Regex

A regular expression is a way to match patterns in data using placeholder characters called operators. It returns documents that contain terms matching a regular expression. Syntax is based on the Lucene regular expression engine.

Example: /.+91[0-9]{9}./

this will match number with +91 xxx xxx xxx

Ranges

To search documents that contain terms within a provided range, use KQL’s range syntax. You can use the range syntax for string values, IP addresses, and timestamps.

KQL Supports the following syntax >, >=, <, and <=

Lucene supports bracketed range syntax. Ranges can be specified for date, numeric or string fields. Inclusive ranges are specified with square brackets [min TO max] and exclusive ranges with curly brackets {min TO max}.

 date:[2012-01-01 TO 2012-12-31]
 count:[1 TO 5]
 tag:{alpha TO omega}

## Proximity Search
Lucine Exclusive
Proximity search(opens in a new tab) allows the specified words to be further apart or in a different order, a proximity search allows us to specify a maximum edit distance of words in a phrase. The closer the text in a field is to the original order specified in the query string, the more relevant that document is considered to be

“open data”~1

The above example will find all the documents where “open data” is found as a phrase and also “open source data” is a hit where one more term is present in between open and data.

Boosting

Lucene

Use the boost operator ^ to make one term more relevant than another. For instance, if we want to find all documents about foxes, but we are especially interested in quick foxes:

quick^2 fox

The default boost value is 1, but can be any positive floating point number. Boosts between 0 and 1 reduce relevance.

Boosts can also be applied to phrases or to groups:

"john smith"^2 (foo bar)^4

Searching for IP addresses

KQL Lucene

Use "" around IP addresses while using Lucene syntax.
Lucene also supports CIDR notation.

Example: "192.168.03.01"

Nested fields

KQL

Querying nested fields requires a special syntax.

Example: user.names:{ first: "Alyssa" and last: "Underwood" }

Runtime fields

KQL Lucene

A runtime field is a field that is evaluated at query time.

KQL and syntex

Mallikarjun H T — Wed, 23 Apr 2025 14:07:07 +0000

Should I use KQL or Lucene in the Kibana query bar?

It depends! KQL and Lucene have quite a lot in common, but there are some differences. Let's explore some key-features one by one to find out the best use-case for each language.

For new users, we recommend starting with KQL(opens in a new tab) and switching to Lucene(opens in a new tab) if any particular feature is unavailable in KQL.

KQL

Kibana Query Language (KQL) is a simple text-based query language for filtering data.
KQL only filters data, and has no role in aggregating, transforming, or sorting data.

Semi-structured search

free text search with field-based search.

The semi-structured search will filter documents for matches, and only return matching documents.

Filter for documents where a field exists

To filter documents for which an indexed value exists for a given field, use the * operator.

For example, to filter for documents where the http.request.method field exists, use the following syntax:

http.request.method: *

This checks for any indexed value, including an empty string.

Filter for documents that match a value

Use KQL to filter for documents that match a specific number, text, date, or boolean value. For example, to filter for documents where the http.request.method is GET, use the following query:
http.request.method: GET

to search all fields for “Hello”, use the following:

Hello

To querying keyword, numeric, date, or boolean fields, the value must be an exact match, including punctuation and case.

To Search text field that has "null pointer" then querry using http.request.body.content: "null pointer" if the \" are not used then you will get all documents that has null pointer OR Pointer null. This is called term querry.

to search urls use \ or you will find syntax error OR try with ".
Example:

http.request.referrer: "https://example.com"  
http.request.referrer: https\://example.com

[!Note]
\():<>"* excape These Charecters with \

Filter for documents within a range

to search for all documents for which http.response.bytes is less than 10000, use the following syntax:
http.response.bytes < 10000

to search In Range use this
http.response.bytes > 10000 and http.response.bytes <= 20000

Filter for documents using wildcards

To search for documents matching a pattern, use the wildcard syntax. For example, to find documents where http.response.status_code begins with a 4, use the following syntax:

http.response.status_code: 4*
By default, leading wildcards are not allowed for performance reasons. You can modify this with the query:allowLeadingWildcards advanced setting.

[!Note]
Only * is currently supported. This matches zero or more characters.

Negating a query

use NOT in front of querry like:
NOT http.request.method: GET

Querying nested fields

Use . operater to query nested fields like:
user.names:{ first: "Alice" and last: "White" }

Installing Elastic Search 7.14 with docker

Mallikarjun H T — Tue, 10 Sep 2024 04:52:53 +0000

Welcome to the ultimate guide on Elastic Search and Kibana! If you're looking for a powerful, open-source tool that can help you search, analyze, and visualize your data like never before, then look no further. In this comprehensive guide, we'll take you through everything you need to know about Elastic Search and Kibana - from what they are and how they work together to their installation with Docker. Whether you're a developer or just getting started with these tools, get ready to learn everything there is to know about using Elastic Search and Kibana with Docker in this exciting blog post!

Introduction to Elasticsearch and Kibana

Elasticsearch is a powerful open source search and analytics engine that makes data easy to explore. Kibana is an open source data visualization platform that allows you to interact with your data through beautiful visualizations. In this guide, we'll show you how to use Docker to install and run Elasticsearch and Kibana on your local machine.

What is Elastic Search?

Elastic Search is a powerful, open source, distributed search and analytics engine. It is built on top of the Apache Lucene search library and supports full text search, as well as structured data search with SQL-like query syntax. Elastic Search can be used to power search for websites, applications, and enterprise search solutions. It is also commonly used for log analysis, monitoring, and security analytics.

What is Kibana?

Kibana is an open source data visualization plugin for the Elasticsearch stack. It provides a rich set of tools to help you visualize and explore your data in Elasticsearch. Kibana is a great way to get started with visualizing your data in Elasticsearch.

How to Install Elastic Search and Kibana with Docker

version: '2.2'
services:
  node01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.14.1
    container_name: node01
    environment:
      - node.name=node01
      - cluster.name=es-my-cluster
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
  kib01test:
    image: docker.elastic.co/kibana/kibana:7.14.1
    container_name : kib01test
    environment:
      - ELASTICSEARCH_HOSTS=http://node01:9200
    ports:
      - 5601:5601
    networks: 
      - elastic

volumes:
  data01:
    driver: local

networks:
  elastic:
    driver: bridge

ES - Node APIs

Mallikarjun H T — Tue, 13 Aug 2024 09:40:44 +0000

# If no filters are given, the default is to select all nodes
curl -X GET "localhost:9200/_nodes?pretty"
# Explicitly select all nodes
curl -X GET "localhost:9200/_nodes/_all?pretty"
# Select just the local node
curl -X GET "localhost:9200/_nodes/_local?pretty"
# Select the elected master node
curl -X GET "localhost:9200/_nodes/_master?pretty"
# Select nodes by name, which can include wildcards
curl -X GET "localhost:9200/_nodes/node_name_goes_here?pretty"
curl -X GET "localhost:9200/_nodes/node_name_goes_*?pretty"
# Select nodes by address, which can include wildcards
curl -X GET "localhost:9200/_nodes/10.0.0.3,10.0.0.4?pretty"
curl -X GET "localhost:9200/_nodes/10.0.0.*?pretty"
# Select nodes by role
curl -X GET "localhost:9200/_nodes/_all,master:false?pretty"
curl -X GET "localhost:9200/_nodes/data:true,ingest:true?pretty"
curl -X GET "localhost:9200/_nodes/coordinating_only:true?pretty"
# Select nodes by custom attribute (e.g. with something like `node.attr.rack: 2` in the configuration file)
curl -X GET "localhost:9200/_nodes/rack:2?pretty"
curl -X GET "localhost:9200/_nodes/ra*:2?pretty"
curl -X GET "localhost:9200/_nodes/ra*:2*?pretty"

ES - CAT APIs

Mallikarjun H T — Tue, 13 Aug 2024 09:38:43 +0000

cat APIs

Introduction

All the cat commands accept a query string parameter help to see all the headers and info they provide, and the /_cat command alone lists all the available commands.

Common parameters

Verbose - curl -X GET "localhost:9200/_cat/master?v&pretty"
Help - curl -X GET "localhost:9200/_cat/master?help&pretty"
Headers - curl -X GET "localhost:9200/_cat/nodes?h=ip,port,heapPercent,name&pretty"
Numeric formats - curl 'localhost:9200/_cat/indices?bytes=b' | sort -rnk8
Response as text, json, smile, yaml or cbor - curl 'localhost:9200/_cat/indices?format=json&pretty'
Sort - curl -X GET "localhost:9200/_cat/templates?v&s=order:desc,index_patterns&pretty"

cat aliases

curl -X GET "localhost:9200/_cat/aliases?v&pretty"

cat allocation

curl -X GET "localhost:9200/_cat/allocation?v&pretty"

cat count

curl -X GET "localhost:9200/_cat/count?v&pretty"

curl -X GET "localhost:9200/_cat/count/twitter?v&pretty"

cat fielddata

curl -X GET "localhost:9200/_cat/fielddata?v&pretty"

cat health

curl -X GET "localhost:9200/_cat/health?v&pretty"

cat indices

curl -X GET "localhost:9200/_cat/indices?pretty"

Apply Filter

curl -X GET "localhost:9200/_cat/indices/twi*?pretty"

cat un healthy index

indices which are not ready

curl -X GET "localhost:9200/_cat/indices?v&health=yellow&pretty"

sort by dock count

curl -X GET "localhost:9200/_cat/indices?v&s=docs.count:desc&pretty"

cat master

curl -X GET "localhost:9200/_cat/master?v&pretty"

cat nodes

curl -X GET "localhost:9200/_cat/nodes?v&pretty"

Documentation

cat pending tasks

curl -X GET "localhost:9200/_cat/pending_tasks?v&pretty"

cat plugins

curl -X GET "localhost:9200/_cat/plugins?v&s=component&h=name,component,version,description&pretty"

cat recovery

curl -X GET "localhost:9200/_cat/recovery?v&pretty"

cat repositories

curl -X GET "localhost:9200/_cat/repositories?v&pretty"

cat thread pool

curl -X GET "localhost:9200/_cat/thread_pool?pretty"

for field data visit Documentation

cat shards

curl -X GET "localhost:9200/_cat/shards?pretty"

Reasion for un assigned shared look Documentation

INDEX_CREATED | Unassigned as a result of an API creation of an index.

CLUSTER_RECOVERED | Unassigned as a result of a full cluster recovery.

INDEX_REOPENED | Unassigned as a result of opening a closed index.

DANGLING_INDEX_IMPORTED | Unassigned as a result of importing a dangling index.

NEW_INDEX_RESTORED | Unassigned as a result of restoring into a new index.

EXISTING_INDEX_RESTORED | Unassigned as a result of restoring into a closed index.

REPLICA_ADDED | Unassigned as a result of explicit addition of a replica.

ALLOCATION_FAILED | Unassigned as a result of a failed allocation of the shard.

NODE_LEFT | Unassigned as a result of the node hosting it leaving the cluster.

REROUTE_CANCELLED | Unassigned as a result of explicit cancel reroute command.

REINITIALIZED | When a shard moves from started back to initializing, for example, with shadow replicas.

REALLOCATED_REPLICA | A better replica location is identified and causes the existing replica allocation to be cancelled.

cat segments

this where your data is saved on the FS.

curl -X GET "localhost:9200/_cat/segments?v&pretty"

snapshots

curl -X GET "localhost:9200/_cat/snapshots/repo1?v&s=id&pretty"

templates

curl -X GET "localhost:9200/_cat/templates?v&s=name&pretty"

ES - Index API - part 4

Mallikarjun H T — Mon, 05 Aug 2024 08:04:39 +0000

Indices Stats

curl -X GET "localhost:9200/_stats?pretty"

Indices Segments

curl -X GET "localhost:9200/test/_segments?pretty"

curl -X GET "localhost:9200/test/_segments?verbose=true&pretty"

Indices Recovery

curl -X GET "localhost:9200/_recovery?human&pretty"

curl -X GET "localhost:9200/_recovery?human&detailed=true&pretty"

Indices Shard Stores

# return information of only index test
curl -X GET "localhost:9200/test/_shard_stores?pretty"
# return information of only test1 and test2 indices
curl -X GET "localhost:9200/test1,test2/_shard_stores?pretty"
# return information of all indices
curl -X GET "localhost:9200/_shard_stores?pretty"

Clear Cache

curl -X POST "localhost:9200/twitter/_cache/clear?pretty"

Flush

curl -X POST "localhost:9200/twitter/_flush?pretty"

Synced Flush

curl -X POST "localhost:9200/twitter/_flush/synced?pretty"

Refresh

curl -X POST "localhost:9200/twitter/_refresh?pretty"

Force Merge

curl -X POST "localhost:9200/twitter/_forcemerge?pretty"

ES Index API - part 3

Mallikarjun H T — Mon, 05 Aug 2024 07:47:49 +0000

PUT Mappings

create index with mappings

curl -X PUT "localhost:9200/twitter?pretty" -H 'Content-Type: application/json' -d'
{}'

curl -X PUT "localhost:9200/twitter/_mapping/_doc?pretty" -H 'Content-Type: application/json' -d'
{
  "properties": {
    "email": {
      "type": "keyword"
    }
  }
}'

update existing index

curl -X PUT "localhost:9200/my_index?pretty" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "_doc": {
      "properties": {
        "name": {
          "properties": {
            "first": {
              "type": "text"
            }
          }
        },
        "user_id": {
          "type": "keyword"
        }
      }
    }
  }
}'

curl -X PUT "localhost:9200/my_index/_mapping/_doc?pretty" -H 'Content-Type: application/json' -d'
{
  "properties": {
    "name": {
      "properties": {
        "last": { 
          "type": "text"
        }
      }
    },
    "user_id": {
      "type": "keyword",
      "ignore_above": 100 
    }
  }
}'

Get Mappings

curl -X GET "localhost:9200/twitter/_mapping/_doc?pretty"

GET field mapping

curl -X GET "localhost:9200/publications/_mapping/_doc/field/title?pretty"

ES Index API- part 2

Mallikarjun H T — Mon, 05 Aug 2024 07:33:31 +0000

Delete Index

curl -X DELETE "localhost:9200/twitter?pretty"

Get Index

this will return mappings and settings if index exist

curl -X GET "localhost:9200/twitter?pretty"

check if index exist

curl -I "localhost:9200/twitter?pretty"

Open / Close Index API

use this with caution - like updating or to prevent new document insertion - search will work but indexing will not work

curl -X POST "localhost:9200/my_index/_close?pretty"
curl -X POST "localhost:9200/my_index/_open?pretty"

Shrink Index

use this to reduce the number of shards for version < 7 it is 5 and above it is 1 shard.

Preparing an index for shrinking

curl -X PUT "localhost:9200/my_source_index/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index.routing.allocation.require._name": "shrink_node_name", 
    "index.blocks.write": true 
  }
}
'

Shrinking an index

curl -X POST "localhost:9200/my_source_index/_shrink/my_target_index?copy_settings=true&pretty" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index.routing.allocation.require._name": null, 
    "index.blocks.write": null 
  }
}
'

Shrinking with settings

curl -X POST "localhost:9200/my_source_index/_shrink/my_target_index?copy_settings=true&pretty" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index.number_of_replicas": 1,
    "index.number_of_shards": 1, 
    "index.codec": "best_compression" 
  },
  "aliases": {
    "my_search_indices": {}
  }
}
'

Rollover Index

Defining the new index

curl -X PUT "localhost:9200/logs-000001?pretty" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    "logs_write": {}
  }
}'

curl -X POST "localhost:9200/logs_write/_rollover?pretty" -H 'Content-Type: application/json' -d'
{
  "conditions" : {
    "max_age": "7d",
    "max_docs": 1000,
    "max_size": "5gb"
  },
  "settings": {
    "index.number_of_shards": 2
  }
}'

Dry run

curl -X PUT "localhost:9200/logs-000001?pretty" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    "logs_write": {}
  }
}'

curl -X POST "localhost:9200/logs_write/_rollover?dry_run&pretty" -H 'Content-Type: application/json' -d'
{
  "conditions" : {
    "max_age": "7d",
    "max_docs": 1000,
    "max_size": "5gb"
  }
}'