DEV Community: Mallikarjun H T

Giving Claude Code a Real Engineering Process

Mallikarjun H T — Thu, 17 Sep 2026 04:44:26 +0000

Wizard: Giving Claude Code a Real Engineering Process

Ask an AI coding agent to "fix the bug" and it will fix a bug — usually the
one nearest the surface, in whatever file it opens first. It rarely stops to
ask whether the fix addresses the actual requirement, whether a sibling
caller has the same problem, or whether the change is something a reviewer
would sign off on before it ships. That's not a model failure so much as a
missing process: humans don't skip design and review because we're smarter
than that in the moment, we skip it because nobody built the discipline into
the loop.

Wizard is an attempt to build that discipline into the loop — a Claude Code
plugin that turns a single free-text request into a small software team's
worth of process: a business analyst who writes the spec, an architect who
turns it into a task-owned plan, a bench of specialists who each implement
their piece, and a QA lead who renders a final verdict against the plan's
own definition of done. All of it running as Claude Code agents, all of it
gated by you at the two points that matter most.

The core idea: spec → plan → code → review, with real stop points

Every plain /wiz <task> invocation runs through four phases:

Spec (Blake). A business-analyst agent reads the task, queries whatever knowledge sources are available about your codebase, and writes a spec.md — problem, proposed solution, explicit non-goals, risks, and Given/When/Then acceptance criteria. No implementation detail yet.
Gate 1. You read the spec and approve, ask for changes, or cancel. The orchestrator does not proceed without an explicit answer — this isn't a rhetorical "does this look right?" the model can talk past, it's a real stop.
Plan (Archie). An architect agent turns the approved spec into plan.md: a concrete Definition of Done, a file map, and a numbered task list where each task carries a suggested owner — one of twelve specialist roles, scored against trigger keywords (.java/spring → the Java developer, CSS/accessibility → the UI developer, CVE/ OWASP → security, and so on). The orchestrator re-scores every task itself afterward, deterministically, so owner assignment never depends on how convincingly the architect argued for a role.
Gate 2. Same approval mechanism, now against the plan's Definition of Done and finalized task list.
Code phase. Each task gets its own specialist, spawned with the full spec, the full plan, and one explicit instruction: if the task contradicts the plan or the actual codebase, don't improvise — report SPEC_DRIFT and let a human decide whether the plan or the spec needs to change. Silent improvisation is exactly the failure mode this whole pipeline exists to prevent.
Final QA (Quinn). Once every task is checked off, a QA agent independently re-verifies the Definition of Done against the actual repository state — not the task list's checkmarks — and renders PASS, CONCERNS, or FAIL. A FAIL reopens exactly the failing tasks; it doesn't restart the whole pipeline.

For a genuine one-liner, /wiz quick <task> skips all of this: one
knowledge query, one complexity estimate, one specialist, done. The
mandatory pipeline is the default because most tasks aren't one-liners, even
when they look like one at 9pm.

Twelve specialists, one routing table

Wizard doesn't have a single "coder" persona — it has an architect, a senior
developer as the generalist fallback, and dedicated specialists for Java,
Angular, React, Python, UI/CSS, DevOps, QA, business analysis, database
work, and security. Routing is keyword-scored against one fixed trigger
table, shared by the quick-mode router and the plan phase's task-owner
assignment — a task never gets routed one way in one mode and a different
way in another:

Role	Triggers
`architect`	design, architecture, how should we, system, trade-off, ADR, strategy
`seniorDev`	implement, refactor, review, PR, pull request, code review
`javaDev`	`.java`, spring, springboot, maven, gradle, hibernate, JPA, `@Bean`, `@Service`
`angularDev`	`.component.ts`, `.module.ts`, angular, `@Component`, `@NgModule`, rxjs
`uiDev`	CSS, SCSS, HTML, template, styling, layout, accessibility, Tailwind
`reactDev`	react, jsx, tsx, `useState`, `useEffect`, Next.js, hook
`pythonDev`	`.py`, django, fastapi, flask, pytest, pandas, pip
`devops`	Dockerfile, k8s, pipeline, CI, CD, Helm, Terraform, kubectl, yaml
`qa`	test, spec, assertion, bug, regression, JUnit, Jest, coverage
`ba`	requirement, user story, acceptance criteria, business rule, stakeholder
`dba`	SQL, schema, migration, index, stored procedure, Flyway, Liquibase
`securityEng`	vulnerability, CVE, OWASP, auth, token, injection, XSS, pentest

Ties resolve architect > seniorDev > any specialist; nothing matching
falls back to seniorDev. It's a small detail, but it's the difference
between "an AI wrote this" and "the person who actually knows Spring Boot
wrote this."

Context that isn't just "paste the file"

Before any routing decision, Wizard queries a layered knowledge pipeline:
a pure-Java unified index (LuceneDb) covering both documentation and code
structure, falling back to a graphify code graph if the index hasn't been
built yet, plus a relational knowledge graph for typed entity relationships,
merged and weighted into a single context block. If none of that is
available, it falls back to a plain grep. The point isn't sophistication for
its own sake — it's that a spec written with zero awareness of the existing
codebase is a spec that will drift the moment implementation starts.

The merge logic itself is a short, readable cascade — each source is tried
in priority order, and only contributes if it actually returns something:

def retrieve(task_summary, working_dir, config=None):
    """Run all enabled knowledge sources and return a merged context string."""
    cfg = config or {}
    wd = Path(working_dir)
    budget = int(cfg.get("maxContextTokens", _DEFAULT_BUDGET))
    weights = {**_DEFAULT_WEIGHTS, **cfg.get("weights", {})}
    parts = []  # (source_name, content, weight)

    # Source 0: LuceneDb — preferred when built, no Node/npm/network needed.
    lucenedb_used = False
    if cfg.get("lucenedb", {}).get("enabled", True) and _lucenedb.available():
        result = _lucenedb.query(task_summary, lucenedb_index_path, limit=10)
        if result:
            parts.append(("lucenedb", result, weights.get("graph", 0.5) + weights.get("qmd", 0.3)))
            lucenedb_used = True

    # Source 1: graphify code graph — automatic fallback if LuceneDb isn't built.
    if not lucenedb_used and _graphify.available(graph_path):
        graph_budget = int(budget * weights["graph"])
        result = _graphify.query(task_summary, graph_path, budget=max(500, graph_budget))
        if result:
            parts.append(("graphify", result, weights["graph"]))

    # Source 2: relational knowledge graph — typed entity links.
    if rel_cfg.get("enabled") and _relational.available(rel_path):
        rel_budget = int(budget * weights["relational"])
        edges = _relational.query(_extract_terms(task_summary), rel_path, limit=10)
        formatted = _relational.format_results(edges, budget_chars=rel_budget)
        if formatted:
            parts.append(("knowledge-graph", formatted, weights["relational"]))

    # Last resort: plain grep, no index required at all.
    if not parts:
        grep_results = _grep_fallback(task_summary, wd, cfg.get("fallbackGlob", "**/*.{md,txt,java,ts,py}"))
        if grep_results:
            parts.append(("grep", grep_results, 1.0))

    if not parts:
        return ""
    return "\n\n".join(
        f'<knowledge_context source="{source}">\n{content}\n</knowledge_context>'
        for source, content, _ in parts
    )

No index built yet on a brand-new checkout? You still get grep results.
LuceneDb built? It wins by default weight. Nothing ever hard-fails just
because one source isn't available.

A case study in eating your own dog food

This plugin was itself repackaged out of an older, looser setup: a skill
file under ~/.claude/skills/wizard, a folder of agent definitions, and a
standalone Python wizard repo that supplied the knowledge-retrieval code.
The repackaging was supposed to make the plugin fully self-contained —
install it, and it runs, no sibling repo required.

It didn't, quite. A routing exercise (asking Wizard itself, in quick mode,
whether the new plugin would work without the old repo present) turned up
three separate breakages, each more interesting than the last:

A stale absolute path. The plugin's knowledge-query step had a path hardcoded that didn't exist on the machine in question at all, because the source repo had since moved:

  - sys.path.insert(0, 'C:/projects/wizard')
  + sys.path.insert(0, r'${CLAUDE_PLUGIN_ROOT}/scripts/wizard')

Wrong in two ways at once: pointing outside the plugin, and pointing at
a location that was already gone. ${CLAUDE_PLUGIN_ROOT} — the same
variable the plugin's other scripts already used correctly — resolves
inside the plugin's own install directory, wherever that happens to be.

An install script referencing the wrong folder name. The skill had been renamed from skills/wizard/ to skills/wiz/ during the repackaging, but install.sh still copied from the old name:

  - cp "$REPO_DIR/skills/wizard/SKILL.md" "$SKILLS_DIR/wizard/SKILL.md"
  + cp "$REPO_DIR/skills/wiz/SKILL.md" "$SKILLS_DIR/wizard/SKILL.md"

It would have failed the moment anyone actually ran it. The same script
also checked for top-level knowledge/, retrieval/, wizard/
directories that no longer existed at that level — they'd been nested
under scripts/wizard/ during the repackaging:

  - if [ -d "$REPO_DIR/knowledge" ] && [ -d "$REPO_DIR/retrieval" ] && [ -d "$REPO_DIR/wizard" ]; then
  -   ln -sfn "$REPO_DIR/knowledge" "$WIZARD_LIB_DIR/knowledge"
  -   ln -sfn "$REPO_DIR/retrieval" "$WIZARD_LIB_DIR/retrieval"
  -   ln -sfn "$REPO_DIR/wizard" "$WIZARD_LIB_DIR/wizard"
  + if [ -d "$REPO_DIR/scripts/wizard/config" ] && [ -d "$REPO_DIR/scripts/wizard/retrieval" ] && [ -d "$REPO_DIR/scripts/wizard/wizard" ]; then
  +   ln -sfn "$REPO_DIR/scripts/wizard/config" "$WIZARD_LIB_DIR/config"
  +   ln -sfn "$REPO_DIR/scripts/wizard/retrieval" "$WIZARD_LIB_DIR/retrieval"
  +   ln -sfn "$REPO_DIR/scripts/wizard/wizard" "$WIZARD_LIB_DIR/wizard"

Silent, not loud: the original condition just evaluated false and skipped
the block, so install.sh never actually crashed — it just quietly never
linked the knowledge modules it advertised, which is arguably worse than
a hard failure.

A missing package, not just a missing path. The deepest issue: retrieval/unified.py imports retrieval/relational.py unconditionally at module load —

  from . import graphify as _graphify
  from . import lucenedb as _lucenedb
  from . import relational as _relational

— and relational.py needs a KnowledgeGraph class:

  try:
      from ..knowledge.graph import KnowledgeGraph
  except (ImportError, ValueError):
      from knowledge.graph import KnowledgeGraph

Both branches assume a knowledge package sits somewhere reachable — as
a parent-package sibling, or as a top-level package on sys.path. Neither
was true: the entire knowledge/ directory (graph.py, extractor.py,
build.py, five files total) had simply never been copied into the
plugin's bundled Python tree during the carve-out. Fixing the path alone
would have traded one ModuleNotFoundError for another the instant
anything tried to import retrieval.unified. The fix required going back
to the original repo, confirming the missing package had no further
hidden dependencies of its own (a quick grep for its imports — only
pathlib, re, json, dataclasses, and its own relative imports came
back), and copying it in as a sibling of the packages that were already
there:

  scripts/wizard/
  ├── config/
  ├── retrieval/
  ├── wizard/
  └── knowledge/   <- added: graph.py, extractor.py, build.py, __init__.py, __main__.py

The lesson generalizes past this one plugin: "it imports correctly" and "the
path string points somewhere plausible" are different claims, and only one
of them is verifiable by reading the text of the code. The other one you
have to actually trace, import by import, or run.

Getting started

claude --plugin-dir /path/to/wizard-plugin
/wizard:wiz setup

setup walks through working directory, model tier preferences, and which
specialist roles you want enabled. After that:

/wizard:wiz <task>          # the full spec -> plan -> code -> review pipeline
/wizard:wiz quick <task>    # skip the gates for a genuine one-liner
/wizard:wiz knowledge status

Runtime configuration lives at ~/.wizard/config.json, outside the plugin
itself, so upgrading the plugin never touches your working-directory or
model-tier preferences.

Why bother with all this ceremony

Because the ceremony is the point. A model that writes code without a spec
will happily write code that solves the wrong problem, quickly and
confidently. A model that writes code without a plan will happily improvise
around a codebase constraint it hasn't actually checked. A model that grades
its own homework will mark it correct. Wizard doesn't make Claude smarter —
it makes the process harder to shortcut, which turns out to matter more.

Lucene V/S KQL

Mallikarjun H T — Thu, 01 May 2025 14:20:10 +0000

A Tour of KQL vs Lucene

Elasticsearch is a search engine, and Kibana can be used to search documents in Elasticsearch.
A search is executed by sending a query to Elasticsearch. A query can answer many different types of questions.

For example:

Who are the customers with the first name Edison?
What are the names of customers in India?
Are there any orders for Men’s Clothing OR Electronics section?

Should I use KQL or Lucene in the Kibana query bar?

It Depends, Both KQL and Lucine have a quite a lot in common.
I recoment to start with KQL then switch to Lucine for Specific Aggrigation use Case.

Learn By Executing.

in order to learn use the Free trial or download the pre prepared data sets to run locally if cloud trial expires.

KQL Features.

once you load your data you can use kibana to explore and learn about the features.

NOTE: KQL is case in sencitive and Lucine operaters are All Caps only

Autocomplete - KQL
Full Testsearch - KQL and Lucine

Exact Match

You can search for documents that contain an exact term in a provided field of data type keyword(opens in a new tab), which will not be analyzed. Not analyzed strings are case sensitive.

Example: Customer_first_name.keyword: Elyssa

Note: keyword is a type that only supports Exact search, if not given while searching then anything close to the word is returned.

Phrase search

To search text fields where the search terms are in the order provided, surround the value in double quotation marks, as follows:

Example: “Elyssa Underwood” - this is all field search
Example: Customer_full_name: “Elyssa Underwood” - this ia field specific search

Fuzzy Search only available in Lucine

arjun~1 - will match any word with 1ch differance, the number determines the no of charecters that can differ
arj*n - whild charecter search one or more charecters can occur in between arj and n
arj?n - single charecter only

NOTE: bad idea to use ** or ?* as this can cause perfomance issues.
to allow change the settings in elastic search.

Regex

A regular expression is a way to match patterns in data using placeholder characters called operators. It returns documents that contain terms matching a regular expression. Syntax is based on the Lucene regular expression engine.

Example: /.+91[0-9]{9}./

this will match number with +91 xxx xxx xxx

Ranges

To search documents that contain terms within a provided range, use KQL’s range syntax. You can use the range syntax for string values, IP addresses, and timestamps.

KQL Supports the following syntax >, >=, <, and <=

Lucene supports bracketed range syntax. Ranges can be specified for date, numeric or string fields. Inclusive ranges are specified with square brackets [min TO max] and exclusive ranges with curly brackets {min TO max}.

 date:[2012-01-01 TO 2012-12-31]
 count:[1 TO 5]
 tag:{alpha TO omega}

## Proximity Search
Lucine Exclusive
Proximity search(opens in a new tab) allows the specified words to be further apart or in a different order, a proximity search allows us to specify a maximum edit distance of words in a phrase. The closer the text in a field is to the original order specified in the query string, the more relevant that document is considered to be

“open data”~1

The above example will find all the documents where “open data” is found as a phrase and also “open source data” is a hit where one more term is present in between open and data.

Boosting

Lucene

Use the boost operator ^ to make one term more relevant than another. For instance, if we want to find all documents about foxes, but we are especially interested in quick foxes:

quick^2 fox

The default boost value is 1, but can be any positive floating point number. Boosts between 0 and 1 reduce relevance.

Boosts can also be applied to phrases or to groups:

"john smith"^2 (foo bar)^4

Searching for IP addresses

KQL Lucene

Use "" around IP addresses while using Lucene syntax.
Lucene also supports CIDR notation.

Example: "192.168.03.01"

Nested fields

KQL

Querying nested fields requires a special syntax.

Example: user.names:{ first: "Alyssa" and last: "Underwood" }

Runtime fields

KQL Lucene

A runtime field is a field that is evaluated at query time.

KQL and syntex

Mallikarjun H T — Wed, 23 Apr 2025 14:07:07 +0000

Should I use KQL or Lucene in the Kibana query bar?

It depends! KQL and Lucene have quite a lot in common, but there are some differences. Let's explore some key-features one by one to find out the best use-case for each language.

For new users, we recommend starting with KQL(opens in a new tab) and switching to Lucene(opens in a new tab) if any particular feature is unavailable in KQL.

KQL

Kibana Query Language (KQL) is a simple text-based query language for filtering data.
KQL only filters data, and has no role in aggregating, transforming, or sorting data.

Semi-structured search

free text search with field-based search.

The semi-structured search will filter documents for matches, and only return matching documents.

Filter for documents where a field exists

To filter documents for which an indexed value exists for a given field, use the * operator.

For example, to filter for documents where the http.request.method field exists, use the following syntax:

http.request.method: *

This checks for any indexed value, including an empty string.

Filter for documents that match a value

Use KQL to filter for documents that match a specific number, text, date, or boolean value. For example, to filter for documents where the http.request.method is GET, use the following query:
http.request.method: GET

to search all fields for “Hello”, use the following:

Hello

To querying keyword, numeric, date, or boolean fields, the value must be an exact match, including punctuation and case.

To Search text field that has "null pointer" then querry using http.request.body.content: "null pointer" if the \" are not used then you will get all documents that has null pointer OR Pointer null. This is called term querry.

to search urls use \ or you will find syntax error OR try with ".
Example:

http.request.referrer: "https://example.com"  
http.request.referrer: https\://example.com

[!Note]
\():<>"* excape These Charecters with \

Filter for documents within a range

to search for all documents for which http.response.bytes is less than 10000, use the following syntax:
http.response.bytes < 10000

to search In Range use this
http.response.bytes > 10000 and http.response.bytes <= 20000

Filter for documents using wildcards

To search for documents matching a pattern, use the wildcard syntax. For example, to find documents where http.response.status_code begins with a 4, use the following syntax:

http.response.status_code: 4*
By default, leading wildcards are not allowed for performance reasons. You can modify this with the query:allowLeadingWildcards advanced setting.

[!Note]
Only * is currently supported. This matches zero or more characters.

Negating a query

use NOT in front of querry like:
NOT http.request.method: GET

Querying nested fields

Use . operater to query nested fields like:
user.names:{ first: "Alice" and last: "White" }

Installing Elastic Search 7.14 with docker

Mallikarjun H T — Tue, 10 Sep 2024 04:52:53 +0000

Welcome to the ultimate guide on Elastic Search and Kibana! If you're looking for a powerful, open-source tool that can help you search, analyze, and visualize your data like never before, then look no further. In this comprehensive guide, we'll take you through everything you need to know about Elastic Search and Kibana - from what they are and how they work together to their installation with Docker. Whether you're a developer or just getting started with these tools, get ready to learn everything there is to know about using Elastic Search and Kibana with Docker in this exciting blog post!

Introduction to Elasticsearch and Kibana

Elasticsearch is a powerful open source search and analytics engine that makes data easy to explore. Kibana is an open source data visualization platform that allows you to interact with your data through beautiful visualizations. In this guide, we'll show you how to use Docker to install and run Elasticsearch and Kibana on your local machine.

What is Elastic Search?

Elastic Search is a powerful, open source, distributed search and analytics engine. It is built on top of the Apache Lucene search library and supports full text search, as well as structured data search with SQL-like query syntax. Elastic Search can be used to power search for websites, applications, and enterprise search solutions. It is also commonly used for log analysis, monitoring, and security analytics.

What is Kibana?

Kibana is an open source data visualization plugin for the Elasticsearch stack. It provides a rich set of tools to help you visualize and explore your data in Elasticsearch. Kibana is a great way to get started with visualizing your data in Elasticsearch.

How to Install Elastic Search and Kibana with Docker

version: '2.2'
services:
  node01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.14.1
    container_name: node01
    environment:
      - node.name=node01
      - cluster.name=es-my-cluster
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
  kib01test:
    image: docker.elastic.co/kibana/kibana:7.14.1
    container_name : kib01test
    environment:
      - ELASTICSEARCH_HOSTS=http://node01:9200
    ports:
      - 5601:5601
    networks: 
      - elastic

volumes:
  data01:
    driver: local

networks:
  elastic:
    driver: bridge

ES - Node APIs

Mallikarjun H T — Tue, 13 Aug 2024 09:40:44 +0000

# If no filters are given, the default is to select all nodes
curl -X GET "localhost:9200/_nodes?pretty"
# Explicitly select all nodes
curl -X GET "localhost:9200/_nodes/_all?pretty"
# Select just the local node
curl -X GET "localhost:9200/_nodes/_local?pretty"
# Select the elected master node
curl -X GET "localhost:9200/_nodes/_master?pretty"
# Select nodes by name, which can include wildcards
curl -X GET "localhost:9200/_nodes/node_name_goes_here?pretty"
curl -X GET "localhost:9200/_nodes/node_name_goes_*?pretty"
# Select nodes by address, which can include wildcards
curl -X GET "localhost:9200/_nodes/10.0.0.3,10.0.0.4?pretty"
curl -X GET "localhost:9200/_nodes/10.0.0.*?pretty"
# Select nodes by role
curl -X GET "localhost:9200/_nodes/_all,master:false?pretty"
curl -X GET "localhost:9200/_nodes/data:true,ingest:true?pretty"
curl -X GET "localhost:9200/_nodes/coordinating_only:true?pretty"
# Select nodes by custom attribute (e.g. with something like `node.attr.rack: 2` in the configuration file)
curl -X GET "localhost:9200/_nodes/rack:2?pretty"
curl -X GET "localhost:9200/_nodes/ra*:2?pretty"
curl -X GET "localhost:9200/_nodes/ra*:2*?pretty"

ES - CAT APIs

Mallikarjun H T — Tue, 13 Aug 2024 09:38:43 +0000

cat APIs

Introduction

All the cat commands accept a query string parameter help to see all the headers and info they provide, and the /_cat command alone lists all the available commands.

Common parameters

Verbose - curl -X GET "localhost:9200/_cat/master?v&pretty"
Help - curl -X GET "localhost:9200/_cat/master?help&pretty"
Headers - curl -X GET "localhost:9200/_cat/nodes?h=ip,port,heapPercent,name&pretty"
Numeric formats - curl 'localhost:9200/_cat/indices?bytes=b' | sort -rnk8
Response as text, json, smile, yaml or cbor - curl 'localhost:9200/_cat/indices?format=json&pretty'
Sort - curl -X GET "localhost:9200/_cat/templates?v&s=order:desc,index_patterns&pretty"

cat aliases

curl -X GET "localhost:9200/_cat/aliases?v&pretty"

cat allocation

curl -X GET "localhost:9200/_cat/allocation?v&pretty"

cat count

curl -X GET "localhost:9200/_cat/count?v&pretty"

curl -X GET "localhost:9200/_cat/count/twitter?v&pretty"

cat fielddata

curl -X GET "localhost:9200/_cat/fielddata?v&pretty"

cat health

curl -X GET "localhost:9200/_cat/health?v&pretty"

cat indices

curl -X GET "localhost:9200/_cat/indices?pretty"

Apply Filter

curl -X GET "localhost:9200/_cat/indices/twi*?pretty"

cat un healthy index

indices which are not ready

curl -X GET "localhost:9200/_cat/indices?v&health=yellow&pretty"

sort by dock count

curl -X GET "localhost:9200/_cat/indices?v&s=docs.count:desc&pretty"

cat master

curl -X GET "localhost:9200/_cat/master?v&pretty"

cat nodes

curl -X GET "localhost:9200/_cat/nodes?v&pretty"

Documentation

cat pending tasks

curl -X GET "localhost:9200/_cat/pending_tasks?v&pretty"

cat plugins

curl -X GET "localhost:9200/_cat/plugins?v&s=component&h=name,component,version,description&pretty"

cat recovery

curl -X GET "localhost:9200/_cat/recovery?v&pretty"

cat repositories

curl -X GET "localhost:9200/_cat/repositories?v&pretty"

cat thread pool

curl -X GET "localhost:9200/_cat/thread_pool?pretty"

for field data visit Documentation

cat shards

curl -X GET "localhost:9200/_cat/shards?pretty"

Reasion for un assigned shared look Documentation

INDEX_CREATED | Unassigned as a result of an API creation of an index.

CLUSTER_RECOVERED | Unassigned as a result of a full cluster recovery.

INDEX_REOPENED | Unassigned as a result of opening a closed index.

DANGLING_INDEX_IMPORTED | Unassigned as a result of importing a dangling index.

NEW_INDEX_RESTORED | Unassigned as a result of restoring into a new index.

EXISTING_INDEX_RESTORED | Unassigned as a result of restoring into a closed index.

REPLICA_ADDED | Unassigned as a result of explicit addition of a replica.

ALLOCATION_FAILED | Unassigned as a result of a failed allocation of the shard.

NODE_LEFT | Unassigned as a result of the node hosting it leaving the cluster.

REROUTE_CANCELLED | Unassigned as a result of explicit cancel reroute command.

REINITIALIZED | When a shard moves from started back to initializing, for example, with shadow replicas.

REALLOCATED_REPLICA | A better replica location is identified and causes the existing replica allocation to be cancelled.

cat segments

this where your data is saved on the FS.

curl -X GET "localhost:9200/_cat/segments?v&pretty"

snapshots

curl -X GET "localhost:9200/_cat/snapshots/repo1?v&s=id&pretty"

templates

curl -X GET "localhost:9200/_cat/templates?v&s=name&pretty"

ES - Index API - part 4

Mallikarjun H T — Mon, 05 Aug 2024 08:04:39 +0000

Indices Stats

curl -X GET "localhost:9200/_stats?pretty"

Indices Segments

curl -X GET "localhost:9200/test/_segments?pretty"

curl -X GET "localhost:9200/test/_segments?verbose=true&pretty"

Indices Recovery

curl -X GET "localhost:9200/_recovery?human&pretty"

curl -X GET "localhost:9200/_recovery?human&detailed=true&pretty"

Indices Shard Stores

# return information of only index test
curl -X GET "localhost:9200/test/_shard_stores?pretty"
# return information of only test1 and test2 indices
curl -X GET "localhost:9200/test1,test2/_shard_stores?pretty"
# return information of all indices
curl -X GET "localhost:9200/_shard_stores?pretty"

Clear Cache

curl -X POST "localhost:9200/twitter/_cache/clear?pretty"

Flush

curl -X POST "localhost:9200/twitter/_flush?pretty"

Synced Flush

curl -X POST "localhost:9200/twitter/_flush/synced?pretty"

Refresh

curl -X POST "localhost:9200/twitter/_refresh?pretty"

Force Merge

curl -X POST "localhost:9200/twitter/_forcemerge?pretty"

ES Index API - part 3

Mallikarjun H T — Mon, 05 Aug 2024 07:47:49 +0000

PUT Mappings

create index with mappings

curl -X PUT "localhost:9200/twitter?pretty" -H 'Content-Type: application/json' -d'
{}'

curl -X PUT "localhost:9200/twitter/_mapping/_doc?pretty" -H 'Content-Type: application/json' -d'
{
  "properties": {
    "email": {
      "type": "keyword"
    }
  }
}'

update existing index

curl -X PUT "localhost:9200/my_index?pretty" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "_doc": {
      "properties": {
        "name": {
          "properties": {
            "first": {
              "type": "text"
            }
          }
        },
        "user_id": {
          "type": "keyword"
        }
      }
    }
  }
}'

curl -X PUT "localhost:9200/my_index/_mapping/_doc?pretty" -H 'Content-Type: application/json' -d'
{
  "properties": {
    "name": {
      "properties": {
        "last": { 
          "type": "text"
        }
      }
    },
    "user_id": {
      "type": "keyword",
      "ignore_above": 100 
    }
  }
}'

Get Mappings

curl -X GET "localhost:9200/twitter/_mapping/_doc?pretty"

GET field mapping

curl -X GET "localhost:9200/publications/_mapping/_doc/field/title?pretty"

ES Index API- part 2

Mallikarjun H T — Mon, 05 Aug 2024 07:33:31 +0000

Delete Index

curl -X DELETE "localhost:9200/twitter?pretty"

Get Index

this will return mappings and settings if index exist

curl -X GET "localhost:9200/twitter?pretty"

check if index exist

curl -I "localhost:9200/twitter?pretty"

Open / Close Index API

use this with caution - like updating or to prevent new document insertion - search will work but indexing will not work

curl -X POST "localhost:9200/my_index/_close?pretty"
curl -X POST "localhost:9200/my_index/_open?pretty"

Shrink Index

use this to reduce the number of shards for version < 7 it is 5 and above it is 1 shard.

Preparing an index for shrinking

curl -X PUT "localhost:9200/my_source_index/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index.routing.allocation.require._name": "shrink_node_name", 
    "index.blocks.write": true 
  }
}
'

Shrinking an index

curl -X POST "localhost:9200/my_source_index/_shrink/my_target_index?copy_settings=true&pretty" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index.routing.allocation.require._name": null, 
    "index.blocks.write": null 
  }
}
'

Shrinking with settings

curl -X POST "localhost:9200/my_source_index/_shrink/my_target_index?copy_settings=true&pretty" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index.number_of_replicas": 1,
    "index.number_of_shards": 1, 
    "index.codec": "best_compression" 
  },
  "aliases": {
    "my_search_indices": {}
  }
}
'

Rollover Index

Defining the new index

curl -X PUT "localhost:9200/logs-000001?pretty" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    "logs_write": {}
  }
}'

curl -X POST "localhost:9200/logs_write/_rollover?pretty" -H 'Content-Type: application/json' -d'
{
  "conditions" : {
    "max_age": "7d",
    "max_docs": 1000,
    "max_size": "5gb"
  },
  "settings": {
    "index.number_of_shards": 2
  }
}'

Dry run

curl -X PUT "localhost:9200/logs-000001?pretty" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    "logs_write": {}
  }
}'

curl -X POST "localhost:9200/logs_write/_rollover?dry_run&pretty" -H 'Content-Type: application/json' -d'
{
  "conditions" : {
    "max_age": "7d",
    "max_docs": 1000,
    "max_size": "5gb"
  }
}'

ES index API - part 1

Mallikarjun H T — Mon, 05 Aug 2024 07:13:56 +0000

Indices APIs

Index management:

Create Index
Delete Index
Get Index
Indices Exists
Open / Close Index API
Shrink Index
Split Index
Rollover Index

### Mapping management:

Put Mapping
Get Mapping
Get Field Mapping
Types Exists

Alias management:

Index Aliases

Index settings:

Update Indices Settings
Get Settings
Analyze
Index Templates

Monitoring:

Indices Stats
Indices Segments
Indices Recovery
Indices Shard Stores

Status management:

Clear Cache
Refresh
Flush
Force Merge

Create Index

index creation without any mappings

curl -X PUT "localhost:9200/twitter?pretty"

NOTE:

Index name limitations

There are several limitations to what you can name your index. The complete list of limitations are:

Lowercase only
Cannot include \, /, *, ?, ", <, >, |, ` ` (space character), ,, #
Indices prior to 7.0 could contain a colon (:), but that’s been deprecated and won’t be supported in 7.0+
Cannot start with -, _, +
Cannot be . or ..
Cannot be longer than 255 bytes (note it is bytes, so multi-byte characters will count towards the 255 limit faster)

Index Settings

To update index use this, in this payload we are setting number of shards and no of replicas

curl -X PUT "localhost:9200/twitter?pretty" -H 'Content-Type: application/json' -d'
{
    "settings" : {
        "number_of_shards" : 3,
        "number_of_replicas" : 2
    }
}
'

Mappings

setting mappings to an index while creation

curl -X PUT "localhost:9200/test?pretty" -H 'Content-Type: application/json' -d'
{
    "settings" : {
        "number_of_shards" : 1
    },
    "mappings" : {
        "_doc" : {
            "properties" : {
                "field1" : { "type" : "text" }
            }
        }
    }
}
'

setting Aliases on existing index

curl -XPUT localhost:9200/index1/_alias/index2
multiple Aliases and filtering

curl -X PUT "localhost:9200/test?pretty" -H 'Content-Type: application/json' -d'
{
    "aliases" : {
        "alias_1" : {},
        "alias_2" : {
            "filter" : {
                "term" : {"user" : "kimchy" }
            },
            "routing" : "kimchy"
        }
    }
}
'

java prep - part 3

Mallikarjun H T — Tue, 18 Jun 2024 02:42:49 +0000

can we disable the default behaviour of spring explain with example

Yes, in Spring Framework, you can disable or customize default behaviors by overriding configurations or using specific annotations to alter how Spring manages components and processes requests. Let's explore a couple of examples where you might want to disable default behaviors in Spring:

Example 1: Disabling Default Spring Security Configuration

By default, Spring Security applies a set of security configurations to your application, including basic authentication and authorization rules. You might want to disable these default behaviors if you have custom security requirements.

Step-by-step Example:

Disable Security Auto-Configuration: To disable Spring Security's default configuration, you can exclude the SecurityAutoConfiguration class from auto-configuration in your main application class or configuration class.

   import org.springframework.boot.SpringApplication;
   import org.springframework.boot.autoconfigure.SpringBootApplication;
   import org.springframework.boot.autoconfigure.security.servlet.SecurityAutoConfiguration;

   @SpringBootApplication(exclude = {SecurityAutoConfiguration.class})
   public class MyApplication {

       public static void main(String[] args) {
           SpringApplication.run(MyApplication.class, args);
       }
   }

In this example:

@SpringBootApplication is annotated with exclude = {SecurityAutoConfiguration.class}, which disables Spring Security's auto-configuration.

Customize Security Configuration: After disabling default configuration, you can then customize security settings by creating your own SecurityConfig class and configuring it based on your specific requirements using @EnableWebSecurity and extending WebSecurityConfigurerAdapter.

   import org.springframework.context.annotation.Configuration;
   import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
   import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;

   @Configuration
   @EnableWebSecurity
   public class SecurityConfig extends WebSecurityConfigurerAdapter {

       @Override
       protected void configure(HttpSecurity http) throws Exception {
           // Customize your security configuration here
           http
               .authorizeRequests()
                   .antMatchers("/public/**").permitAll()
                   .anyRequest().authenticated()
                   .and()
               .formLogin()
                   .loginPage("/login")
                   .permitAll()
                   .and()
               .logout()
                   .permitAll();
       }
   }

This SecurityConfig class provides customized security configurations, overriding the default behavior that was disabled.

Example 2: Disabling Spring Boot Banner

Spring Boot displays a banner by default during application startup. You may want to disable this banner for specific reasons, such as cleaner log output.

Step-by-step Example:

Disable Banner Programmatically: You can disable the banner programmatically by setting spring.main.banner-mode property to OFF in your application.properties or application.yml.

   spring.main.banner-mode=OFF

This configuration instructs Spring Boot not to display the banner during application startup.

Disable Banner via Configuration Class: Alternatively, you can disable the banner by creating a configuration class and using SpringApplication.setBannerMode(Banner.Mode.OFF) in your main application class.

   import org.springframework.boot.Banner;
   import org.springframework.boot.SpringApplication;
   import org.springframework.boot.autoconfigure.SpringBootApplication;

   @SpringBootApplication
   public class MyApplication {

       public static void main(String[] args) {
           SpringApplication app = new SpringApplication(MyApplication.class);
           app.setBannerMode(Banner.Mode.OFF);
           app.run(args);
       }
   }

Here, SpringApplication.setBannerMode(Banner.Mode.OFF) disables the banner explicitly for the MyApplication.

Conclusion:

Spring Framework provides flexibility to disable default behaviors by excluding auto-configurations, overriding configuration classes, or configuring properties. This approach allows developers to customize Spring applications according to specific requirements, ensuring that the framework's default behaviors align with application needs.

How caching works with example/ archtecture diagram as code

Caching in software systems, including those built with Spring Boot, involves storing frequently accessed data in a faster access storage to improve performance and reduce load on backend systems.

import org.springframework.cache.annotation.Cacheable;
import org.springframework.stereotype.Service;

@Service
public class UserService {

    @Cacheable(value = "usersCache", key = "#userId")
    public User getUserDetails(Long userId) {
        // Simulate fetching user details from a database
        User user = userRepository.findById(userId);
        return user;
    }
}

# Enable caching
spring.cache.type=ehcache

with Redis example

import java.io.Serializable;

public class Book implements Serializable {
    private String isbn;
    private String title;
    private String author;

    // Getters and setters (omitted for brevity)

    @Override
    public String toString() {
        return "Book{" +
                "isbn='" + isbn + '\'' +
                ", title='" + title + '\'' +
                ", author='" + author + '\'' +
                '}';
    }
}

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.stereotype.Service;

@Service
public class BookService {

    private static final String REDIS_KEY = "books";

    @Autowired
    private RedisTemplate<String, Book> redisTemplate;

    public void saveBook(Book book) {
        redisTemplate.opsForHash().put(REDIS_KEY, book.getIsbn(), book);
    }

    public Book getBook(String isbn) {
        return (Book) redisTemplate.opsForHash().get(REDIS_KEY, isbn);
    }

    public void deleteBook(String isbn) {
        redisTemplate.opsForHash().delete(REDIS_KEY, isbn);
    }
}

java interview prep part 2

Mallikarjun H T — Tue, 18 Jun 2024 02:25:52 +0000

Explain restcontroller annotation in springboot

In Spring Boot, @RestController is a specialized version of the @Controller annotation. It is used to indicate that the class is a RESTful controller that handles HTTP requests and directly maps them to the methods inside the class.

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class HelloController {

    @GetMapping("/hello")
    public String hello() {
        return "Hello, World!";
    }
}

Purpose of @RestController:
API Endpoint Handling:

@RestController combines @Controller and @ResponseBody. It is primarily used to create RESTful web services where HTTP requests (GET, POST, PUT, DELETE, etc.) are mapped directly to the methods in the class.
Automatic JSON/XML Conversion:

Methods in a @RestController return domain objects directly. Spring Boot automatically converts these objects into JSON or XML responses (based on the Accept header of the request) using Jackson or another configured message converter.
Simplification of Controller Code:

By annotating a class with @RestController, developers can streamline the code as they no longer need to add @ResponseBody to every request handling method.

Explain microstructure architecture

Microservices Architecture:
Microservices Architecture is an architectural style that structures an application as a collection of loosely coupled services. Each service is self-contained, independently deployable, and typically focuses on performing a single business function.

Benefits of Microservices Architecture:
Scalability: Individual microservices can be scaled independently based on demand.
Flexibility and Agility: Enables rapid development, deployment, and updates of services.
Improved Fault Isolation: Failures are contained within a single service, reducing impact on the overall system.
Technology Diversity: Allows for the use of different technologies and frameworks for different services.
Enhanced Maintainability: Smaller, focused services are easier to understand, modify, and maintain compared to monolithic applications.

Challenges:
Increased Complexity: Managing a distributed system introduces complexity in deployment, testing, monitoring, and debugging.
Data Management: Maintaining consistency and managing data across multiple services can be challenging.
Service Discovery and Communication: Ensuring reliable communication and discovering services dynamically in a distributed environment.
Operational Overhead: Requires robust infrastructure and operational practices to support and monitor microservices effectively.

Microservices architecture can be categorized into several types or patterns, each emphasizing different aspects of service decomposition, communication patterns, and deployment strategies. Here are some common types of microservices architecture patterns:

1. Monolithic Application Decomposition:

Description: In this approach, a monolithic application is decomposed into smaller services, but each service may still be relatively large and cohesive compared to other microservices architectures.
Characteristics:
- Services are typically larger and might handle multiple related functionalities.
- Often uses synchronous communication (e.g., RESTful APIs) between services.
- Deployment and scaling are often handled per service, but still may share some components (e.g., databases).

2. Layered Architecture:

Description: Services are organized into layers, where each layer represents a set of related functionalities or responsibilities.
Characteristics:
- Each layer may be implemented as one or more microservices.
- Communication between layers can be synchronous or asynchronous.
- Promotes separation of concerns and scalability within each layer.

3. Event-Driven Architecture:

Description: Services communicate through events (messages) asynchronously. Events represent state changes or significant actions within the system.
Characteristics:
- Services are decoupled and communicate through message brokers or event buses.
- Enables loose coupling and scalability by allowing services to react to events without direct dependencies.
- Supports eventual consistency and fault isolation.

4. API Gateway Pattern:

Description: An API Gateway acts as a single entry point for clients to interact with multiple microservices.
Characteristics:
- Provides a unified interface for clients, routing requests to appropriate microservices.
- May handle authentication, rate limiting, and request aggregation.
- Improves client-side performance and simplifies the client's view of the system architecture.

5. Service Mesh:

Description: A dedicated infrastructure layer for handling service-to-service communication, including load balancing, service discovery, encryption, and monitoring.
Characteristics:
- Enhances visibility, reliability, and security of microservices communication.
- Typically implemented using a sidecar proxy (e.g., Envoy) alongside each microservice instance.
- Allows centralized management of microservices communication policies.

6. Saga Pattern:

Description: Handles long-lived transactions that span multiple microservices, ensuring eventual consistency without distributed transactions.
Characteristics:
- Uses a series of local transactions (compensating actions) within each microservice.
- Coordination and orchestration are typically managed by a Saga orchestrator.
- Enables rollback and compensation in case of failures, ensuring data consistency across microservices.

7. Containerization and Orchestration:

Description: Microservices are deployed as lightweight, isolated containers (e.g., Docker) and orchestrated using platforms like Kubernetes.
Characteristics:
- Simplifies deployment, scaling, and management of microservices.
- Provides infrastructure automation and supports microservices resilience.
- Enables efficient resource utilization and scalability through container orchestration.

Conclusion:

Each type of microservices architecture pattern offers distinct advantages and is suitable for different scenarios based on scalability needs, communication requirements, and operational considerations. The choice of architecture pattern depends on factors such as application complexity, team expertise, scalability requirements, and operational goals.