DEV Community: erfankashani

Model Migration on Productionized AI Applications: What Changes Beyond the API

erfankashani — Sat, 03 Oct 2026 04:20:42 +0000

Google is retiring gemini-2.5-pro on Vertex AI on October 20, 2026. If your product runs on it, you need to migrate before then!

I recently went through this migration practice on a legal document extraction AI-native product that runs on production. I knew going in that, it would be more than a config change, but I did not expect how many layers it would touch.

Changing the model in an AI product is very different from pointing a client at a new endpoint or upgrading a library. Each model has its own strengths and weaknesses, and its own way of filling the gaps when the input is not clear. When you swap the model, your product starts behaving differently. If you do not have the right harnesses, evals, and monitoring in place, you find out through accuracy drops, and the worst of them show up exactly where you were counting on the model's thinking capability on inferring data.

Below are the issues I ran into, in order.

Layer 1: Where the model lives

One of the most consequential decisions you have to make (not as a developer but as an organization) is where the model is being provisioned and how this impacts your data residency requirements. Gemini 2.5 Pro could be called from a single region, such as us-central1. Gemini 3.8 Flash cannot. It is offered only as a multi-region deployment (us or eu) or a global deployment, and each one uses a different endpoint shape:

Deployment type	Example location	Endpoint
Single-region	`us-central1`	`{region}-aiplatform.googleapis.com`
Multi-region	`us`, `eu`	`aiplatform.{loc}.rep.googleapis.com`
Global	`global`	`aiplatform.googleapis.com`

This difference is significant for a multinational company. A regional deployment gave you a clear answer to "where is our customer data processed?" Multi-region still keeps processing inside a jurisdiction such as the US or the EU. Global routes each request to wherever Google has capacity, so there is no data residency guarantee. If your contracts, regulators, or compliance team care about residency, you need to discuss the model location with them before you change it.

The caveats of moving to multi-regional endpoint

If your application is deployed on a cloud run with private VPC sharing and choose to move forward with a multi-region deployment option, There are chances that your first call fails with the following error:

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch,
certificate is not valid for 'aiplatform.us.rep.googleapis.com'. (_ssl.c:1016)

The code is fine but the networking rules.. not so much. The Cloud Run service sent its egress through a centrally managed Shared VPC subnet with Private Google Access enabled. That setup works for regional and global endpoints, but not for multi-region ones. Multi-region endpoints only became generally available on May 15, 2026, so many existing network setups were never designed with them in mind. Google documents the limitation in About accessing the Vertex AI API:

Private Google Access isn't supported for multi-region endpoints. If you attempt to connect to a multi-region endpoint using Private Google Access, you might experience connectivity issues, SSL/TLS handshake errors, or certificate mismatch warnings.

To establish private connectivity to multi-region endpoints, you must configure Private Service Connect endpoints for regional Google APIs.

The proper fix is a Private Service Connect endpoint for the regional Google APIs. When a central team owns networking, that is a medium-to-large change with its own timeline.

An interim workaround is to use the global deployment, which its endpoint works with the existing network. It has two trade-offs you should know up front:

There is no data residency guarantee, for the reasons above.
There is no context caching. The global endpoint does not support it, so make sure your code is not relying on the caching mechanism or if it does currently, you have implemented fail-safe approaches.

If you are planning this migration, check your egress path before you write any code.

Layer 2: The API contract changed

The syntax for passing tools is the same between the two models, which makes it easy to assume nothing else changed. Several things did.

Thinking configuration. Gemini 2.5 Pro takes a numeric thinking_budget. Gemini 3.8 Flash rejects it and expects a thinking_level enum: LOW, MEDIUM (default), or HIGH. MINIMAL is not supported on 3.8 Flash and fails API validation (unlike Gemini 3.5 Flash).

Sampling parameters. The Gemini 3 family drops the legacy sampling options. Remove temperature, top_p, and top_k from your GenerateContentConfig. candidate_count is not supported at all from Gemini 3.x onward. A lot of extraction code hardcodes a low temperature to get "deterministic" output, and Google calls out that exact pattern in its Gemini 3 prompting guide:

If your existing code explicitly sets temperature (especially to low values for deterministic outputs), we recommend removing this parameter and using the Gemini 3 default of 1.0... Changing the temperature (setting it below 1.0) may lead to unexpected behavior, such as looping or degraded performance, particularly in complex mathematical or reasoning tasks.

In code, the change looks like this:

from google.genai import types

# Before: gemini-2.5-pro
config = types.GenerateContentConfig(
    temperature=0.1,
    top_p=0.95,
    thinking_config=types.ThinkingConfig(thinking_budget=8192),
)

# After: gemini-3.8-flash
config = types.GenerateContentConfig(
    thinking_config=types.ThinkingConfig(thinking_level="MEDIUM"),
)

Changing the syntax is the easy part. A budget is a number you tune, but a level is a product decision. You now have to decide which flows deserve HIGH and which are fine on MEDIUM or LOW. Each choice changes model behavior, so each one needs its own accuracy validation.

Function calling. If your application relies on them, there are three differences to check:

If you orchestrate multi-turn tool use yourself, every FunctionResponse you send back must include both a call_id and a name.
If your prompt places inline instructions directly before a tool call, 2.5 Pro handles it, but 3.8 Flash can return MALFORMED_FUNCTION_CALL. Separate those instructions with clean blank lines (\n\n).
If your tools return images or audio, 3.8 Flash expects the assets inside the response payload rather than referenced through an external pointer.

Multi-turn chats. 3.8 Flash enforces server-side state tracking through previous_interaction_id, and it restricts manually pre-filling model turns in the payload history more than 2.5 did.

Context caching minimum. Once you are on a regional or multi-region endpoint where caching works, note that the minimum cache size goes from 1,024 tokens on 2.5 Pro to 4,096 tokens on 3.8 Flash. Short inputs, such as a single small document, will no longer trigger caching. There is no code change, but your cost profile changes, so measure it on real traffic instead of estimating it. See Create a context cache.

Layer 3: The model thinks differently

This part is not in any release note, and it is where the feel the migration can change your product (or its taste).

Both models are strong thinkers, but they approach the same problem differently, much like two experienced people would. Part of that difference is the Pro and Flash tiers being designed for different jobs, and part of it is the generation. From the product's point of view, the cause matters less than the effect.

Gemini 2.5 Pro is the more verbose of the two. It is comfortable with long reasoning chains, and it likes to connect dots, sometimes ones that are not there. In one case it inferred a transaction date from the date of the email chain the document came from. In another, a contract's start date depended on a condition under a separate agreement being met. 2.5 Pro took the start date at face value, added the contract term, and returned a confident end date.

Output format was the other difference. Our response schema is referenced in the prompt rather than enforced through function calling. With 2.5 Pro, this approach gave better results than enforcing the schema, which left us with many null records in the output. The cost was that 2.5 Pro sometimes drifted from the requested JSON shape.

Gemini 3.8 Flash on MEDIUM is more concise and less inclined to overthink. In the same conditional-contract case, it noticed that the start date depended on an event that had not happened yet, and it did not assume either date. When the evidence was not clear, it left the field null, which is exactly what our instructions asked for. It followed the output conventions in the prompt much more cleanly, kept to the JSON shape from the prompt-referenced schema without being forced by function calling, and did not do more than it was asked.

Which behavior is better depends on the product. For legal extraction, an inferred date that looks authoritative is worse than an honest null. For another product, the 2.5 Pro habit of connecting dots might be exactly what you want.

What this taught me about choosing a model

Every thinking model has its own personality. A prompt tuned for one model is not tuned for the next, and the habits you build into your prompts, such as how firmly you state the output format or how you tell the model to handle missing data, can have a different effect on another model. In my experience, even the same model can behave differently depending on which cloud serves it, since providers can wrap your input with their own system-level instructions before it reaches the model.

That changes how I think about model selection. Tuning prompts and validation around a model takes real effort, so choose a model with enough lifespan to make that effort worth it. Be careful with model routers in precision work like legal document extraction, because the same document can get a different interpretation depending on which model answered it. This is also why some teams invest in smaller models they host themselves and fine-tuned for specific task.

How to actually learn a model

Release notes list what changed in the API, but they cannot show you how the model handles your own documents.

One of the most effective ways I have found is to build an eval set for your application along with a set of standard baselines. With that in place, you can run different models, and the same model at different thinking levels, against the same tasks and compare the results directly. You can also see where each model is weak and where you can add quality through context management, prompt management, or tool calling, instead of guessing.

I will write about my approach to eval testing in my next post.

Resources

Moving Your Local Airflow to GCP for under $150 a month

erfankashani — Mon, 28 Sep 2026 02:17:33 +0000

My team runs ML training and prediction pipelines on Vertex AI. For a long
time, the thing telling those pipelines when to run was Apache Airflow,
installed by hand on a few on-prem Linux boxes. From there it orchestrated
our hybrid cloud setup from one place.

Next to Airflow sat our own configuration management system. It lets people
change a pipeline's settings on demand, and every change is tracked and
logged over time, so if a model behaves differently on Tuesday, we can see
what changed on Monday.

Both of these lived on Linux boxes we maintained ourselves, and those boxes
had to go: our organization is phasing out its on-prem setup and moving fully
cloud native. We are in the middle of that move right now. The on-prem
pipelines are moving out to different clouds, and our own pipelines and
compute are moving into GCP. With the work landing in GCP, putting the
orchestrator there too was the natural answer.

The goal was to make that move without ending up any less safe or less
reliable than what we had.

In this post I want to share how we did it, chapter by chapter. The code is
in the self-managed-airflow-on-gcp
repo, and everything below is a trimmed-down, working version of it.

Chapter 1: Trying Cloud Composer

If you ask "how do I run Airflow on GCP?", the answer is Cloud Composer.
It is managed Airflow: Google runs the scheduler, the database and the
workers, and you drop DAGs into a bucket.

We built it first. We stood up a Composer 3 environment with Terraform,
pointed our DAGs at it, and it worked.

The cost was the problem. With the pricing calculator, a small Composer
environment came to about $350 a month. Composer 3 bills for compute
units every hour the environment is alive, whether a DAG runs or not. Our
pipelines do not need a cluster awake around the clock to schedule a handful
of jobs, because the heavy work happens in Vertex AI anyway.

My search was not done there.

Chapter 2: Self-managed Airflow on a VM

Our Airflow does not do the heavy work. It decides when work happens and
hands it to Vertex AI or BigQuery. A scheduler like that fits on one small
VM.

So we built the same thing a second time:

one Compute Engine VM running a self-managed Airflow 3.x, with Postgres for the metadata database
Terraform to provision the platform, per environment (dev, test, prod)
a Fabric script to install Airflow, deploy code, and handle day-to-day maintenance

The estimate for this came in under $150 a month. About $20 of that is
the HTTPS Load Balancer in front of our config app (GCP charges a flat
$0.025/hour for the first five forwarding rules, roughly $18/month, plus
data). The rest is the VM, its disk, and Cloud NAT.

That number is not the whole cost. With Composer, Google upgrades Airflow and
patches the machine. On a VM, we do. We had both versions running side by
side, and the team picked the self-managed one: we are a technical team that
already ran Airflow ourselves, so the extra work was work we knew.

Cheaper only counts if it is also safe and reliable, so the rest of this post
is about how we got there.

Chapter 3: How the pieces fit

There are two components we deploy:

config_hq, the new home of our configuration system. It is a small Flask app with a plain HTML/JS frontend, running on Cloud Run. A user types config, presses Save, and it is written to a GCS bucket.
airflow-vm, the orchestrator. Its DAGs read that config and kick off the ML jobs.

The part I like most in this design is that Airflow never calls config_hq.
The two only share a bucket.

Our org policy disables Cloud Run's default run.app URL, so the service is
deployed with default_uri_disabled = true and there is no hostname for a
DAG to call anyway. Instead, config_hq writes every save as a new object
named config/<timestamp>-<id>.txt in a versioned bucket. That bucket is
also our change history. The VM gets read-only access to it, and the DAG
reads it directly:

def fetch_configs(**context):
    hook = GCSHook()
    configs = {
        blob_name: hook.download(bucket_name=CONFIG_HQ_BUCKET, object_name=blob_name).decode("utf-8")
        for blob_name in hook.list(bucket_name=CONFIG_HQ_BUCKET)
    }
    context["ti"].xcom_push(key="configs", value=configs)

If config_hq is down, Airflow still reads the last config that was saved.

Chapter 4: Security

Both components are closed by default.

config_hq is only reachable through an External HTTPS Load Balancer
with Identity-Aware Proxy (IAP) on the backend:

Cloud Run ingress is set to INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER, so the Load Balancer is the only way in.
IAP only lets through Google accounts listed in iap_authorized_members.
Its service account can write to its own bucket and nothing else.

The VM has no external IP at all (org policy
constraints/compute.vmExternalIpAccess), so there is no public path to it.
Getting in means passing two separate gates:

The firewall only allows port 22 (and 8080 for the Airflow UI) from Google's IAP range, 35.235.240.0/20.
Opening the tunnel needs one IAM role, and logging in needs a second one (OS Login).
Every SSH session shows up in Cloud Audit Logs under the person who opened it.
It is a Shielded VM (Secure Boot, vTPM, integrity monitoring), required by constraints/compute.requireShieldedVm.

For outbound traffic, Private Google Access covers *.googleapis.com, and
Cloud NAT handles everything else the VM needs for installs and git pull
(GitHub, PyPI, and dev.azure.com for the git remote).

Chapter 5: The ML pipeline

The repo has one sample pipeline. It shows the same training and prediction
flow we use for our own models.

The model code lives in app/ as a Python package, ml_experiment. It
trains a logistic regression on BigQuery's public penguins dataset. We build
it into a wheel, upload it to the ml-artifacts bucket, and a DAG submits
it to Vertex AI as a custom training job inside Google's prebuilt
sklearn-cpu.1-0 container. Here is a full run, from saving config to the
job finishing:

The Vertex job runs as the VM's own service account, which holds
roles/aiplatform.user, roles/bigquery.jobUser, and write access to its
staging bucket. Airflow gets its GCP credentials from the VM's metadata
server, so there are no key files on the machine.

Chapter 6: Deployment

The platform is Terraform, one folder per component:

terraform/
├── config_hq/     # Cloud Run, Load Balancer, IAP, config bucket
└── airflow_vm/    # VM, firewall, Cloud NAT, ml-artifacts + vertex-staging buckets

Each one uses Terraform workspaces, so dev, test and prod get their own
copies inside the same project, with the env name as a suffix
(airflow-vm-prod, example-project-config-hq-prod). A precondition
refuses to apply on the unnamed default workspace, so nobody creates
resources without an environment by accident.

Terraform only builds a bare VM. Airflow goes on with Fabric, from
deployer/fabfile.py, over the IAP tunnel. These are the tasks we use:

Task	What it does
`deploy_from_scratch`	Clones the repo, sets up Postgres and the conda env, installs Airflow, starts it, and creates the `config_hq_bucket` variable and `google_cloud_default` connection
`light_deploy`	`git pull` plus a rebuild of the app package. This is our normal deploy.
`restart_airflow`	Stops and starts the scheduler, DAG processor and API server
`complete_teardown`	Stops everything and removes the database, env and project files

Airflow's AIRFLOW_HOME points at the cloned repo, so a DAG change is a
commit followed by light_deploy. config_hq has no deploy script at all.
terraform apply builds the container with Cloud Build, pushes it, and
rolls out a new Cloud Run revision.

Chapter 7: Running it day to day

Access lives in Terraform variables, in three tiers:

Variable	Grants
`iap_tunnel_members`	open the tunnel
`oslogin_members`	log in as a normal user
`oslogin_admin_members`	log in with sudo (empty by default)

We put Google Groups in there, not individual people, so adding someone to
the team is a group change and not a Terraform change. If you ever grant
access by hand in an emergency, add it to terraform.tfvars afterwards, or
it drifts.

If the VM dies, almost everything that matters lives somewhere else. The
DAGs and the ML code are in git. Configs are in the versioned config_hq
bucket. Model wheels and Vertex AI outputs are in their own buckets. Getting
back is terraform apply on airflow_vm and then deploy_from_scratch,
which is safe to re-run if it fails partway. What does not come back is
Airflow's own Postgres database, with its run history. It lives on the VM's
disk, and nothing in the repo backs it up yet.

Tearing down goes in reverse: terraform destroy on airflow_vm first,
because it depends on config_hq's bucket, then config_hq. The IAP brand,
the OAuth client and the Terraform state objects stay behind and have to be
removed by hand.

Where it stands

It is running now and costs less than half of what Composer would.

There are still chores. Airflow upgrades and OS patching are on us. The
Postgres backups mentioned above are still missing. And config_hq uses a
self-signed certificate until it gets a real domain, so the browser shows a
warning every time someone opens it.

I hope you have enjoyed this one. Feel free to share your comments,
especially if you made the opposite call and stayed on Composer.

Regards,

Erfan

Designing Coding Agent Skills That Actually Work

erfankashani — Sat, 30 May 2026 17:37:57 +0000

I've spent the last few months obsessed with AI coding agents. Not just using them. Building alongside them and optimizing them for my use-cases. I've designed and tested various skills for agents like Claude Code, VScode Copilot, Codex, and Cursor, and I've watched closely to see where they hold up and where they quietly fall apart.

And they do fall apart. Usually in the same way. The agent does great for three steps, then does something dumb on step four, it starts backtracking and analyzing went wrong, and by the time it figures out the issue, the conetxt window is almost full, leaving you with 3 what-have-I-done-so-far.md files instead of a task done.

After enough of those moments, I stopped trying to make the agent smarter. I changed how I think about skill design entirely. This blog is about that shift, and the patterns that came out of it.

The skill that taught me this

The example I keep coming back to is a skill I built called confluence-publisher. The origin is boring but honest: I didn't have corporate access to a Confluence MCP server, so my agent had no native way to publish pages. Instead of waiting on IT, I built the bridge myself: a skill that takes markdown and publishes it to Confluence.

I used this skill design as my sandbox for my new skill framework. Every pain point I'd hit with other skills showed up here too, so I used it to work out what actually makes an agent skill reliable. The full code is on GitHub if you want to follow along:

🔗 GitHub: link

The mental shift

Now to the part that changed how I work.

When a skill misbehaves, the instinct is to constrain the agent: add more rules, more guardrails, more "do not do this" in the prompt. That's the wrong knob. You end up fighting the model.

The better move is to constrain the tools, not the agent. Make the things the agent reaches for, the scripts, deterministic and predictable. Then let the agent do the one thing it's genuinely great at: understanding what the user wants and reasoning about it. The agent reads intent and decides. The scripts perform the action.

This buys you two things. You get far fewer nondeterministic results, because the risky work runs as plain, tested code which you can itterate through and write tests for. Moreover, you shrink the agent's context window, because it's holding decisions, not implementation details. Most of what follows is just this idea applied in different places.

Scripts as decision boundaries

The most important choice in the whole design: the agent doesn't write code at runtime. It runs pre-written scripts and decides with the correct context.

Each step in the workflow is a standalone Python script — step_01_convert.py, step_02_publish.py — with a clean interface, just like any tool in the agent's toolkit. CLI arguments go in, structured output and exit codes come out. The agent's job shrinks to four things:

Decide what arguments to pass
Run the script
Read the output
Decide what to do next

That removes the ambiguity. Instead of asking the model to reason about Confluence API semantics, authentication, HTTP headers, and XHTML formatting on the fly, the scripts hold all that complexity. The agent only has to understand the contract: what goes in, what comes out.

In my tests, even a non-reasoning model like GPT-4o follows this without getting lost, because the SKILL.md reads like a recipe. Set your variables, run step 1, check the output, run step 2. There's no open-ended problem-solving at runtime, so there's very little room to improvise a mistake.

Failures that speak up

Every script is built to fail loudly and specifically. Exit codes aren't just 0 or 1 — they carry meaning:

Exit Code	Meaning	What the Agent Should Do
`0`	Success	Continue to next step
`1`	Conversion error	Check the markdown input
`2`	API/auth error	Verify credentials, run diagnostics

And when something breaks, the message isn't a useless "failed". It tells the agent exactly what's missing:

ERROR: --parent-page-id is required for create/upsert mode.
ERROR: Missing environment variables: CONFLUENCE_URL, CONFLUENCE_API_TOKEN
ERROR: HTML artifact not found: /path/to/expected/file.html

This is what lets the agent self-correct. If it forgot --parent-page-id, the error says so, and the agent fixes it. If credentials are missing, it knows to ask the user instead of guessing.

I took this one step further with a diagnose_permissions.py script. When step 2 fails on an auth error, the agent can run a dedicated diagnostic that tests connectivity, space access, and page-level permissions in sequence. Each test prints SUCCESS or FAILED on its own, so the agent can pinpoint exactly where the permission chain breaks, instead of guessing.

Artifacts as memory

One of the hardest problems in agentic workflows is the context window. A 16,000-character HTML document doesn't need to live in the agent's memory. It just needs to be reachable.

The artifacts/ folder handles that:

Step 1 converts markdown to XHTML and writes it to artifacts/{Title}_{RUN_TS}.html
Step 2 reads from that file path
The agent only holds the path, never the content

So the agent can move large documents through the workflow without burning context. It can also inspect an artifact between steps if needed (ex. read the HTML to check the conversion looks right before publishing). The naming convention ({Title}_{RunTimestamp}.html) makes each artifact self-describing, so if a run does something weird weeks later, you can find the exact file from that run.

Work-logs as governance

Every script invocation appends to a shared work-log: work-logs/worklogs_{RUN_TS}.md. This is more than logging — it's a record of what the skill actually did.

A typical entry looks like this:

## Step 02: publish
**Script**: `script/step_02_publish.py`
**Started**: 2026-05-28T18:36:24.677335+00:00

### Script Output
HTML artifact loaded: .github/.../Main_DAG_Analysis_20260528_143418.html (16631 chars)
Title: Main DAG Analysis
Space: LCA
Mode: create
Confluence connection established.
Creating page under parent 4180803775...
Page published successfully.
PAGE_ID=4238049513
PAGE_URL=https://company.atlassian.net/wiki/pages/viewpage.action?pageId=4238049513

This matters for a few reasons:

Error tracking: when a run fails mid-way, the log shows which step succeeded and which didn't, with timestamps and the exact parameters used.
Side-effect tracking: publishing a page is irreversible. The log records when it happened, what went out, and where it landed. If the skill fails on a later step, you know exactly what state it left behind.
Agent self-reference or self-improvement: on the next run, the agent can read past work-logs to see what happened before, what parameters worked, what failed, what the output looked like. It's a form of long-term memory that doesn't pollute the context with raw data and allows the agent to correct itself in case of past failures.

Secrets stay outside the context window

Credentials should not be exposed to the agent context window. The authentication should be done programatically and preferably from a service account designed for the agnet itself. The credentials can be read from secret managers and perform authentication aside from the agent's context. In this example, I am using a .env file that is not tracked on git for simplicity of demo. The scripts load the credentialls internally through _load_env_file(). The agent never sees the token value in its context.

def _load_env_file() -> None:
    """Load variables from the skill's .env file into os.environ."""
    env_path = Path(__file__).resolve().parents[1] / ".env"
    ...
    for line in f:
        key, _, value = line.partition("=")
        if key and key not in os.environ:
            os.environ[key] = value

The scripts are designed to check for required variables to be set and if they cannot identify them then a "Missing environment variables: CONFLUENCE_API_TOKEN" error would notify the agent.

A .template-env file shows the expected shape without real values:

CONFLUENCE_URL=https://your-company.atlassian.net
CONFLUENCE_EMAIL=your-company-email
CONFLUENCE_API_TOKEN=developer-api-token-get-from-confluence-site

The human operator manages the secrets, and the agent runs with whatever permissions the authenticated app already has. No OAuth flows, no token rotation for the agent to reason about. It just operates in the sandbox that is allowed to access.

Design for the floor, not the ceiling

As you notice, the thread running through all of this is that SKILL.md is written in a way to reduce confusion so that a model without chain-of-thought reasoning can also run it correctly. A few techniques make that work:

Numbered steps with exact commands. The agent doesn't figure out how to call a script — the exact command is right there with placeholder variables marked clearly.
One decision per step. Each step has one outcome to check (the exit code) and one branch to take (continue or stop).
No nested logic. There's no "if the page exists and the mode is upsert and the parent changed, then..." Each mode is independent, and the script handles its own logic.
Structured output. Scripts print JSON on success, so any model can parse the result without extra parsing logic.

This is a deliberate tradeoff. A stronger model could handle a messier interface, but by designing for the floor, the skill works across model tiers (even local models that run on the consumer devices).

When to reach for this (and when not to)

A fair question for this audience: if MCP exists, why build a skill out of scripts at all? I started here because I didn't have MCP access, but the two aren't really competitors. An MCP server hands the agent a set of tools. This pattern hands the agent an ordered recipe, plus on-disk memory and an audit trail of what it did. You can even run both, and point the recipe at MCP tools instead of local scripts.

The honest tradeoff: this works best for workflows that are mostly linear and have real side effects you want recorded. Publish, deploy, migrate, generate-and-ship. It's a poor fit when the work is exploratory or needs the agent to branch in ways you can't predict ahead of time. Pre-written scripts can't adapt to a situation you didn't script for, and writing them is upfront cost. If a task is a one-off, skip the ceremony and let the agent improvise. The moment it becomes something you'll run again and you care about the outcome, the structure pays for itself.

Wrapping up

If I had to compress everything I learned into one line: don't make the agent smarter. Make its tools more predictable. Let the agent reason about intent, and let deterministic scripts do the acting.

The same idea, broken down by what each piece does:

Principle	How it's applied
Remove ambiguity	Pre-written scripts with CLI contracts
Fail vocally	Specific exit codes + descriptive error messages
Manage context	Artifacts on disk, not in memory
Govern side effects	Work-logs capture every action with timestamps
Protect secrets	`.env` loaded by scripts, never surfaced to the agent
Design for simplicity	Linear steps, exact commands, structured output
Enable self-correction	Past logs readable by future runs

The result is a skill that a high-reasoning model can design, but a basic model can still operate. It runs predictably, and you can trace exactly what it did.

The full confluence-publisher code is here if you want to dig in or fork it:

🔗 GitHub: link

If you enjoyed this, feel free to follow me on LinkedIn — I share what I learn as I keep building.

Regards,

Erfan

Deploy Application on Azure App Services

erfankashani — Wed, 28 Dec 2022 21:16:53 +0000

Introduction

Hello, after a long break of blogging, I am dedicating myself to share my knowledge with the community via series of posts regarding different architecture designs that I have worked on throughout past year. These designs corresponds to my current learning journey and are open to feedbacks and discussions. Without further due, let's dig in.

Senario

Imagine you are in process of creating a product in form of web application, mobile application, or API endpoint. You have designed and prototyped your business logic in form of an executable package. While this application undergoes different levels of development maturity, you require a mean of presenting and demoing the proof of concept (POC) to your client/audiences with different technical skill levels. There are plenty approaches which can achieve this purpose. Some are mentioned below:

Ship the application as a package to be installed and tested on other developer's laptops.
Package and Deploy the application on an internal server, on-premises infrastructure or internal network for few techsavy audiences to test.
Deploy your application and serve on a public/private cloud provider and create instructions for how the end-user would interact with your app.

As you can see, you have many choices to deploy and present your application. One of the most important lessons I have learnt is that there will never be a silver-bullet tool, design, or architecture which works for all problems. Each project comes with sets of requirements and constraints. An architect or a developer has to make considerations and trade-off which will affect the product's end-goal. Some of the factors you can consider while choosing the correct approach for your product demo are time, skillset, budget, product's maturity level, and enterprise infrastructure availability.

having said that, let us assume we decided to deploy our application to a cloud provider (such as Microsoft Azure). There are yet different abstraction levels we can choose to lower the entry-cost of using cloud services. The price of these abstractions are to lose control over certain parts of the infrastructure your application will run on. Some of the available options are as such:

Deploy your application as a serverless architecture using Azure functions
Containerize and deploy the application using one of the container delivery services on Azure like App Services, Container Instances, or Kubernetes Services.
Deploy the application on a cloud hosted Virtual Machine

As mentioned before, there are situations which choosing one solution would be more beneficial than others. In this example, we would like to deploy our application with minimal architecture alteration and scalability in mind. We will try to containerize and deploy the application on Azure App Service.

Azure App Service

App Service is an Azure cloud offering which allows fast build, deploy, and scaling of Web, Mobile, and API applications. It supports .NET, .NET Core, Node.js, Java, Python, or PHP programs inside of containers. App Service provide a fully managed infrastructure which deals with maintenance, security patches, and scaling. With it's 99.95 percent service-level agreement (SLA)-backed uptime availability, App Service can be good candidate for some production grade applications. We will deploy our first App Service in the following section.

Requirements:

docker
az-cli

Steps:

Generate a new resource group: We will use this logical container to easier manage, monitor, deploy, scale, grant access of resources to a specific group or identity.
Create a new Azure Container Registry (ACR): This image registry hosts your docker images on Azure. For enterprise use-cases look into options such as Jfrog Artifactory.
Create an Azure App Service Plan
Create an Azure App Service
Upload your docker image to ACR and configure the App Service to use the image

We will configure the components using Azure CLI for more interactive approach. For more advance use-cases we can look into Azure ARM templates or terraform for future blogs.

Login using az cli

# install azure-cli on mac-os
brew update && brew install azure-cli

# login
az login

# check the active subscription:
az account show --output table

# change the active subscription (if needed):
az account set --subscription <subscription_name>

Create new resource group

Note: Naming conventions should follow consistent rules throughout your developments. It is a good idea to research your organizational naming conventions and follow them. I will use the following convention in a Camel Case format:

[Compname, Teamname, Projectname, Envname, Resourcelocation (abbreviated), Resourcetype (abbreviated)]
example: SpacexFlycontrolLandingtimeProdCcRg

# create a new resource group:
az group create --name DemoFlaskappDevCcRg --location canadacentral

# check the resource group list
az group list

Create Azure Container Registry (ACR)

# create ACR
az acr create --name demoflaskappacr --resource-group DemoFlaskappDevCcRg --sku Basic --admin-enabled true

# retrieve the registry's credentials
az acr credential show --resource-group DemoFlaskappDevCcRg --name demoflaskappacr

Note: Copy the "password" (not password2) and username in the step above as we will use this to login via docker.

Login and push the image into ACR

In this step you can use the boiler plate flask + docker project from github and follow the instructions:

# login to ACR 
docker login demoflaskappacr.azurecr.io --username demoflaskappacr

# clone the sample flask docker project
git clone https://github.com/erfankashani/docker_flask_app.git
cd docker_flask_app

# build the image
docker build -t docker-flask-app .

# run the image
docker run -p 8888:5000 docker-flask-app

This local container exposes to port 8888 on the host. The successful application can be checked on http://localhost:8888/ URL on your local computer's browser.

Now let us push this built image to docker registry:

# change the image name and tag to latest
docker tag docker-flask-app demoflaskappacr.azurecr.io/docker-flask-app:latest

# push to registry
docker push demoflaskappacr.azurecr.io/docker-flask-app:latest

# check the ACR for the pushed image
az acr repository list -n demoflaskappacr

Create App service

In this step we will create the App Service Plan and then an App Service which can pull images from the registry

# create app service plan:
az appservice plan create --name DemoFlaskappDevAsp --resource-group DemoFlaskappDevCcRg --is-linux

# create the app service (webapp)
az webapp create --resource-group DemoFlaskappDevCcRg --plan DemoFlaskappDevAsp --name DemoFlaskappDevAs --deployment-container-image-name demoflaskappacr.azurecr.io/docker-flask-app:latest

# expose the container's port 5000
az webapp config appsettings set --resource-group DemoFlaskappDevCcRg --name DemoFlaskappDevAs --settings WEBSITES_PORT=5000

# enable the system assigned managed identity (principle-id):
az webapp identity assign --resource-group DemoFlaskappDevCcRg --name DemoFlaskappDevAs --query principalId --output tsv

# retrieve az account subscription-id (subscription-id)
az account show --query id --output tsv


# grant the pulling access from ACR to the webapp managed identity (use principle-id and subscription-id from above)
az role assignment create --assignee <principle-id> --scope /subscriptions/<subscription-id>/resourceGroups/DemoFlaskappDevCcRg/providers/Microsoft.ContainerRegistry/registries/demoflaskappacr --role "AcrPull"

# configure the app to use the managed identity (use subscription-id from above)
az resource update --ids /subscriptions/<subscription-id>/resourceGroups/DemoFlaskappDevCcRg/providers/Microsoft.Web/sites/DemoFlaskappDevAs/config/web --set properties.acrUseManagedIdentityCreds=True

# configure the web app to use the correct docker registry:
az webapp config container set --name DemoFlaskappDevAs --resource-group DemoFlaskappDevCcRg --docker-custom-image-name demoflaskappacr.azurecr.io/docker-flask-app:latest --docker-registry-server-url https://demoflaskappacr.azurecr.io

Grab the URL from azure portal:

use the URL to access the app (please note that the first time load might take longer):

We have a deployed working version of this App Service. There are few more considerations which we make before demoing this application.

Networking

The applications hosted on Azure App Service are by default accessible through public internet and can only reachout to public endpoints. There are series of networking features which can help you define guidelines on the app's ingress (inbound traffic) and egress (outbound traffic). It is important to note that some of these capabilities are dependant on your mode of deployment (aka. single-tenant vs. multi-tenant plans). The following features are available on our multi-tenant service hosted App Service:

Ingress:

App-assigned address: Enables IP-based SSL checks and dedicated unshared inbound addresses for your app
Access restriction: Can limit the app access to specific well-defined addresses. This is a good idea for early demo phases when you would want to ensure your application can be viewed only by specific people.
Service endpoints and private endpoints: Expose a private endpoint and manage the access from resources within a VNet. There are more advanced techniques such as integrating your application with firewalls and load-balancer services like Azure Front Door. These applications are more useful for production and we can discuss them in future blogs based on interest levels.

Egress:

Hybrid connections: Access to private networks that are not connected to Azure.
Gateway-required Vnet-integrations: Access to resources in other VNets in different region via VNet peering.
VNet Integrations: This integration with Virtual Network opens the door to options such as configuring private subnets, exposing static IPs via NAT gateway, defining Network Security Groups (NSGs) for outbound traffic, integrating your app with a secure hub-and-spoke model to communicate with on-premises resources etc.

User Authentication

In this section we will enable user authentication using Microsoft AD Identity provider. Navigate to your App Service resource inside azure portal. Then click on authentication and "Add Identity Provider":

Select "Microsoft" as identity provider:

Your authenticator will require an app registration or service principal name (SPN) to communicate with the Identity provider. You have the preferred options of creating a new app registration from the same page or use an existing app registration followed by this link. In this example we:

allow application access to same tenant users only.
restrict app access to authenticated requests.
return 302 Error code on unauthenticated requests.
enable token storage.

Then press "add" which brings the following result:

To try this feature, attempt to access the URL again. This will require a Microsoft account login and the following pop up would show up:

Basic Scaling

there are two means of scaling your application to support higher number of requests:

Scaling-up: Means adding more resources like CPU, memory, and disk space to your existing app. You can achieve this by altering the App service Plan tier which your App Service is using.

Scaling-out: Create more instances of the VMs that run your application. You can scale out to as many as 30 instances, depending on your pricing tier. There are also autoscaling options to scale up and down your instances based on rules and schedules.

Thank you for following this blog to the end. In the next blog we will look into other aspects of this solution such as:

Secret management
Deployment options
Monitoring
Disaster Recovery

Resources:

Awesome diagrams using Mermaid.js

erfankashani — Thu, 14 Apr 2022 13:02:01 +0000

Today I want to share an awesome library which I discovered recently. Creating diagrams for complicated software can be a time consuming and tricky if you are not using the right tool. Maintaining the diagram to show the most recent structure can be struggle. One of the most common go-to is diagram.net:

This web application allow users to drag and drop elements on a page and rename them to create variety of graphs. In my experience, this application can create really beautiful graphs which take a long Time to create.

My search was not done there. I aim to introduce speed in the diagram creations, version control, and easier interface than finding elements, resizing, and renaming them plenty of times. This is where I found Mermaid

This library is written in Javascript and allows diagram creation using a markdown like syntax. Mermaid can create the following diagrams:

Flowchart
Sequence diagram
Class diagram
State Diagram
Entity Relationship diagram
User journey
Gantt
Pie Chart
Requirement diagram

Some of the benefits of using Mermaid.js:

creating graphs as code
saving time on styling the graphs
version controlling your graphs
easy integration with other libraries like diagram.net

To start we can install the “Markdown Preview Mermaid Support” extension on VSCode:

Then create a diagram directory in your desired codebase (optional). Finally create a markdown file like “app_diagram.md”. Choose the small (open preview to the side) icon on too right corner of your editor or ( shift + command + v for mac). Now you are ready to edit the file. Lets start with some easy examples:

The following showcases a Flowchart using mermaid:

graph TD;
            A-->B;
            A-->C;
            B-->D;
            C-->D;

Any mermaid code should be entered as a code snippet with “mermaid” tag. Each letter represents a node which are connected via edges.

There are many samples of diagram creation on the mermaid website. I am going to demonstrate a settup that worked for me while creating flowcharts:


<!-- ```mermaid
flowchart Guide
    direction TB
    f[flow]
    s[[Step]]
    i([input])
    d[(Database)]
    Parameters>Parameters]
``` -->
<!-- Resource: https://mermaid-js.github.io/mermaid/#/ -->
<!-- Look at the graph here: https://mermaid.live -->
<!-- Connect to diagrams.net: https://www.diagrams.net/blog/mermaid-diagrams -->

```mermaid
flowchart LR
    subgraph Guide
        direction TB

        f[flow]
        s[[Step]]
        d[(Database)]
        Parameters>Parameters]
    end

    %% add in-line style
    Guide:::someclass
    classDef someclass fill:#f96;


    %%list of parameters
    p1>bank_name]
    p2>seller_info]


    %%list of steps
    s1[[1-withdraw_money]]
    s2[[2-purchase_bike]]


    %%list of flows
    f1[go_to_bank]
    f2[withdraw_from_atm]
    f3[contact_seller]
    f4[trade_bike]


    %%list of databases
    d1[(reads: customer_identification)]
    d2[(reads/writes: customer_balance)]
    d3[(writes: e_document_sign)]


    %% Create the step flows
    s1-.->s2;


    %% s1 flow
    p1-->|inputs|s1
    s1-->|calls|f1
    f1-->|calls|f2
    f2-->d1
    f2-->d2


    %% s2 flow
    p2-->|inputs|s2
    s2-->f3
    f3-->f4
    f4-->d3
\```

There are also other resources where you can export your graphs as PNG or SVG like mermaid.live. Moreover, you can checkout this blog regarding how to transfer your graphs from mermaid to diagram.net (FYI, digram.net does not support all mermaid functionalities)

I hope you have enjoyed this tutorial. Feel free to express your comments.

Regards,

Erfan

Nail the Fundamentals : Git_Good (Git flow best practices)

erfankashani — Sat, 28 Mar 2020 19:06:46 +0000

Have you ever you wished you had a time machine to go back in time and fix that awkward thing you said to someone? or you wanted to choose a different path of decisions for your life? ... as much as I want to write a tutorial on time machines, This blog is about a valuable tool similar to a time machine for developers ...

Git is your best friend when it comes to coding and I regret not using it enough during my undergraduate education. Today I want to discuss the git flow I use day to day in my workplace and hope it brings value to your development journey.

Any repository has two main branches: Master and Development.

Master (Production): is the production code which is being used by the customers. This branch should be ideally bug free. This is not a playground to test new lines of code.

Development (Staging): is the next version of Staging (Master) environment. This means this branch is still close to the master but has one or more new features which are being tested.

To start a new feature in the code base we should:

1) Start with the latest version of the development branch : This is important to always keep your development branch up to date. In big teams you can usually followup with the newest merge requests via the slack or email. This indicates that someone will be altering the current state of development soon and you should pull the latest version of the code: (this avoids future merge conflicts).

to perform this task you can:

git checkout development
git pull

2) Create your new branch: You should create a new branch to work on your new feature and avoid working on development branch. There is usually a convention for naming your branches among your team. My teams's convention is as such: <initials>-<ticket-number>-<featurename> for example I would make my branch using following command:

git checkout -b ek-132-slackbotapirouting

3) Work on your branch: You can make any changes you like to this branch and make sure you commit your changes often as you can revert to older version of code when needed. The commit messages should be a one liner description of what has changes in that commit. This will help you later if you wish to change or squash your commits. Couple of useful git commands when adding and committing code:

git diff                                # will show you all the changes you made. Here if you think you don’t want to add a file just add it to the .gitignore file
git add .                               # will add all the changes 
git add -p                              # here you can see the changes one by one and write (y/n) to add or not add a change 
git commit -m “what this change does”   # to commit the added changes

4) Time to push the code up: When you have the final version of your code, documented, tested, and refactored, it is the time to add this feature or the development branch.

First we will rebase with the development branch to compare the latest changes on development with our branch and deal with any possible merge conflicts:

git checkout development
git pull
git checkout ek-132-slackbotapirouting
git rebase -i development.

You triggered the interactive rebasing. Here you can “reword” or “fixup” commits (basically squashing the commits ) and deal with code-conflict. After rebasing is done, it is time for pushing the code up to the development.

git push -f                       #Force pushing to the ek-132-slackbotapirouting branch

Currently you have your latest changes pushed to the remote branch and rebased with the current state of development branch. This means you will not face a merge conflict when you submit a merge request.

5) Create the Merge request: Now you can create a merge request by going on the web version of your remote repository (ex. Github or Gitlab). Assign the merge request to your software lead for review and copy the request link into the developer's chat for extra code-review. Enable the checkmark for deleting the branch after merging so your repository stays clean.

6) Delete the feature branch: It is always recommended to keep the repository clean.

Deleting the feature branch locally by:

git branch --delete ek-132-slackbotapirouting

 
Deleting the branch remotely by:

git push origin --delete ek-132-slackbotapirouting

Good Practices:

Delete your local and remote branch after merging to keep the repository clean
If you have commits like "fixed a bug" or "fixed white space" make sure to squash them before making merge request
Always rebase before submitting merge request to avoid merge-conflicts
Avoid committing specific files to your development machine or process(ex. Gemfile.lock, .env).

Nail the Fundamentals : Documenting Using Yardoc (YARD)

erfankashani — Fri, 11 Oct 2019 23:09:18 +0000

Disclaimer: Nail the Fundamental series attempt to discuss good coding practices. Theses are personal industry lessons, taught by senior developers and software architects to enhance my skills. Most of the topics are general coding principles while some of them focus on specific coding language.

If you are a junior developer, this is the right place for you as I aim to fill the gap between coding lessons taught in the academia and what happens in the the industry. If you are a senior developer, you can compare your methods with the ones practiced by different organizations in the comments.

Without further ado, In this tutorial we will look into YARD!

Not that yard ... but YARD as in (Yay, A Ruby Documentation Tool). YARD or Yardoc is a documentation generating gem for the Ruby programming language.

Before we jump into it, there is an important question to answer!

Why should I document?

That is a fair question and I carried it when I started working in a development team. Some of the most important reasons to document your codes are:

You will be using them in 6 months: Code that you wrote 6 months ago is often indistinguishable from code that someone else has written. Documentation is the key to understand the code's syntax and implementation.
Increasing usability of the code: If you writing a feature, the code most likely will be integrated into a bigger project. Therefore, you should document it to be easily understood and interpreted by other developers.
Enhancing the On-boarding process: In case of adding a new developer to your team or on-boarding a new department to use your internal tool, documentation reduces the time and effort for them to adapt the codebase.
Simplifying the debugging: Software logs help to indicate where to look for a bug and documentations allow pinpointing the issue in low-level.
Assisting with test-cases: Black-box testing which analyzes functionality of the software relies solely on the requirement specifications document of your code.

I hope by now you have enough reasons to start documenting your code. That definitely makes your code look more professional. so please ...

Why Yardoc?

Yardoc is one of the well known ruby documentation tools and its benefits can be summarized as:

Can be hosted on personal DSL and private servers.
Generates consistant documentation.
Customizable documentation template (using markup language).
Accepts metadata such as tags along with the documentation (similar to Python, Java, and Objective-C).

Installation

# First make sure Rdoc is installed (dependancy on Ubuntu)
sudo apt-get install rdoc

# Mac users don't need to install rdoc since it is installed with
# the newest ruby installation.

# Next, install YARD using gem:
gem install yard

# Finally, to ensure YARD is installed
yard help

Documentation Styles

Most common tags:

@param = Describes the input parameters and their types (variable name can place before or after the type).
- Example: # @param [String] name Inputs the name of the user
- Example: # @param [Numeric] age Inputs the age of the user
- Example: # @param list [Array<String, Symbol>] the list of strings and symbols.
@option = In case you dealing with different options for the same input (use this for hash or array-list).
- Example: # @param [Hash] area_parameter numeric Inputs
- Example: # @option area_parameter [Integer] :length
- Example: # @option area_parameter [Integer] :width
@return = Documents the return-parameter from a method.
- Example: # @return [String] greeting message for the user.
@since = Usually, since is used to show app versions.
- Example: # @since v2.0.0
@raise = Use it only if method raises any exceptions.
- Example: # @raise [ZeroDivisionError] expectation if dividing by zero
@see = Refer user to a URL or other class within the documentation for further explanation.
- Example: # @see www.yardoc.com
@note = Miscellaneous messages that are low priority.
@todo = Miscellaneous messages that are high priority.
@example = Demonstrates an example of how to call the method.
- Example:

# @example Raise an exception using a message from +debug_message+
#   raise AppDebug.fetch('job_number', app_name: 'ExampleApp')

Common practice

add the file .yardoc and /doc to your .gitignore sine they can be generated easily when needed.
Document your private and protected methods. They will not be seen or used by the users but it helps other developers working on the same codebase. It is important to assure the private methods are not visible if you plan to publish the documentations. You can hide them in generating phase by:
yardoc --no-private --protected
When documenting a method, first line holds the Description. Leave a blank # between the description and tags. For Example:

# this is the method's description. 
#
# @param [String] text_input input parameter
# @return [String] returns the printed text
def print_text(text_input)
    #some code ...
end

Keep the order of input parameters consistent with the param orders. This results in easier flow for the reader.
Wrap the text with + to highlight
+highlighted_text+

Hands-on Demonstration

The code is available on GitHub. clone the repository using:

git clone https://github.com/erfankashani/Yardoc_Practice.git

Lets look into the files inside:

cd Yardoc_Practice
nano practise.rb

Most of the tags and the best practices are demonstrated in the code. Feel free to use this file as a reference for yardoc. Next step is to generate the documentation and spin-up the server.

Generate Yardoc

1- Go inside of the repository which you are using
2- Create the documentation by: yardoc <name_of_the_file>, in our case:

yardoc practise.rb

3- Run the web page on your local network (Port:8808):

yard server

After this step, you will notice a new directory called 'doc'. This is where all the HTML and css files are kept to create the documentation webpage. Observe the documentations by typing localhost:8808 in your address-bar of a browser. Take a look at the available methods by choosing methods option from the top left menu. You should see something like this:

By now, I hope you have a good understanding of why we document and how we do it using 'yardoc'. As you follow these simple but effective practices mentioned in the series, you will notice your code quality's improvement. Take the first step by start documenting all your ruby codebases. Let's get ittt...

Resources

Long Story Short: Kubernetes

erfankashani — Mon, 30 Sep 2019 00:05:40 +0000

Disclaimer: Long Story Short series are personal learning experiences as I encounter different development journeys. The aim of these series are to summarize and simplify difficult development concepts and teach them in different levels of proficiency.

The most important lesson I want you to take from this blog is to understand what kubernetes is? and why we use it?

In the era of micro-service architecture, applications are structured as a collection of services which are loosely coupled, easily maintained, independently organized, and owned by smaller teams. This is possible by containerizing different services within an application and using API calls to communicate among them.

This all sounds great until your organization thinks of scaling up or achieving above 90% up-time for the application. Managing couple of containers is not an exhausting activity but when thinking of configuring, deploying, and maintaining 100s of containers simultaneously our approach should be re-evaluated. This is where Kubernetes comes in the picture and automates the container management tasks.

Background Information:

Kunernetes: Is an open-source software which helps managing and orchestrating containers.

It was first developed by google engineers and later donated to Cloud Native Computing Foundation (CNCF).
It is an orchestration tool to deploy, scale up, and manage multiple containers together, therefore automating and enhancing the maintenance process of micro-services. Kubernetes main benefits are:
- assist communicating among containers
- Deploy them properly
- Manage them carefully
- Auto scaling: Always making sure there is certain number of container's copies available.
- Load balancing: Distributing the traffic.
Kubernetes cluster are usually made up of two main parts:
- Master (schedules application services)
- Nodes (listens to the orders from the master)

Now that we have a bigger picture of kubernetes and why we use it, let's dive inside of the cluster architecture and learn about the different components.

Components

Master: Is the control plane (main controller) of the cluster. Usually, users communicate with the master through dashboard graphical interface (GUI) or the command line interface (CLI) via YAML config files. Master's responsibility is to manage all the node events to keep the overall cluster configuration stable and healthy. The multiple components of master are as such:

Kube-APIserver: The front-end of kubernetes which exposes API calls and deals with REST services so developers can communicate with the cluster.
ETCD: Manages state of the cluster. Try to look at it as the database of kubernetes. As a developer, you have to find a reliable method to backup these information in case of a master's crash.
Kube-Scheduler: As the name suggests, it schedules the newly created pods into the nodes based on the given resource requirements.
kube-controller-manager: consists of multiple controllers which manages the cluster and nodes. For example:
- Replication Controller: handles the replicas (series of identical pods)

NODE: Is a part of the kubernetes run-time environment and is hosted on a separate VM or physical machine than the master. Nodes have a series of pods inside of them which allow containers to run inside of them. Node responsibility is to monitor pods and report back to the master. Node's components consist of:

Pods: A logical collection of containers (like docker containers) which need to interact with each other for an application to work. Furthermore, Pods have series of cgroup, kernel space, union file systems inside of them.
Kubelet: Each node has a kubelet which makes sure required podSpects are running healthy on the node. This agent can then report back to the master. their other tasks are to:
- Communicate With API server.
- Execute pods' containers via a container-engine.
- Mount and run pod-volumes and secrets.
- Execute health checks and report back to master.

Podspec: Is a YAML file that describes what each pod should have.

Good to note that kubelets usually use an endpoint to communicate such as tcp:10255.

Kube-proxy: Allows network traffic finds the right node and right pod inside (learn more about networking).

I hope by now you have a high level understanding of different kubernete's components and their role to enable orchestration.

Cool beans, but how do I control and use all of these?

As a devops specialist, you should be able to talk to this kubernetes beast by configuring certain controllers. Therefore, you can spin up your cluster and run your containers reliably inside.

Controllers

Controllers help with application reliability, scaling, and Load balancing. We need to understand different kinds of controllers within kubernetes:

Replicates: Ensures the certain number of pods are running at all times. We use replica-sets to scale up and scale down the pods.
Deployments: Declare updates for pods and replicates. Basically, a deployment manages replicates and replicates manage pods.
DaemonSets: Ensures all nodes copy a specific pod. Monitoring the pods on the nodes to make sure there are all identical.
Jobs: Usually a supervisor process. Jobs are individual processes that happen once or periodically such as database backup.
Services: Allow communication between a set of deployment. Services can be:
- Internal: IP only reachable within cluster.
- External: Endpoint available from node ip.
- Load balancer: Expose the application.
Labels: As the name suggests, it helps us to label and organize components. you can search, delete, and run services based on their labels. We use selectors to find labels.
Selectors: You can use them in two different method:
- equality-based selectors: (==, !=)
- Set-based selectors: (IN , NOTIN, EXISTS)
Namespaces: Give authentication and authorization to different users. extremely useful for bigger teams.

Congratulations! now you know the basic knowledge needed to start getting your hands dirty with the code...

Next step .... code