DEV Community: Ramatech Innovations

Reverting Git Won’t Undo Your Database Migration

Ramatech Innovations — Wed, 04 Nov 2026 06:17:17 +0000

A release introduces an application bug. You revert the deployment configuration, and the previous container image starts successfully.

But requests still fail.

The application has returned to its previous version. The database has not.

For stateful applications, a deployment rollback and a recovery plan cover different things.

A simple column rename can break rollback

Consider an application that reads this column:

SELECT customer_name FROM orders;

A new release renames it:

ALTER TABLE orders
RENAME COLUMN customer_name TO billing_name;

The new application expects billing_name. The previous application expects customer_name.

If the release fails after the migration, deploying the previous image leaves it querying a column that no longer exists.

The same compatibility problem can appear during a rolling deployment, while old and new application instances run together.

This is an illustrative example. The locking and operational impact of schema changes depend on the database engine and workload.

Separate schema evolution into stages

The expand, migrate, contract pattern spreads a breaking change across compatible steps. GitLab documents this approach for changes that must work across application versions. GitLab's compatibility guide

For this example, a staged approach could look like:

1. Expand

Introduce the new column while keeping the existing one.

Old application instances can continue using the original schema.

Adding a column still needs operational review: defaults, constraints, table size and database behavior can affect deployment safety.

2. Migrate

Introduce application behavior that supports the transition.

Define how both representations stay consistent while older instances remain active. A one-time backfill is insufficient if ongoing writes can immediately make the columns disagree.

Backfill existing records in controlled batches where appropriate. Make progress observable and retries safe.

Check completeness and consistency before switching reads.

3. Contract

Remove the old representation only after its consumers have stopped using it and the intended rollback window has closed.

Consumers may include background workers, reports, integrations and scheduled jobs—not just web application instances.

Once the old column is removed, an older application version may no longer be a valid recovery target.

Keep GitOps reconciliation in the recovery plan

A manual change to a running Deployment can conflict with the desired state maintained by a GitOps controller.

Decide how recovery changes the authoritative configuration, and how reconciliation behaves during the incident.

Also distinguish a Git revert from the deployment tool's rollback operation. They are separate mechanisms with their own behavior and constraints. Check the documentation for your deployed version, such as Argo CD's automated sync policy.

Neither operation inherently reverses database writes or external side effects.

Test the previous application against the migrated database

A useful compatibility test is:

Create representative starting data
    → Apply the proposed migration
    → Run the previous application version
    → Exercise important reads and writes
    → Run the new application version
    → Verify transition behavior

For a rolling release, also test the period when both versions operate concurrently.

A successful migration command proves that the database accepted the change. It does not prove application compatibility.

Write down where rollback stops being safe

Before release, answer:

Which application versions support the resulting schema?
Can migration jobs resume safely after interruption?
What happens to writes made during the transition?
At what step does recovery require a forward fix?
If restoration is necessary, what happens to data written since the backup?

The recovery target is a compatible application-and-data state.

How does your team define the rollback window for releases that change the database?

[Boost]

Ramatech Innovations — Tue, 03 Nov 2026 04:23:18 +0000

youtu.be

Your RAG Answer Can Be Correct—and Still Leak Data

Ramatech Innovations — Tue, 03 Nov 2026 04:22:35 +0000

A RAG assistant answers a question accurately and includes the right citation.

There is still a critical question to ask:

Was this user allowed to read that source?

Imagine an internal assistant searching engineering runbooks, customer contracts, and HR documents. All three may contain relevant information. They do not necessarily share the same audience.

Retrieval relevance and access permission are separate decisions.

The failure starts before generation

Consider this pipeline:

User question
    → Search the entire document collection
    → Send matching chunks to the model
    → Ask the model to hide restricted information

The restricted content has already reached the model. A prompt instruction cannot replace an authorization check.

The application should establish which content the user may access before constructing the generation context.

Microsoft's document-level access control documentation describes approaches for restricting search results using document permissions and security filters. The exact implementation depends on the data source and search platform.

Carry permissions into the retrieval layer

When a document is split into chunks, its access rules must remain enforceable.

For example, a chunk might carry metadata such as:

{
  "chunk_id": "runbook-42-chunk-3",
  "document_id": "runbook-42",
  "tenant_id": "tenant-a",
  "allowed_groups": ["platform-operations"],
  "permission_version": 7
}

This is illustrative metadata, not a complete authorization implementation.

The backend must derive the requesting user's identity and permissions from a trusted authentication system. Accepting an arbitrary tenant ID or group list from the browser would undermine the control.

A useful flow is:

Authenticate user
    → Resolve tenant and access permissions
    → Retrieve within the authorized scope
    → Build context from authorized chunks
    → Generate answer
    → Return permitted citations

The authorization rule must apply across every retrieval path, including keyword search, vector search, and fallback queries.

Permission changes are data changes

A document can remain unchanged while its permissions change.

Someone moves teams. A customer contract becomes restricted. A user loses access to a workspace.

If the retrieval index still contains yesterday's permissions, today's answer may expose information the user can no longer read.

Define how permission updates reach the index, how quickly revocations take effect, and what happens when permission information is missing or stale.

The same question applies to cached answers: can a response generated for one user be returned to another? Cache reuse must respect the requesting user's current access rights.

Test the denial paths

A useful evaluation set includes more than questions the assistant should answer.

Check these cases too:

Two users ask the same question but have different document access.
Two tenants have documents with similar titles.
A user's access is revoked after an answer is cached.
A chunk has missing permission metadata.
No authorized source supports the requested answer.

Inspect retrieved chunks and citations as well as the final response. A polite refusal does not prove that restricted content stayed out of the pipeline.

Make access boundaries part of answer quality

A production RAG evaluation should ask:

Did retrieval find useful evidence?
Did the answer accurately represent that evidence?
Was every included source authorized for this user?

Getting the first two right is not enough.

How are you testing permission changes and cross-tenant retrieval in your RAG applications?

Your CI/CD Pipeline Is Green. Did the Deployment Actually Work?

Ramatech Innovations — Mon, 02 Nov 2026 03:59:40 +0000

A successful build, an accepted manifest and a working user journey answer three different questions. A deployment pipeline should make those differences visible.

Consider an example release: the container image builds, the registry accepts it, and the deployment command exits successfully. The pipeline turns green. Meanwhile, the application cannot complete its main request because a dependency is unreachable.

The pipeline reported what it checked. The missing piece was the definition of success.

Define four release checks

Check	Question
Artifact identity	Are we deploying the intended build?
Workload rollout	Did the workload converge to the intended state?
Application behavior	Does a representative request produce the expected result?
Operational observation	Are errors and latency acceptable after release?

These checks complement one another. None proves the entire release is correct.

1. Connect the running release to its build

Record the source revision, image digest, target environment and deployment identifier together. If you use mutable image tags, a tag name alone may not identify the artifact you intended to release.

Choose a way to identify the running application version. A version endpoint, build metadata or a deployment annotation can help, but decide what it proves. An annotation records an intention; a response from the application can provide different evidence.

Keep credentials and sensitive configuration out of release reports.

2. Wait for the rollout, with a deadline

For a Kubernetes Deployment named web in namespace demo, a rollout check can look like:

kubectl rollout status deployment/web -n demo --timeout=180s

Replace the example resource names and choose a timeout appropriate to your workload. The command's success describes the rollout conditions Kubernetes evaluates; it is not an end-to-end business test.

Ensure a failed check actually fails the pipeline. A command placed in a script that ignores its exit status is not an effective gate.

If another system reconciles deployment state, such as a GitOps controller, wait for the intended revision through that system as well. Avoid assuming that a successful repository update means the cluster is already running it.

3. Test through the intended application path

A representative check should exercise the behavior users depend on. For a read-only HTTP check, a starting point is:

curl --fail --silent --show-error \
  --connect-timeout 5 --max-time 15 \
  https://app.example.com/health

The URL and timeouts are placeholders. curl's --fail makes HTTP responses of 400 or higher fail in ordinary cases; a successful exit does not validate response content or prove business correctness.

A shallow health endpoint may not exercise a database, an authorization rule or a downstream API. Add explicit assertions appropriate to a safe test request. Check the status, expected fields and result, not just whether any response arrived.

For operations that write data, use an approved synthetic test design with clear ownership and cleanup. Do not retry a business transaction blindly.

4. Observe after the initial check

A single request is a small sample. Define an observation window and the signals that matter: request failures, latency, dependency errors and relevant application outcomes.

Decide who receives an alert and who can stop or reverse a release. A rollback command is only part of recovery; configuration changes and database migrations may require a different approach.

Do not treat an automatic rollback as proof that every effect of the release has been reversed.

Make the pipeline result explainable

Prefer a release summary that says which checks passed, which revision they evaluated and when they ran.

For example:

Artifact identity recorded.
Intended workload rollout completed.
Representative request assertions passed.
Observation window completed within agreed thresholds.

These are proposed report fields, not results from a real deployment. They make it easier for an operator to distinguish “deployment accepted” from “release behavior verified.”

The useful question is not whether the pipeline is green. It is what evidence made it green.

References

Your RAG Answer Is Wrong. Check What Reached the Prompt Before Changing the Model.

Ramatech Innovations — Mon, 26 Oct 2026 16:19:35 +0000

A retrieval-augmented generation system can return a fluent answer to the wrong question, cite an outdated document, or invent a detail that its sources never supplied. Those failures can look similar in the chat window. They need different fixes.

Before changing the model, follow one request through three checkpoints: indexed evidence, retrieved evidence, and context actually delivered to the generator.

A small example

Suppose an internal assistant is asked:

How long are application audit logs retained?

Imagine two documents in its knowledge base. An older handbook says 30 days; a current policy says 90 days for application audit logs, with separate rules for other categories. The assistant answers “30 days.” These numbers are illustrative policy values, not retention recommendations.

A stronger model might still repeat the old policy if that is the only evidence it receives. First determine where the current information disappeared.

1. Check the evidence before the search

Locate the current source document. Confirm that ingestion succeeded and inspect the extracted text, not just the original PDF or page.

Does the extracted text include the 90-day rule?
Is “application audit logs” still attached to that rule?
Are headings, tables or exceptions split in a way that changes the meaning?
Can each chunk be traced to a document version and section?

A successfully uploaded file is not proof that the right passage is searchable. If extraction lost the table containing the answer, changing a generation prompt will not recover it.

2. Inspect retrieval separately

For the exact failing question, inspect the returned passages and their identifiers before generating an answer.

If only the old handbook appears, investigate collection selection, metadata filters, query wording, chunk boundaries and any reranking step.

Do not assume that increasing the number of results will solve the problem. It may introduce more obsolete or irrelevant passages. Compare the actual passages returned before and after each change.

Version metadata is useful only when your application applies it correctly. A recent ingestion timestamp is not necessarily a policy's effective date.

3. Inspect the assembled context

The search results are not always the same as the model input. A later stage may drop, shorten, reorder or summarize passages.

Checkpoint	What to inspect
Retrieval output	Was the current passage returned?
Context assembly	Did it survive filtering and the context budget?
Generator input	Did the model receive the rule, category and effective version together?

If retrieval returns the current policy but prompt assembly omits it, the immediate problem is between retrieval and generation.

Use access-controlled traces. Avoid copying confidential passages or personal information into unrestricted logs.

4. Evaluate the answer against its evidence

Once the correct passage reaches the generator, inspect what the answer does with it.

“All logs are retained for 90 days” is still incorrect in this example: it expands a category-specific rule into a universal statement.

Check whether each important claim is supported by the supplied passage. Inspect citations directly; a citation marker alone does not establish that a source supports the sentence.

For conflicting or insufficient evidence, define how the assistant should respond. It might explain the conflict, ask which log category the user means, or say it cannot determine the answer. A fallback instruction defines intended behavior; it does not guarantee compliance.

5. Keep a small regression set

Save representative questions with expected evidence and answer requirements. Include:

A straightforward answerable question.
A question affected by an old and a current document.
A question with an important exception.
A question the available documents cannot answer.

Record retrieval quality and answer support separately. Correct-looking answers can hide poor retrieval, while relevant passages can still produce unsupported answers.

Change one component at a time and compare the same cases. A useful experiment might adjust a version filter, a chunking rule or context assembly. Keep other settings stable enough to interpret the difference.

A practical debugging order

Start with source availability, then extraction and indexing, retrieval, context assembly, and finally generation. This is a diagnostic order, not a rule that every failure belongs to exactly one stage.

Identify the first point where the evidence stops supporting the requested answer. That gives you a concrete change to test instead of another blind model swap.

OpenShift Pod Running, but the Application Is Unreachable? Check the Service Port

Ramatech Innovations — Sat, 24 Oct 2026 12:11:28 +0000

A Pod showing Running does not prove that users can reach its application. One possible cause is a mismatch between the application's listening port and the Service's targetPort.

The example

Assume the application listens on TCP port 8080 and the Service exposes port 80. The Service incorrectly forwards traffic to port 80 on the selected Pods.

This is one possible diagnosis, not a conclusion to apply to every connectivity failure.

1. Inspect before changing anything

Replace the example namespace and resource names with your own:

oc get pods -n demo -l app=web -o wide
oc get service web -n demo -o yaml
oc get deployment web -n demo -o yaml
oc get endpointslices -n demo -l kubernetes.io/service-name=web

Confirm that the Service selector matches the intended Pods and inspect endpoint readiness. Check the application's actual listening port using its configuration and logs. Declaring a containerPort does not, by itself, make an application listen on that port.

2. Recognize the mismatch

The incorrect part of the Service could look like this:

ports:
  - name: http
    port: 80
    targetPort: 80

Here, port is the Service port; targetPort is the destination port on the selected Pods. They do not have to be identical.

3. Correct the source configuration

If the application actually listens on 8080, the corresponding Service configuration should be:

ports:
  - name: http
    port: 80
    targetPort: 8080

These YAML blocks are fragments, not complete Service manifests. Preserve the selector and other fields. Update the manifest in your source repository and deploy through your normal change process. Where a controller manages the Service, change its source configuration rather than relying on a manual edit that may be reverted.

4. Verify from inside the cluster

From an existing diagnostic Pod with curl installed:

oc exec -n demo <diagnostic-pod> -- curl -v --connect-timeout 5 http://web:80/

The example assumes that the diagnostic Pod is in the same namespace as the Service. Use a path with a known expected application response. An HTTP error response and a connection timeout are different outcomes; interpret the result rather than treating every non-200 response as a networking failure.

If the request still fails, investigate the application bind address, Pod readiness, Service selectors and NetworkPolicies. For external access, separately inspect the OpenShift Route, its selected Service port and any TLS configuration.

Takeaway

Check each connection separately: application listener, Pod readiness, Service selection, port mapping and external routing. Avoid changing multiple layers at once without evidence.

Reference

Kubernetes: Debug Services