Your Engineering Dashboard May Be Lying to You

Basavaraju Kidiyappa — Fri, 02 Oct 2026 10:29:54 +0000

Your Engineering Dashboard May Be Lying to You

Velocity is up.

Automation coverage is up.

Code coverage is up.

Defects are down.

Everything looks great.

But are we actually building better software?

That question has been on my mind for a long time.

After working across Engineering, Quality Engineering, Delivery, DevOps, cloud transformation and engineering leadership, I've noticed a recurring problem:

We measure engineering performance using metrics that are individually useful but collectively misleading.

A metric can improve while the engineering outcome gets worse.

When 85% automation coverage isn't necessarily better

Imagine an engineering organization moves from:

60% automation coverage → 85%

The dashboard shows a clear improvement.

But at the same time:

Defect leakage increases
Production incidents increase
Test maintenance effort increases
Critical user journeys remain poorly covered

Did automation actually improve?

The answer isn't obvious anymore.

The problem isn't the automation metric.

The problem is looking at it in isolation.

What about 90% code coverage?

Code coverage is another good example.

Suppose a team reaches:

90% code coverage

That sounds impressive.

But coverage doesn't tell us:

Whether the right scenarios are being tested
Whether assertions are meaningful
Whether critical business paths are protected
Whether the code is maintainable
Whether defects are escaping into production

Two codebases can both report 90% coverage and have very different levels of quality.

Again, the metric isn't necessarily wrong.

Our interpretation of the metric may be incomplete.

And then AI changes the equation

AI makes this even more interesting.

Imagine an engineering team using AI coding assistants and suddenly producing twice as much code.

That's a significant productivity signal.

But did engineering value increase by 2×?

Not necessarily.

We also need to understand:

Defect trends
Code quality
Review effort
Technical debt
Test effectiveness
Production reliability
Customer impact

AI can increase the amount of software produced.

The harder question is whether it increases the value of software delivered.

The problem with engineering dashboards

Most engineering organizations have no shortage of metrics.

They measure:

Delivery

Sprint velocity
Planned vs. completed work
Cycle time
Capacity utilization

Quality

Defects
Defect leakage
Severity
Defect aging

Automation

Automated tests
Automation coverage
Execution frequency
Test pass rates

Code

Code coverage
Code quality
Technical debt
Maintainability

Engineering productivity

Pull requests
Review time
Deployment frequency
Lead time

AI

AI-assisted coding
Code generation
AI test generation
Developer adoption

All of these can be useful.

But here's the problem:

They are usually viewed as separate dashboards.

That's where the real opportunity is.

The signal is in the relationship between metrics

Consider a simple example.

Metric	Previous	Current
Velocity	40 SP	55 SP
Automation	60%	85%
Code Coverage	72%	90%
Defect Leakage	8%	14%

If we look at the first three metrics independently, the story looks fantastic.

But once we connect them with defect leakage, the story becomes much more interesting.

Something changed.

And that's the question an engineering leader should investigate:

Why?

Maybe the organization is optimizing for delivery metrics.

Maybe automation is focused on the wrong areas.

Maybe tests are providing coverage without meaningful validation.

Maybe AI-generated code increased output faster than review and quality processes could adapt.

Maybe the organization is simply measuring the wrong things.

The dashboard doesn't give us the answer.

But it should help us ask the right question.

From measuring metrics to understanding engineering systems

This is the thinking behind OpenVector — Software Engineering Metrics Unleashed, an open-source project I'm building.

OpenVector on GitHub

The idea is simple:

Don't just collect more metrics. Connect the metrics you already have.

OpenVector looks at engineering through multiple dimensions:

📈 Delivery

Sprint velocity, capacity, planned vs. completed work

🧪 Quality

Defects, leakage, severity, aging and sources

🤖 Automation

Automation coverage and execution trends

💻 Code Health

Code coverage and code quality

👥 Capacity

Engineering capacity and productivity signals

🧠 AI Impact

Understanding AI adoption alongside engineering outcomes

The objective isn't to create another dashboard full of charts.

It's to make relationships between engineering signals visible.

The metric isn't the outcome

This is perhaps the most important distinction.

Velocity is not value.

Automation coverage is not quality.

Code coverage is not test effectiveness.

AI-generated code is not engineering productivity.

A declining defect count is not automatically improving quality.

These metrics are signals.

The engineering outcome emerges from how those signals interact.

That's why I believe the next generation of engineering intelligence needs to move from:

Metric → Dashboard

to:

Metrics → Relationships → Insights → Decisions

What should engineering leaders measure?

I don't think the answer is simply "more."

In fact, I think we should ask a different question:

Which relationships between metrics help us make better engineering decisions?

For example:

Velocity + Defect Leakage

Can increased delivery speed be achieved without sacrificing quality?

Automation + Defect Leakage

Are we automating the right things?

Code Coverage + Defects

Is increased coverage translating into fewer escaped defects?

AI Adoption + Code Quality

Is AI-assisted development improving engineering outcomes or simply increasing output?

Capacity + Delivery

Are teams becoming more efficient, or are they simply working harder?

Those relationships are much more interesting than any individual number.

More metrics ≠ more insight

Engineering organizations don't necessarily need another hundred metrics.

They need better ways to understand the metrics they already collect.

More metrics ≠ more insight.

Better-connected metrics = better decisions.

That's the problem I'm exploring with OpenVector.

If you're an engineering or technology leader, I'd be interested in your perspective:

Which engineering metric do you trust most when making a serious decision?

And perhaps more importantly:

Which metric does your organization measure simply because it's easy to measure?

Explore OpenVector on GitHub

EngineeringMetrics #SoftwareEngineering #QualityEngineering #AI

Practical AI Adoption in Test Automation

Basavaraju Kidiyappa — Wed, 27 May 2026 06:44:49 +0000

After years leading engineering and quality organizations, I wanted to explore practical AI adoption by building something hands-on.

Most conversations around AI in testing focus on:

AI-generated tests
Autonomous testing
self-healing magic

But in real engineering systems, reliability, transparency, scalability, and cost still matter.

The Classic Problem: Traditional automation frameworks come with several pain points. Complexity causing maintenance overhead, Lot of coding, Creating skill dependency. Duplicated logic, Test sprawl. Yes, there are commercial solutions to these, well they are expensive, causes vendor lock-in and less flexible and not scalable.

AI testing problems: Smart IDEs, Agentic QA, Tool calling all are evolving in making life easier, however the challenges of Inconsistent usage across team, cloud LLM dependency, expensive test building and execution, over reliance on AI are some of the persistent problems.

That led me to build OpenSecant — an AI-native resilient automation framework built on Playwright.

The goal was not to replace deterministic automation with AI, but to augment automation intelligently where it adds value.

What OpenSecant currently supports:

✅ Natural language-driven test execution
✅ Local-first locator intelligence
✅ LLM fallback when local resolution fails
✅ Self-healing selectors
✅ Step memory / reusable automation intelligence
✅ QA agent mode for exploratory workflows
✅ Parallel execution support
✅ Cloud + local LLM providers (OpenAI, Azure OpenAI, Bedrock, Ollama)
✅ Test Report with screenshots

... and More on the way.

Architecture:

Key Concepts:

Simplicity:
No more feature files, no more step definition, just test files and step to code mapping. catering to all roles QA, Dev, BA etc.

Deterministic-first approach:
Use of local locator engine to resolve locator identification reducing dependency on LLM calling

Self-healing philosophy:
On error fall back to Locater engine, pass the Hybrid element structure combining DOM structure and Accessibility snapshot. If the suggested locator fails, then 2nd level fall back to AI calling with local LLM or cloud, with retries in each layer

Local LLM support:
Cloud LLMs are Expensive - I have been using cloud LLMs from multiple providers. I have realized when you have 1000s of tests to build run, heal, relying on cloud models is expensive.

Data Privacy, Security by Design - Local LLMs are way to go when your tests are to deal with sensitive data

Tool Calling Flexibility - Easy to test integration in Internal Environments, Databases etc.

Enterprise Adoption - Easier to Scale

I used Ollama with models like phi4-mini, gemma4-e2b etc. They would need a machine with 8 GB GPUs to respond in seconds. This is getting to norm lately on local machines or pipeline set ups. Which makes scaling easier.

Integration with OpenAI, Bedrock and Azure exist, use with caution of cost.

Agentic QA:
Agentic QA if the future of Quality Engineering and I have taken a strong step towards it. Use cases like autonomously exploring sites or apps, building and saving tests for future execution

The working code:

## Roadmap:

Smarter healing strategies
Better agentic exploration
MCP integrations
Distributed execution
Plugin ecosystem
Extending to API and Native Mobile App Automation
Multi Agent Orchestration

DEV Community: Basavaraju Kidiyappa

Your Engineering Dashboard May Be Lying to You

Your Engineering Dashboard May Be Lying to You

When 85% automation coverage isn't necessarily better

What about 90% code coverage?

And then AI changes the equation

The problem with engineering dashboards

Delivery

Quality

Automation

Code

Engineering productivity

AI

The signal is in the relationship between metrics

From measuring metrics to understanding engineering systems

The metric isn't the outcome

What should engineering leaders measure?

Velocity + Defect Leakage

Automation + Defect Leakage

Code Coverage + Defects

AI Adoption + Code Quality

Capacity + Delivery

More metrics ≠ more insight

EngineeringMetrics #SoftwareEngineering #QualityEngineering #AI

Practical AI Adoption in Test Automation