DEV Community: MeL

Midweek Elevate: Raise the Baseline

MeL — Wed, 07 Jan 2026 20:24:25 +0000

The week is far enough along that the plan has met reality, but not so far that it is too late to change the outcome. It is the point where momentum matters more than intention.

I have had "Elevate" by Machel Montano featuring Nailah Blackman on repeat. Not for the lyrics as much as for the energy behind it. It feels like a reminder that we can lift our level, carry the moment forward, and let the movement pull more out of us than we expected.

That idea translates cleanly into software work.

Not as hustle culture. Not as "push harder."

As a design choice: raise the baseline.

Because the baseline is what you live in every day.

Elevation Is Not a Sprint

Most of us do not need a brand new system midweek. We usually need a slightly better system than Monday.

Elevation looks like:

a rough edge removed
a default that stops surprising people
a process that becomes lighter to carry
a decision that reduces the number of decisions later

When we raise the baseline, we are not chasing peaks. We are reducing friction.

Three Places I "Elevate" Midweek

1) Elevate the Baseline (the daily path)

Midweek is when I ask myself: _What is making the work heavier than it needs to be?

Then I look for the smallest change that improves the default experience:

renaming the confusing thing
tightening validation
moving the logic to where it belongs
reducing the number of clicks
removing a repeated manual step

It is not glamorous, but it is reliable.

If you want a technical framing: I am trying to reduce the "activation energy" required for us to do the right thing.

Less friction → more consistency → better outcomes

2) Elevate the Signal (what matters right now)

Midweek is also where noise accumulates:

too many open tabs (literal and mental)
too many half-decisions
too many "we will circle back" notes

Elevation here is clarity:

picking the one outcome that will make Friday easier
turning it into a small list of next actions
deleting, deferring, or delegating the rest

Sometimes elevation is simply choosing what not to do.

3) Elevate the Team (share the lift)

Energy spreads. So does confusion.

When a feature is stuck, a teammate can either inherit the confusion or inherit clarity. Midweek is a good time to leave clarity behind:

a short note describing current behavior vs expected behavior
a screenshot of the state that feels "off"
a 2–3 sentence summary of the decision you made (and why)
a small "here is what I would try next" suggestion

Elevation is not only self-improvement. It is making the work easier to carry for the next person.

A Midweek Elevate Ritual (15 minutes)

When I want something concrete, this is a midweek reset that does not pretend the week is starting over:

Name the lift: What does "better by Friday" actually mean?
Remove one drag: Fix a small recurring friction point.
Close one loop: Merge the small PR, answer the question, write the note.
Pick the next beat: Identify the next smallest step that keeps momentum.

It is not a reinvention. It is a lift.

The Point

When I think about "elevate" in work, I am not thinking about intensity. I am thinking about trajectory.

Midweek is a checkpoint, not a verdict.

We do not need a perfect week to have a meaningful one. We only need the next lift.

What is one small thing you can elevate today that your Friday self will feel immediately?

Turning the Page (Without Resetting the System)

MeL — Fri, 02 Jan 2026 22:40:01 +0000

A new year is not a reset button.

There is no git reset --hard on experience — no clean slate that wipes away what worked, what failed, or what quietly taught us something important.

Instead, a new year feels more like a continuation:

the same codebase,
more context,
better instincts, and
hopefully, fewer self-inflicted bugs.

What changes is not the past, but how we interpret it.

Discernment Over Reinvention

For a long time, I thought growth meant constant reinvention—new goals, new tools, new habits, new versions of myself. But over time, I have learned that real progress often comes from discernment, not reinvention.

Knowing:

what to keep,
what to refactor,
what to deprecate, and
what no longer needs to be carried forward.

That applies just as much to systems as it does to careers, workflows, and personal expectations.

Building With Honesty

This year arrives gently.

It does not demand perfection.
It asks for honesty.

Honesty about:

where my energy actually goes,
what kind of work sustains me,
which problems are worth solving, and
where I need to slow down instead of pushing harder.

Alignment matters more than velocity. Clarity matters more than noise.

Turning the Page

So I am not starting over.
I am turning the page.

With more focus.
With clearer boundaries.
With curiosity intact.

Still learning.
Still unlearning.
Still refining.

Growth is not always loud — but it is always cumulative.

Happy New Year!

The Bug That Was Not a Bug

MeL — Sun, 21 Dec 2025 19:18:29 +0000

When the problem is between the keyboard and the chair

I spent an hour yesterday debugging why my API endpoint was returning 404 errors.

I checked the routes.
I verified the controller.
I tested the database connection.
I even restarted the server three times.

The endpoint was fine.
The code was fine.
The server was fine.

I was calling the wrong URL.

It happens. We all do it.

But there is something about those moments — when I realize the bug that I have been hunting does not exist — that makes me pause.

How much time do we spend solving problems that are not actually problems?

The Real Question

This was not about a typo in a URL.

It was about the process.

When something does not work, our first instinct is often to assume the code is wrong. We dive into logs, check configurations, and start questioning architecture decisions.

Sometimes, the code is wrong.

But sometimes, the code is fine.

Sometimes it is just:

a typo
a cached response
the wrong environment, or
a request hitting the wrong endpoint altogether

We all know the list.

The Checklist

Before diving deep into logs and architecture, run through the basics:

[ ] Verify the URL/endpoint — Is it the right path? Any typos?
[ ] Check the environment — Dev, staging, or production? Are you hitting the right one?
[ ] Clear the cache — Browser cache, API cache, application cache
[ ] Check the request — Method (GET vs POST), headers, body format
[ ] Verify authentication — Are you logged in? Is the token valid?
[ ] Check the console — Browser console, server logs, terminal output
[ ] Restart the service — Sometimes it really is that simple
[ ] Check the network — Is the service actually running? Can you reach it?
[ ] Read the error message — What is it actually saying?

Most of these take seconds. They feel too obvious to matter. But starting here saves hours.

The Lesson

The best debugging skill is not knowing every framework or language.

It is knowing when to step back.

When to slow down.

When to check the simple things first.

When to ask, “What if I am looking in the wrong place?”

This is not an argument against thorough debugging.

It is a reminder to start with the basics — even when they feel too obvious to matter.

Because sometimes, that is exactly where the problem lives.

What is the most “obvious” bug that took you way too long to find? We have all been there.

Why Most Reporting Systems Fail Before the UI Even Loads

MeL — Thu, 18 Dec 2025 17:09:33 +0000

Reporting Is a System, Not a Table — Part 1

Most reporting systems do not fail because the table library is wrong.
They fail long before a column is rendered or a chart is drawn.

After working across multiple internal systems, user groups, and reporting requirements, one pattern keeps showing up. When reports start behaving unpredictably, the user interface (UI) often takes the blame, but the root causes usually live deeper in the system. Data shape, query responsibility, permissions, and performance decisions quietly determine whether a report will hold up under real usage or slowly become a source of confusion and mistrust.

It is easy to approach reports as "just tables with filters." I have done this myself, especially early on. In practice, reports behave more like contracts. They define what data is visible, how it is grouped, how it is filtered, and who is allowed to see which parts of it. When those rules are unclear or inconsistently enforced, the interface ends up carrying responsibilities it was never designed to handle.

Many implementations start at the wrong end. A table library is chosen, an endpoint returns raw rows, and filtering or permissions are pushed into the browser. It often works in development. It may even pass early testing. Over time, as datasets grow and business rules evolve, the cracks begin to surface.

This series is not about replacing one JavaScript solution/library with another. It is about treating reporting as a system. That means being intentional about:

where responsibilities live,
how data is shaped before it reaches the UI, and
why early design decisions matter long after the first release.

Before talking about tools like Tabulator, Plotly, or Chart.js, it helps to pause and examine why many reporting implementations struggle before the interface even loads. That foundation matters more than any component choice.

A quick example before we go any further

A common pattern in reporting apps looks like this:

The frontend needs a report table.
The backend exposes "all the rows."
The browser does filtering, pagination, and sometimes totals.
It works… until it does not.

The "bad" version (common, and understandable)

It usually starts with an endpoint that returns too much data, shaped like raw rows:

GET /api/reports/transactions

Response (raw rows only):

{
  "results": [
    {"id": 1, "date": "2025-12-01", "scope": "Ops", "amount": 1200.00},
    {"id": 2, "date": "2025-12-01", "scope": "IT",  "amount": 450.00}
  ]
}

Then the UI handles:

filtering (scope, date range, status)
pagination
totals
"export what I am seeing", and
sometimes permission rules via hidden columns

It is not that anyone "did it wrong." It is that the UI is being asked to do backend or server-side work.

The "good" version (same report, but treated like a contract)

The endpoint accepts report parameters, enforces access rules, and returns a response shaped for both table + summary + charts:

GET /api/reports/transactions?date_from=2025-12-01&date_to=2025-12-12&scope=IT&page=1&page_size=50

Response (rows + totals + metadata):

{
  "meta": {
    "page": 1,
    "page_size": 50,
    "total_rows": 2314,
    "filters": {"date_from": "2025-12-01", "date_to": "2025-12-12", "scope": "IT"}
  },
  "summary": {
    "amount_total": 182340.50,
    "count": 2314
  },
  "results": [
    {"date": "2025-12-01", "scope": "IT", "amount": 450.00, "ref": "INV-10421"}
  ]
}

Now:

the UI renders and interacts
the backend owns correctness, filtering, totals, and permissions
charts and exports can reuse the same contract (no mismatched totals)

This is the foundation the rest of this post is pointing to.

Sketch: application architecture and data movement

Here is a simple, realistic reporting flow (Python/Django-style, but not locked to any stack):

Key idea: the UI sends intent (filters/sort/page), the API validates and enforces access, and the database does what it is good at (filtering, aggregations), with the query layer making the output consistent.

Reports Are Not Tables

A table is a presentation detail.
A report is a decision surface.

That distinction sounds subtle, but it changes how systems are designed. When reports are treated as tables, the focus tends to land on columns, sorting, and visual polish. When reports are treated as systems, attention shifts to intent, constraints, and guarantees.

A report usually answers a specific question:
What happened? What changed? What needs attention? What is allowed to be seen?

Those answers depend on more than rows in a database. They depend on context. Time ranges. Business rules. Aggregations. User roles. Sometimes they depend on historical snapshots rather than current state. When those factors are not made explicit, the report becomes ambiguous, even if it looks correct.

This is where many reporting issues quietly begin. Raw data is exposed, and meaning is reconstructed later in the UI. Filters are layered on top of assumptions. Totals are recalculated in multiple places. Different users see slightly different results and assume the system is broken, even when it is technically doing what it was told to do.

Treating reports as systems encourages a different approach. It helps to define what the report represents before deciding how it is displayed. It pushes aggregation, filtering, and permission logic closer to the data. It reduces the need for frontend workarounds and makes behavior more predictable across users and environments.

Once that mindset is in place, the choice of table, vanilla JavaScript or charting library becomes much clearer. The UI stops carrying hidden logic and starts doing what it does best: presenting information that has already been shaped with intention.

The Real Failure Point: Data Shape (and the "Report Contract")

Once I started treating reports as systems, the first thing I began looking for was not the UI. It was the contract between the UI and the backend.

Not "what columns do we want," but:

What filters exist, and which ones are allowed for this user?
What does "total" mean (total rows, total amount, totals for current filter, totals for all time)?
What does the table need versus what does the export need?
What metadata must be returned so the UI can stay honest?

If that contract is vague, the UI will end up guessing. If the UI is guessing, totals drift, exports disagree with what is on screen, and performance work becomes reactive.

Below is a concrete Python/Django + Django Rest Framework (DRF) example of a report endpoint that returns rows + summary + metadata in a consistent shape. It is intentionally structured to keep "report logic" out of serializers and views and still keep the API readable.

Django/DRF example: a report endpoint with a serializer contract

1) The contract: response serializer (rows + meta + summary)

This is what the frontend can depend on, regardless of whether rendering Tabulator, Plotly, 3s.js, or a CSV export later.

# reports/serializers.py
from rest_framework import serializers


class ReportMetaSerializer(serializers.Serializer):
    """Metadata about the report request and response."""
    page = serializers.IntegerField(help_text="Current page number (1-indexed)")
    page_size = serializers.IntegerField(help_text="Number of items per page")
    total_rows = serializers.IntegerField(help_text="Total number of rows matching filters")
    ordering = serializers.CharField(
        required=False,
        allow_blank=True,
        help_text="Current sort order (e.g., '-date', 'amount')"
    )
    filters = serializers.DictField(
        child=serializers.JSONField(),
        required=False,
        help_text="Active filters applied to this report"
    )


class TransactionsRowSerializer(serializers.Serializer):
    """Individual transaction row in the report."""
    date = serializers.DateField(help_text="Transaction date")
    scope = serializers.CharField(help_text="Scope code (e.g., department, customer, company)")
    ref = serializers.CharField(help_text="Transaction reference number")
    amount = serializers.DecimalField(
        max_digits=12,
        decimal_places=2,
        help_text="Transaction amount"
    )
    status = serializers.CharField(help_text="Transaction status")


class TransactionsSummarySerializer(serializers.Serializer):
    """Summary statistics for the filtered dataset."""
    amount_total = serializers.DecimalField(
        max_digits=14,
        decimal_places=2,
        help_text="Sum of all transaction amounts in the filtered set"
    )
    count = serializers.IntegerField(help_text="Total count of transactions in the filtered set")


class TransactionsReportResponseSerializer(serializers.Serializer):
    """Complete report response contract."""
    meta = ReportMetaSerializer(help_text="Report metadata and pagination info")
    summary = TransactionsSummarySerializer(help_text="Aggregated summary statistics")
    results = TransactionsRowSerializer(many=True, help_text="Paginated transaction rows")

This is not "extra work." This is the difference between a report that stays coherent and one that slowly turns into exceptions and UI patches.

2) Query param validation: request serializer

This keeps view code clean and avoids the "stringly-typed query params everywhere" problem. It also provides clear error messages when validation fails.

# reports/serializers.py
from datetime import date
from rest_framework import serializers


class TransactionsReportRequestSerializer(serializers.Serializer):
    """Validates and normalizes incoming report request parameters."""
    date_from = serializers.DateField(
        required=False,
        help_text="Start date for filtering (inclusive)"
    )
    date_to = serializers.DateField(
        required=False,
        help_text="End date for filtering (inclusive)"
    )
    scope = serializers.CharField(
        required=False,
        allow_blank=True,
        max_length=50,
        help_text="Scope code to filter by (e.g., department, customer, company)"
    )
    status = serializers.ChoiceField(
        required=False,
        choices=["pending", "approved", "rejected", "paid"],
        help_text="Transaction status to filter by"
    )

    page = serializers.IntegerField(
        required=False,
        min_value=1,
        default=1,
        help_text="Page number (1-indexed)"
    )
    page_size = serializers.IntegerField(
        required=False,
        min_value=1,
        max_value=500,
        default=50,
        help_text="Number of items per page (max 500)"
    )

    ordering = serializers.ChoiceField(
        required=False,
        choices=["date", "-date", "amount", "-amount", "scope", "-scope"],
        default="-date",
        help_text="Field and direction to sort by"
    )

    def validate(self, attrs):
        """Cross-field validation for date ranges."""
        date_from = attrs.get("date_from")
        date_to = attrs.get("date_to")
        if date_from and date_to and date_from > date_to:
            raise serializers.ValidationError({
                "date_from": "date_from must be on or before date_to."
            })
        return attrs

3) The API endpoint (in api.py)

The view does three jobs only:

validate input
call a report service
return the shaped contract

This keeps the view thin and testable.

# reports/api.py
import logging
from django.conf import settings
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework.permissions import IsAuthenticated
from rest_framework import status

from .serializers import (
    TransactionsReportRequestSerializer,
    TransactionsReportResponseSerializer,
)
from .services import build_transactions_report

logger = logging.getLogger(__name__)


class TransactionsReportAPI(APIView):
    """API endpoint for transaction reports with filtering, pagination, and permissions."""
    permission_classes = [IsAuthenticated]

    def get(self, request):
        """
        Handle GET requests for transaction reports.

        Validates query parameters, builds the report, and returns
        a structured response with rows, summary, and metadata.
        """
        # Validate incoming parameters
        req = TransactionsReportRequestSerializer(data=request.query_params)
        if not req.is_valid():
            return Response(
                {"errors": req.errors},
                status=status.HTTP_400_BAD_REQUEST
            )

        try:
            # Build the report using the service layer
            payload = build_transactions_report(
                user=request.user,
                params=req.validated_data,
            )

            # Optional but useful: validate the outgoing contract in development
            # This catches contract drift early during development
            out = TransactionsReportResponseSerializer(data=payload)
            if not out.is_valid():
                logger.error(
                    "Report contract validation failed",
                    extra={"errors": out.errors, "payload_keys": list(payload.keys())}
                )
                # In production, one might want to return the payload anyway
                # or raise an exception depending on the error handling strategy
                if settings.DEBUG:
                    return Response(
                        {"errors": out.errors, "payload": payload},
                        status=status.HTTP_500_INTERNAL_SERVER_ERROR
                    )

            return Response(out.data, status=status.HTTP_200_OK)

        except Exception as e:
            logger.exception("Error building transactions report", extra={"user_id": request.user.id})
            return Response(
                {"error": "An error occurred while generating the report."},
                status=status.HTTP_500_INTERNAL_SERVER_ERROR
            )

That last response validation is optional, but it is a quiet quality gate. It catches drift early. In production, you might disable it for performance, but keeping it in development and staging helps maintain contract integrity.

4) The report service (where the real correctness lives)

This is where filtering, permissions, pagination, ordering, and summary live together so they cannot contradict each other. All of these operations happen on the same queryset, ensuring consistency.

# reports/services.py
import logging
from decimal import Decimal
from typing import Dict, Any, Optional
from django.db.models import Sum, Count, QuerySet
from django.core.exceptions import PermissionDenied
from django.conf import settings

from .models import Transaction  # adjust for app
from .utils import coerce_scope_filter, apply_role_visibility

logger = logging.getLogger(__name__)


def build_transactions_report(*, user, params: Dict[str, Any]) -> Dict[str, Any]:
    """
    Build a complete transaction report with filtering, permissions, pagination, and summary.

    This function ensures that all operations (filtering, permissions, pagination, ordering, 
    and summary) are applied consistently to the same queryset.

    Args:
        user: The authenticated user making the request
        params: Validated request parameters from TransactionsReportRequestSerializer

    Returns:
        Dict matching TransactionsReportResponseSerializer structure:
        {
            "meta": {...},
            "summary": {...},
            "results": [...]
        }

    Raises:
        PermissionDenied: If user lacks access to any transaction data
    """
    # Start with base queryset
    # Note: In production, one might want to use select_related() or prefetch_related()
    # if Transaction has foreign keys that will be accessed later
    qs = Transaction.objects.all()

    # 1) Permissions / visibility rules (applied first, before any filtering)
    # This ensures users can only see data they are authorized to access
    qs = apply_role_visibility(qs=qs, user=user)

    # Early return if user has no access
    if not qs.exists() and not user.is_superuser:
        logger.info(
            "User attempted to access transactions report with no visible data",
            extra={"user_id": user.id}
        )

    # 2) Apply filters (all validated by the request serializer)
    date_from = params.get("date_from")
    date_to = params.get("date_to")
    scope = params.get("scope")
    status = params.get("status")

    if date_from:
        qs = qs.filter(date__gte=date_from)
    if date_to:
        qs = qs.filter(date__lte=date_to)
    if scope:
        # Normalize scope code (handles case-insensitive matching)
        normalized_scope = coerce_scope_filter(scope)
        qs = qs.filter(scope__iexact=normalized_scope)
    if status:
        qs = qs.filter(status=status)

    # 3) Compute summary (on the filtered queryset, before pagination)
    # This ensures totals match the visible rows
    try:
        summary = qs.aggregate(
            amount_total=Sum("amount"),
            count=Count("id"),
        )

        # Handle None values from aggregation (can happen with empty querysets)
        amount_total = summary["amount_total"] or Decimal("0.00")
        count = summary["count"] or 0

    except Exception as e:
        logger.error(
            "Error computing report summary",
            extra={"user_id": user.id, "error": str(e)},
            exc_info=True
        )
        # Fallback to safe defaults
        amount_total = Decimal("0.00")
        count = 0

    # 4) Apply ordering (before pagination)
    ordering = params.get("ordering", "-date")
    qs = qs.order_by(ordering)

    # 5) Pagination (server-side, after all filtering and ordering)
    page = params.get("page", 1)
    page_size = params.get("page_size", 50)

    # Validate pagination parameters (defense in depth, even though serializer validates)
    page = max(1, page)  # Ensure page is at least 1
    page_size = max(1, min(page_size, 500))  # Clamp between 1 and 500

    # Get total count before slicing (needed for pagination metadata)
    total_rows = qs.count()

    # Calculate slice bounds
    start = (page - 1) * page_size
    end = start + page_size

    # Slice the queryset and convert to list of dicts
    # Using values() is more efficient than loading full model instances
    # when only specific fields are needed
    rows = list(
        qs.values("date", "scope", "ref", "amount", "status")[start:end]
    )

    # Build the response payload matching the serializer contract
    return {
        "meta": {
            "page": page,
            "page_size": page_size,
            "total_rows": total_rows,
            "ordering": ordering,
            "filters": {
                k: v for k, v in params.items()
                if k in {"date_from", "date_to", "scope", "status"}
                and v not in (None, "")
            },
        },
        "summary": {
            "amount_total": amount_total,
            "count": count,
        },
        "results": rows,
    }

5) A permissions hook that scales (and does not pretend UI is security)

This utility function centralizes permission logic. In a real system, there would be more complex rules involving role hierarchies, scope relationships (e.g., departments, teams, customers, services, companies, registered users), or time-based access.

# reports/utils.py
import logging
from typing import TYPE_CHECKING
from django.db.models import QuerySet

if TYPE_CHECKING:
    from django.contrib.auth.models import User

logger = logging.getLogger(__name__)


def apply_role_visibility(*, qs: QuerySet, user: "User") -> QuerySet:
    """
    Apply role-based visibility rules to a queryset.

    This function filters the queryset based on the role and permissions of the user.
    It should be called early in the query building process, before other filters
    are applied, to ensure users only see data they are authorized to access.

    Visibility rules:
    - Superusers and staff: see all transactions
    - Scoped users: see only transactions within their scope
    - Other users: see nothing (empty queryset)

    Args:
        qs: The base queryset to filter
        user: The authenticated user

    Returns:
        Filtered queryset based on the role and permissions of the user

    Example:
        >>> qs = Transaction.objects.all()
        >>> filtered_qs = apply_role_visibility(qs=qs, user=request.user)
        >>> # filtered_qs now only contains transactions the user can see
    """
    # Superusers and staff see everything
    if getattr(user, "is_superuser", False) or getattr(user, "is_staff", False):
        logger.debug(f"Full access granted to user {user.id} (superuser or staff)")
        return qs

    # Scoped users see only transactions within their scope
    scope = getattr(user, "scope_code", None)
    if scope:
        logger.debug(f"Scope-restricted access for user {user.id}, scope: {scope}")
        return qs.filter(scope__iexact=scope)

    # Default: no access
    # Choose preference: empty queryset or explicit denial
    # Empty queryset is often safer for reporting (shows "no data" rather than error)
    logger.info(f"No access granted to user {user.id} (no role match)")
    return qs.none()


def coerce_scope_filter(scope: str) -> str:
    """
    Normalize scope code for filtering.

    This handles case-insensitive matching and any scope code
    normalization rules (e.g., stripping whitespace, standardizing format).

    Args:
        scope: Raw scope code from request

    Returns:
        Normalized scope code
    """
    if not scope:
        return ""
    # Normalize: strip whitespace, uppercase for consistency
    return scope.strip().upper()

Why this contract makes Tabulator, Plotly, and exports easier

Once the response shape is consistent:

Tabulator can render results and use meta.total_rows for real pagination.
A chart can use summary without re-aggregating on the client.
CSV/PDF exports can reuse the same report service with "export mode" parameters.
When a user says "the totals are wrong," there is one place to debug.

Also, this pattern stays stable regardless of the underlying database. The differences show up later in query optimization and advanced aggregation, not in the contract itself.

Why I validate the outgoing response serializer (and when I do not)

I used to think response validation was redundant. The view already "knows" what it is returning, and the service already "knows" the shape it builds. In practice, reporting endpoints tend to evolve quickly. A new column is added. A field is renamed. A summary value changes type. Someone tweaks a values() call. Suddenly the frontend is parsing null where it expected a number, or a decimal becomes a string, or an export job starts failing quietly.

Validating the outgoing payload with a response serializer is a small habit that prevents the slow drift that reporting systems are famous for.

It is also a practical way to keep the team honest about the contract. If the response serializer fails, it fails immediately, close to the change that caused it, instead of surfacing later as "Tabulator is acting weird" or "the totals look off."

That said, I do not always keep it on.

In development and test environments, I like it enabled.
In production, I usually disable response validation for high-traffic endpoints, unless the payload is small and the endpoint is not performance-sensitive.
If I am returning large datasets or doing frequent calls (live filtering, typeahead, etc.), I prefer contract tests and schema checks over per-request response validation.

The goal is not to add ceremony (unnecessary process). It is to keep reporting predictable.

Filtering, Pagination, and the Illusion of Performance

Filtering and pagination are where reporting systems often begin to feel "fine" in development and painful in production.

A small dataset makes almost any approach look acceptable. Returning 5,000 rows and filtering in the browser can feel fast on a developer machine. It can even look impressive during a demo. But real systems do not stay small. Data grows. Users multiply. Concurrency becomes normal. Suddenly the report that "loads instantly" becomes a spinner, and the first instinct is to blame the UI.

What usually happened is simpler: the UI was asked to do backend work.

Client-side pagination is not pagination

If an endpoint returns all rows and the UI shows 50 per page, it looks like pagination. It is not. The full cost already happened:

network transfer of the entire dataset
memory usage in the browser
slow rendering and re-rendering
time spent filtering and sorting on the client

It is easy to miss this when testing locally, because local latency is not real latency.

Server-side pagination forces honesty

When the backend owns pagination, the UI stops pretending. The API must answer three questions clearly:

What is the current page?
How many rows are available for the current filters?
What is the consistent ordering?

That is exactly why the "report contract" includes:

meta.page
meta.page_size
meta.total_rows
meta.ordering

With that, the frontend can show real paging, and the user can trust what they are seeing.

Filtering belongs close to the data

The same principle applies to filtering. If the user filters by scope, date range, status, or location, the filter belongs in the query, not in JavaScript.

There is also a correctness angle: filtering in the backend ensures totals and charts stay aligned with the visible rows. If the backend provides summary computed on the same filtered queryset, the UI cannot accidentally compute a different number.

This is where reporting shifts from "table behavior" to "system behavior." It is not only about speed. It is about consistency.

The report service becomes the single source of truth

This is why I prefer the service-layer approach shown earlier. It keeps:

filters
ordering
pagination
summary totals
visibility rules

in one place, instead of scattering them across views, serializers, and frontend code.

Once those pieces live together, performance tuning becomes more straightforward too. Indexes start to matter. Query plans become readable. Caching becomes targeted. Exports can reuse the same core logic without duplicating filters.

Most importantly, the UI becomes simpler. It can focus on interaction and presentation instead of reconstruction.

Permissions Change Everything

Permissions are the point where many otherwise well-designed reports quietly fall apart.

It is tempting to think of permissions as a UI concern. Hide a column. Disable a filter. Grey out a button. That approach often starts with good intentions and ends with confusion. Two users look at "the same report" and see different numbers. Someone exports data they were not meant to see. A support ticket comes in saying the report is "wrong," when in reality it is inconsistent.

What usually happened is that permissions were layered on after the fact, instead of being part of the definition of the report.

Once reports are treated as systems, permissions stop being optional decoration and become part of the contract.

The same report, different realities

In most real systems, the same report must serve different roles:

An administrator who can see everything.
A scoped user who can only see their own slice.
A reviewer who can see rows but not financial totals.
An auditor who needs historical accuracy, not current state.

From the UI perspective, these users may be clicking the same menu item. From the perspective of the system, they are asking different questions.

This is why permissions belong in the backend, close to the data. Not because the UI is untrustworthy, but because the UI does not have enough context to enforce policy safely.

Permissions as part of the query, not the presentation

In the earlier example, visibility rules were applied before filters, summaries, and pagination:

qs = apply_role_visibility(qs=qs, user=user)

That single line has far-reaching implications.

Totals are computed only over rows the user is allowed to see.
Pagination counts are accurate for that user.
Exports reuse the same visibility rules automatically.
Charts derived from the same endpoint cannot drift.

There is no need for special-case logic in the UI. The UI does not need to "know" why a row is missing. It simply renders what the contract allows.

This approach also scales better as rules evolve. When a new role is added or an access rule changes, it is updated in one place. The report does not need to be redesigned. The frontend does not need conditional logic scattered across components.

Hiding data is not the same as securing it

One of the more subtle issues with UI-level permissions is that they often hide data without removing it. A column is not rendered, but the data still arrives in the response. A total is not shown, but the raw rows allow it to be recomputed. This works until someone inspects the network response, or until an export feature is added later.

By enforcing permissions in the query itself, the system avoids that class of problems entirely. The data never leaves the server unless the user is allowed to see it.

Why this matters before choosing UI tools

Libraries like Tabulator, Plotly, Chart.js and 3s.js are powerful precisely because they assume the backend is doing its job. They can paginate, sort, export, and render efficiently when given a clean, honest data source. They become fragile when asked to compensate for missing permission logic or inconsistent data contracts.

Once permissions are part of the report system, UI decisions become simpler and safer. The same endpoint can serve a table, a chart, or an export without special handling. The system behaves predictably, even as requirements change.

This is usually the point where reporting starts to feel stable. Not flashy, but trustworthy.

Why UI Tools Eventually Matter (But Not First)

Once data shape, filtering, pagination, and permissions are handled intentionally, something interesting happens: the UI becomes easier to reason about.

This is usually the point where UI tools start to matter — not because they fix problems, but because they stop being asked to solve them. The focus shifts to user experience (UX) on the UI.

When the backend provides a consistent report contract, table and charting libraries shift from being structural dependencies to presentation choices. They amplify what already exists instead of compensating for gaps.

This is why teams often have very different experiences with the same UI tools. A library that feels brittle in one system can feel elegant in another. The difference is rarely the library itself. It is the quality of the data contract behind it.

When UI tools are doing too much

UI libraries struggle when they are asked to:

infer totals from raw rows
manage permissions through hidden columns
simulate pagination on large datasets
reconcile mismatched numbers between tables and charts

At that point, configuration grows, edge cases multiply, and confidence erodes. Every new requirement feels like a workaround.

When UI tools do exactly enough

Once the backend owns correctness, UI tools can focus on what they are good at:

rendering large tables efficiently
handling column visibility and resizing
supporting user-driven sorting and filtering
exporting what the user is allowed to see
presenting the same data in multiple forms

This is where tools like Tabulator start to shine. Not because they are "better" in isolation, but because they assume server-side responsibility for data integrity. They work best when pagination is real, totals are authoritative, and permissions are already enforced.

Charting libraries follow the same pattern. Plotly, Chart.js amd 3s.js are most effective when they consume pre-aggregated, trustworthy data. They should not be recalculating business logic in the browser or guessing which rows matter.

Tool choice becomes a secondary decision

Once reporting is treated as a system, the conversation changes. Instead of asking "Which table library should we use?" the question becomes:

What interactions do users need?
How much control should they have?
What level of customization is justified?
What performance guarantees do we need?

Those answers naturally guide tool selection. The UI stops being the foundation and becomes the final layer.

That order matters.

This foundation — treating reports as systems with clear contracts, server-side responsibility, and permission-aware queries — makes everything else easier. In Part 2, we will dive into implementation details and advanced patterns.

Have you encountered reporting systems that struggled with these fundamentals? What worked (or did not work) in your experience?

Behind the Screens: The Quiet Work That Shapes the Code

MeL — Fri, 05 Dec 2025 18:28:24 +0000

Most of the real work in development happens before the first line of code. It is in the structure, the workflow, the decisions, and the way everything fits together behind the scenes.

Last week, I was deep in data logic for a service scheduling application, debugging unexpected behavior and juggling shifting priorities. It was another reminder that developing a system is never just about typing syntax — it is the entire system design.

Here are a few realizations from the trenches:

1. Micro-inefficiencies matter

A slow query, a chaotic folder structure, or that one repetitive manual data-entry step — these small friction points quietly drain energy. Fixing them does not just save time; it resets mental space.

2. The “pre-code” work is the work

Planning, designing, creating test plans, refining logic, organizing workflow — this is real development. When more time is spent here, the actual implementation usually takes half the time. When the logic flows, the code follows.

3. AI is a workflow multiplier

I use AI to handle boilerplate, code reviews and documentation outlines. It is not about replacing my brain and coding activities; it is about keeping a free, clear mind for deeper coding, reasoning and system-level thinking and gaining improvement in project management.

4. Better systems → Better state of mind

A smoother workflow leads to improved project management, better code, better decision-making, and a calmer, more focused developer.

I am taking this leap to finally start sharing here about building internal/external systems, refining data for reporting, and improving enterprise workflows — the parts of development we do not generally talk about, but that shape our entire engineering experience.

Discussion: What is one invisible part of your workflow that saves you the most headaches? Let me know in the comments!