DEV Community: Tural Muradov

Why Log Masking Matters in Kubernetes (and How We Enforced PCI Safety with Fluent Bit)

Tural Muradov — Mon, 29 Dec 2025 05:57:10 +0000

Logging is one of the most underestimated security risks in systems.

Not because engineers are careless, but because logs change faster than security reviews, and production pressure always wins over perfect hygiene.

We will go through:

why sensitive data ends up in logs
why relying on developers alone is not enough
how we enforced PCI-safe logging at the infrastructure level
a complete Fluent Bit + Fluentd configuration
a production-ready Lua masking script

This solution is used in a real Kubernetes production environment.

The uncomfortable truth about logs

Every experienced engineer has seen logs like these:

POST /pay?pan=4111111111111111&cvv=123
password=secretpass
Authorization failed for user=admin pass=admin123

These usually appear not because someone intended to leak data, but because of:

temporary debug logging during incidents
forgotten log statements after hotfixes
third-party SDKs logging request payloads
rushed troubleshooting under pressure
team or ownership changes

Why this is dangerous

Logs often:

live longer than databases
are accessible to more people
are copied across multiple systems
bypass encryption and access reviews
sit outside normal data-protection controls

A single leaked PAN or password in logs can result in PCI DSS violations, GDPR incidents, and audit findings that are hard to fix retroactively.

Why “just don’t log sensitive data” does not work

Even strong engineering teams eventually log sensitive data:

humans forget
incidents force shortcuts
third-party libraries log unexpected payloads
reviews do not cover every log line

Security controls must assume mistakes will happen.

That is why we moved responsibility from application code into the logging pipeline.

Enforcing security at the logging layer

We defined one rule:

Sensitive data must never leave the node unmasked, even if an application logs it.

Architecture:

Application
   ↓
Fluent Bit (masking and filtering)
   ↓
Fluentd
   ↓
Elasticsearch / Kibana

Masking happens before logs are forwarded anywhere.

Why Fluent Bit with Lua

Fluent Bit runs:

on every node
close to the log source
before logs leave the host

Lua allows:

conditional logic
validation (for example, Luhn checks)
partial masking instead of blind replacement
zero application changes

What we mask and why

Data type	Reason
PAN (card number)	PCI DSS 3.4
CVV / PIN	PCI DSS 3.3
Track1 / Track2	PCI DSS
Passwords	PCI DSS 8.x
IBAN	GDPR / PII
IP addresses	GDPR / PII

Example

Before:

password=secretpass pan=4111111111111111&cvv=123 ip=10.233.105.74

After:

password=****retpass pan=411111******1111&cvv=*** ip=xxx.xxx.xxx.xxx

Fluent Bit configuration (production-safe)

Input: exclude system namespaces early

[INPUT]
    Name tail
    Path /var/log/containers/*.log
    Exclude_Path /var/log/containers/*_kube-system_*.log,                 /var/log/containers/*_logging_*.log,                 /var/log/containers/*_monitoring_*.log,                 /var/log/containers/fluent-bit-*.log,                 /var/log/containers/fluentd-*.log
    Parser cri
    Tag kubernetes.*
    DB /tail-db/tail-containers-state.db
    Mem_Buf_Limit 100MB

This prevents Fluent Bit from ingesting its own logs, kube-system components, and logging feedback loops.

Kubernetes metadata enrichment

[FILTER]
    Name kubernetes
    Match kubernetes.*
    Merge_Log On
    K8S-Logging.Exclude On

Namespace and container exclusion

[FILTER]
    Name grep
    Match *
    Exclude kubernetes.namespace_name ^kube-system$
    Exclude kubernetes.namespace_name ^logging$
    Exclude kubernetes.namespace_name ^monitoring$
    Exclude kubernetes.container_name ^fluent-bit$
    Exclude kubernetes.container_name ^fluentd$

Lua masking filter

[FILTER]
    Name lua
    Match kubernetes.*
    script /fluent-bit/scripts/mask_pci.lua
    call filter

Masking occurs before logs leave the node.

Fluentd output configuration (example)

<match **>
  @type elasticsearch
  host elasticsearch.logging.svc
  port 9200
  logstash_format true
  include_tag_key true
</match>

At this point, all sensitive data has already been sanitized.

Full Lua masking script

-- /fluent-bit/scripts/mask_pci.lua
-- Centralized PCI / PII log masking

local function luhn(num)
    local sum, alt = 0, false
    for i = #num, 1, -1 do
        local n = tonumber(num:sub(i, i))
        if alt then
            n = n * 2
            if n > 9 then n = n - 9 end
        end
        sum = sum + n
        alt = not alt
    end
    return sum % 10 == 0
end

function filter(tag, ts, record)
    local msg = record["message"]
    if not msg then return 1, ts, record end

    -- PAN masking
    msg = msg:gsub("(%d%d%d%d%d%d%d%d%d%d%d%d%d?%d?%d?%d?%d?)",
        function(num)
            if #num >= 12 and #num <= 19 and luhn(num) then
                return num:sub(1,6) .. string.rep("*", #num - 10) .. num:sub(-4)
            end
            return num
        end
    )

    -- CVV / PIN
    msg = msg:gsub("(?i)(cvv|cvc|cv2|cvn)=\d{3,4}", "%1=***")
    msg = msg:gsub("(?i)(pin|pinblock)=\d+", "%1=****")

    -- Expiry dates
    msg = msg:gsub("(?i)(exp|expiry|expires)=\d{2}[/\-]\d{2}", "%1=**/**")

    -- Track data
    msg = msg:gsub("(?i)%B%d+%^[^%s]+", "track1=***")
    msg = msg:gsub("(?i);%d+=%d+", "track2=***")

    -- IBAN
    msg = msg:gsub("(?i)\b([A-Z]{2}\d{2}[A-Z0-9]{11,30})\b",
        function(iban)
            return iban:sub(1,6) .. string.rep("*", #iban - 10) .. iban:sub(-4)
        end
    )

    -- Passwords (partial)
    msg = msg:gsub(
        "(?i)\b(pass(word)?|passwd|pwd)\s*([:=])\s*([^\s"&]+)",
        function(key, _, sep, value)
            if #value <= 4 then
                return key .. sep .. string.rep("*", #value)
            end
            return key .. sep .. "****" .. value:sub(5)
        end
    )

    -- IPv4 addresses
    msg = msg:gsub("(%d+%.%d+%.%d+%.%d+)", "xxx.xxx.xxx.xxx")

    record["message"] = msg
    return 1, ts, record
end

PCI DSS alignment

Requirement	Enforcement
3.3	CVV and PIN fully masked
3.4	PAN partially masked
3.5	PAN never stored unmasked
8.x	Passwords protected
GDPR	PII minimized

Final result

No PANs in Elasticsearch
No credentials in Kibana
No application code changes
No reliance on human perfection
Auditors satisfied

Designing a Dynamic Loan Decision Engine in a Regulated FinTech Environment

Tural Muradov — Mon, 22 Dec 2025 03:46:10 +0000

In regulated fintech environments, loan decisioning is often treated as a mathematical problem.
In reality, it is much closer to a system design problem.

When you operate under banking regulations, external audits, and constantly changing compliance rules, the biggest challenge is not calculating a score. The challenge is building a decision engine that can adapt without breaking production, without violating regulations, and without slowing down the business.

I’ve seen decision logic become a bottleneck more than once — not because it was wrong, but because it was too rigid to evolve. This article describes how I approached the design of a dynamic loan decision engine that treats credit decisioning as a configurable, auditable system rather than a static model.

The Constraints of Regulated FinTech

Designing a loan decision engine in a regulated fintech environment means accepting a set of constraints from day one.

Regulations change. Banking requirements evolve. Audit and compliance teams require full explainability of every decision. At the same time, the business expects fast iteration, experimentation, and growth.

These forces often pull the system in opposite directions. A design that optimizes only for flexibility usually fails compliance. A design that optimizes only for compliance quickly becomes unusable for the product.

The decision engine must sit exactly in the middle — flexible enough to adapt, but strict enough to remain predictable and auditable.

Core Design Principles

Before thinking about metrics or rules, I focused on a few core principles that guided the entire architecture.

First, configuration must be treated as a first-class citizen. Decision logic should not live inside application code. If changing a rule requires a redeployment, the system will not scale operationally.

Second, customer data and product logic must be separated. Who the customer is and what a loan product allows are two different concerns. Mixing them leads to fragile and opaque decision-making.

Finally, explainability must be designed in from the start. In regulated environments, it is not enough to make the correct decision — you must be able to explain why it was made, even months later.

Card-Based Decision Model

One of the most important architectural decisions was to avoid treating loan decisioning as a single scoring function.

Instead, I designed the system around a card-based model that separates concerns and keeps decision logic understandable at scale.

Each loan request generates a Client Card — a snapshot of the customer’s current state. It contains signals derived from minimal user input and a large amount of data retrieved automatically from external systems such as government platforms and banking integrations.

Loan products are represented as Product Cards. Each product card defines its own eligibility rules, constraints, and risk parameters. This makes product logic explicit and independent from individual customers.

Between them sits the Pre-Match layer. Its purpose is simple but critical: quickly exclude requests that clearly do not meet minimum requirements, before triggering expensive or paid checks. In practice, this significantly reduces operational cost and unnecessary load on external systems.

The matching engine evaluates client cards against product cards through a configurable ruleset, producing either a loan offer or a clear rejection reason.

Metrics-Driven Decisioning

A dynamic decision engine lives or dies by the quality and structure of its metrics.

In this system, every loan request is evaluated against a large set of metrics — not to create a single opaque score, but to build a transparent decision context. The engine combines behavioral signals, external data, and internal platform information to understand both eligibility and risk.

A key design choice was to minimize required user input. The less a customer needs to manually provide, the better the conversion. At the same time, relying purely on user-provided data introduces obvious risk.

The solution was to treat external integrations as first-class data sources and design the engine so that most signals are resolved automatically. This allows the platform to make informed decisions while keeping the user experience simple and fast.

Dynamic Risk Control Without Redeployments

In regulated fintech environments, risk appetite is not static.

What is acceptable today may be too risky tomorrow, or too conservative next month. Hardcoding risk thresholds into application logic makes this problem almost impossible to manage.

To avoid this, the engine was designed so that all decision metrics and rules are fully configurable. Product teams can tighten or relax eligibility criteria, exclude specific customer segments, or temporarily hide products — without redeploying code or changing the core system.

This approach shifts credit strategy from an engineering problem into a controlled configuration problem, while still preserving technical safeguards and auditability.

Observability & Auditability

One lesson became clear early: in regulated lending, observability is not optional.

Every automated decision must be explainable — not just at the moment it is made, but months later during audits or investigations. A simple “score exceeded threshold” is rarely sufficient.

The engine was designed to record not only the final decision, but also the decision context — which metrics were evaluated, which rules were applied, and why a particular outcome was produced.

This makes automated decisioning defensible and builds trust with compliance, risk, and business stakeholders.

Lessons Learned

Building a dynamic decision engine is not about finding the perfect model.

It is about designing a system that can evolve without becoming fragile. Flexibility without structure leads to chaos. Structure without flexibility leads to stagnation.

The biggest lesson was that credit decisioning should be treated as a living system — one that needs continuous tuning, visibility, and clear ownership — rather than a static artifact delivered once and forgotten.

In modern fintech platforms, especially in regulated environments, loan decisioning is a system design problem before it is a data science problem.

Dynamic, configurable decision engines allow platforms to scale responsibly, adapt to regulation, and evolve with the market — without sacrificing control or transparency.

Static models may still have their place, but they are no longer sufficient on their own. The future of automated lending belongs to systems that can change as fast as the world around them.

Why Static Credit Scoring Fails in Modern FinTech Platforms

Tural Muradov — Sun, 21 Dec 2025 09:59:52 +0000

Introduction

I’ve worked with credit decisioning systems long enough to see the same pattern repeat itself.

A scoring model is built, validated, approved — and at some point, it quietly turns into the biggest obstacle to product growth. Not because it’s inaccurate, but because it can’t keep up with how fast reality changes.

Static credit scoring works well on paper. In real production fintech systems, especially in regulated or emerging markets, it often fails in ways that are not immediately obvious. The problem is rarely the math itself — it’s the rigidity around it.

What Is Static Credit Scoring?

Static credit scoring typically relies on a predefined model where:

input parameters are fixed or change infrequently
decision thresholds are hardcoded or rarely adjusted
scoring logic is tightly coupled to the application code or external vendors
changes require model retraining, redeployment, or long approval cycles

These models are often treated as “black boxes” and integrated as a single decision point in the loan approval process.

While this approach simplifies governance and validation, it assumes that risk profiles, regulations, and product requirements remain relatively stable over time — an assumption that no longer holds true for most fintech platforms.

Why Static Scoring Breaks in Modern FinTech

In theory, static scoring models provide stability and predictability. In practice, they break down once a fintech platform starts operating at scale.

A small change — a new regulatory requirement, a new loan product, or entry into a new market — often turns into weeks of coordination between risk, engineering, and compliance teams. Even minor adjustments may require model retraining, vendor involvement, or full redeployment.

By the time the scoring logic is updated, the business context has already changed again.

What makes this worse is that static models usually treat credit decisioning as a single checkpoint rather than a process. They don’t adapt well to multi-step onboarding flows, product-specific rules, or real-time signals coming from external systems.

Over time, this rigidity forces teams to choose between speed and control — a trade-off that modern fintech platforms can’t afford.

The Reality of Regulated and Emerging Markets

In regulated and emerging markets, the limitations of static scoring become even more visible.

Regulatory requirements may vary significantly between countries and can change faster than traditional scoring models can adapt. In addition, data availability and quality often differ across markets, making one-size-fits-all scoring models ineffective.

Static scoring systems are poorly suited for environments where:
• minimum required data may change
• external data sources evolve over time
• compliance rules differ per product or region
• explainability and auditability are mandatory

In these scenarios, static models introduce operational risk and slow down market expansion.

Business and Product Impact

From a business perspective, static credit scoring often leads to:
• over-rejection of potentially good customers
• increased operational costs due to manual reviews
• slower time-to-market for new products
• limited ability to experiment with user journeys

Instead of enabling growth, the scoring system becomes a constraint that limits both conversion and innovation.

What Modern FinTech Platforms Actually Need

Modern fintech platforms require dynamic decisioning systems, not static scoring models.

Key architectural principles include:
• configuration-driven rules instead of hardcoded logic
• separation of customer data from product eligibility logic
• explainable and auditable decisions
• ability to adjust risk parameters without redeployment
• product-aware and market-aware decisioning

These principles allow fintech platforms to remain compliant, scalable, and adaptable in a constantly changing environment.

Conclusion

Static credit scoring models were built for a different era — one where markets were stable, products were limited, and regulatory change was slow.
In modern fintech platforms, especially those operating across multiple markets and regulatory regimes, static scoring increasingly fails to meet business, risk, and technical requirements.
This doesn’t mean static credit scoring is useless. In some contexts, it’s still the right choice.
But for fintech platforms that need to adapt quickly, operate under changing regulation, and scale across markets, treating credit decisioning as something fixed is increasingly a liability rather than a safeguard.

MadMigration

Tural Muradov — Mon, 15 Feb 2021 10:21:36 +0000

I would like to introduce a recently started open source project designed for database migration. Why would we need such a tool when there are many database migration tools that exist? Let's dive into details. We were asked to add some new features to a project which was made back in time. Upon entering the project we realized that the database structure was not designed very well, had complicated issues on tables. We decided to redesign the database structure and keep data as it was.
We looked at many tools to produce solution for this case, however, they did operate almost the same. Basically, the procedure of the tools, obtaining old database structure and convert data types,change structure of database and transfer it to another database. We have designed new database structure and our main goal to keep data. We needed to take data from different tables and transfer them to a new table. For sure, another option could be dumping old database and change everthing there and import it into new a database. Fortunately we did not apply this tecnnique, we decided to make a tool that does all of it. That is where story begins MadMigration.

MadMigration is written entirely in python. We used the power of the SQLAlchemy library. In this tutorial, we will explain the details of how to use MadMigration.

Installation

pip install madmigration
or
pip3 install -U madmigration

You can check version:
madmigration --version

Let's get start.

For example you have old MySQL database table with name user :

#Mysql
CREATE TABLE `user` (
  `id` int NOT NULL AUTO_INCREMENT,
  `fname` varchar(10) NOT NULL,
  `lname` varchar(10) NOT NULL,
  `nick` varchar(10) NOT NULL,
  `email` varchar(15) NOT NULL,
  `pass` varchar(15) NOT NULL,
 `updated` TIMESTAMP NULL,
 `created` TIMESTAMP NOT NULL,
  `companyID` INT,
  PRIMARY KEY (`id`),
  FOREIGN KEY (companyID)
  REFERENCES company(`id`)
  ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci

And you want to migrate this table to new postgresql table with different column names:

#postgresql

CREATE TABLE public."User" (
created_at timestamp NOT NULL,
updated_at timestamp NULL,
id serial NOT NULL,
"name" varchar NOT NULL,
surname varchar NOT NULL,
nickname varchar NOT NULL,
email varchar NOT NULL,
"password" varchar NOT NULL,
CONSTRAINT "User_email_key" UNIQUE (email),
CONSTRAINT "User_pkey" PRIMARY KEY (id),
company_id integer REFERENCES company (id)
ON DELETE SET NULL
);

Then create yaml file schema.yml for declaring tables and columns. At first stage add Configs section. This section is for connecting to database. Configs section have two subsections SourceConfig,DestinationConfig:

version: 0.1.8
Configs:
  - SourceConfig:
      dbURI: "mysql://root:admin@127.0.0.1/old"
  - DestinationConfig:
      dbURI: "postgresql://root:admin@127.0.0.1/new"

The migrationTables section is intended to describe information about tables:

migrationTables:
  - migrationTable:
      SourceTable:
        name: user
      DestinationTable:
        name: User
        create: true

Nextly, write information about columns in MigrationColumns section.

MigrationColumns:
  - sourceColumn:
      name: id
    destinationColumn:
      name: id
      options:
        primary_key: true
        type_cast: bigint

In case of creating a foreign key you can specify it in the column parameters. By defining foreign key options, such as ondelete,onupdate. The table_name parameter is a reference to the table, column_name is the name of the referenced column.

  - sourceColumn:
      name: companyID
    destinationColumn:
      name: company_id
      options:
        type_cast: int
        nullable: true
        foreign_key:
          table_name: company
          column_name: id
          ondelete: "SET NULL"

No options added sourceColumn section, however in destinationColumn section we could have primary_key,nullable,default,index,unique,autoincrement,foreign_key,length,type_cast options. The final yaml sturtcure will look like this:

version: 0.1.8
Configs:
  - SourceConfig:
      dbURI: "mysql://root:admin@127.0.0.1/old"
  - DestinationConfig:
      dbURI: "postgresql://root:admin@127.0.0.1/new"

migrationTables:
  - migrationTable:
      SourceTable:
        name: user
      DestinationTable:
        name: User
        create: True 

      MigrationColumns:
        - sourceColumn:
            name: id
          destinationColumn: 
            name: id
            options:
              primary_key: true
              type_cast: bigint

        - sourceColumn:
            name: fname
          destinationColumn:
            name: name
            options:
              type_cast: string

        - sourceColumn:
            name: lname
          destinationColumn:
            name: surname
            options:
              type_cast: string

        - sourceColumn:
            name: email
          destinationColumn:
            name: email
            options:
              type_cast: string
              unique: true

        - sourceColumn:
            name: nick
          destinationColumn:
            name: nickname
            options:
              type_cast: string

        - sourceColumn:
            name: pass
          destinationColumn:
            name: password
            options:
              type_cast: string

        - sourceColumn:
            name: updated
          destinationColumn:
            name: updated_at
            options:
              type_cast: datetime

        - sourceColumn:
            name: created
          destinationColumn:
            name: created_at
            options:
              type_cast: datetime

        - sourceColumn:
            name: companyID
          destinationColumn:
            name: company_id
            options:
              type_cast: int
              nullable: true
              foreign_key:
                table_name: company
                column_name: id
                ondelete: "SET NULL"

In addition, you can migrate your data from RDBMS to NOSQL. Below a small example for migration data from PostgreSQl to MongoDB. Work on PostgreSQl to MongoDB still goes, we will add other features as soon as possible.

version: 0.1.8
Configs:
  - SourceConfig:
      dbURI: "postgresql://root:admin@localhost:5432/olddb"
  - DestinationConfig:
      dbURI: "mongodb://localhost:27017/mydb"

migrationTables:
  - migrationTable:
      SourceTable:  #postgresql table name
        name: company  
      DestinationTable: #the collection name:
        name: Company

      MigrationColumns:
        - sourceColumn:
            name: id
          destinationColumn:
            name: id
            options:
              type_cast: uuid

        - sourceColumn:
            name: name
          destinationColumn:
            name: NAME
            options:
              type_cast: varchar

        - sourceColumn:
            name: created
          destinationColumn:
            name: CREATED
            options:    
              type_cast: datetime
        - sourceColumn:
            name: email
          destinationColumn:
            name: EMAIL
            options:     
              type_cast: string
        - sourceColumn:
            name: updated
          destinationColumn:
            name: UPDATED
            options:
              type_cast: datetime
         - sourceColumn:
            name: code
          destinationColumn:
            name: CODE
            options:      
              type_cast: string

Upon finishing declaration of yaml file run the following command.

madmigrate -f schema.yaml

Alternatively, you can separate migration logic into multiple .yaml or .json files. Importantly, one main .yaml file should exist, other files should be imported into this main file.

main.yaml file:

version: 1.1
Configs:
  - SourceConfig:
      dbURI: "mysql://root:admin@127.0.0.1/old"
  - DestinationConfig:
      dbURI: "postgresql://root:admin@127.0.0.1/new"

migrationTables:
  - migrationTable: !import company.yaml
  - migrationTable: !import op_cond.json

company.yaml file

SourceTable:
  name: company
DestinationTable:
  name: company
  create: true 

MigrationColumns:
  - sourceColumn:
      name: id
    destinationColumn: 
      name: id
      options:
        primary_key: true
        type_cast: uuid

  - sourceColumn:
      name: name
    destinationColumn:
      name: name
      options:
        length: 120
        type_cast: varchar
        nullable: false

  - sourceColumn:
      name: created
    destinationColumn:
      name: created
      options:
        type_cast: datetime
  - sourceColumn:
      name: updated
    destinationColumn:
      name: updated
      options:
        type_cast: datetime
        nullable: true

op_conds.json file

{
    "SourceTable": {
      "name": "operation_conditions"
    },
    "DestinationTable": {
      "name": "operation_conditions",
      "create": true
    },
    "MigrationColumns": [
      {
        "sourceColumn": {
          "name": "id"
        },
        "destinationColumn": {
          "name": "id",
          "options": {
            "primary_key": true,
            "type_cast": "uuid"
          }
        }
      },
      {
        "sourceColumn": {
          "name": "interest"
        },
        "destinationColumn": {
          "name": "interest",
          "options": {
            "type_cast": "varchar",
            "length": 30,
            "nullable": false
          }
        }
      },
      {
        "sourceColumn": {
          "name": "FIFD"
        },
        "destinationColumn": {
          "name": "FIFD",
          "options": {
            "type_cast": "varchar",
            "length": 30,
            "nullable": false
          }
        }
      },
      {
        "sourceColumn": {
          "name": "comission"
        },
        "destinationColumn": {
          "name": "comission",
          "options": {
            "type_cast": "varchar",
            "length": 30,
            "nullable": false
          }
        }
      }
    ]
  }

We are not currently fully implemented all functionality for SQL and NOSQL, therefore we are still developing and adding new features on it.
Thank you for amazing work to our team for implementing this tool.

OPA-Python-Client

Tural Muradov — Sat, 01 Feb 2020 21:46:11 +0000

https://pypi.org/project/OPA-python-client/

Python client for work with open policy agent (OPA) API.