DeciFlow: Custom AI Decisions for Logs, JSONL, and HDFS Data

DeciFlow started with a practical question: when reviewing operational logs, which events should I look at first?

It now supports a broader workflow: read text records, apply configurable AI decision criteria, inspect the results, and optionally connect them to follow-up actions.

You can use it to classify application logs, assess urgency, or organize customer inquiries by topic and follow-up needs.

GitHub and setup instructions: https://github.com/call518/DeciFlow

What DeciFlow Does

DeciFlow reads data and provides a browser interface for configuring tasks and reviewing results.

The inference engine runs separately:

• Laya is the default, self-hosted decision engine. It can run on CPU or a compatible NVIDIA GPU.
• Jev is an optional hosted, paid decision API.

DeciFlow sends input text and decision criteria to the selected engine, then stores and displays the returned predictions. It never automatically switches from Laya to the paid Jev API.

The basic workflow is:

Source
  → Criteria
  → Classification
  → Human Review
  → Optional Python Action

Define the Decisions You Need

Decisions are configurable rather than limited to a fixed set of log categories.

A Criteria profile can combine multiple named decisions in one request:

Type	Purpose	Example
Choice	Select from defined categories	Event type or inquiry topic
Score	Evaluate an ordered scale	Urgency
Noul	Estimate a yes/no probability	Whether follow-up is required

For operational logs, a profile might classify the event as Normal, Warning, Error, Security, or Other, while also assessing urgency.

For customer inquiry records, it might identify the topic, handling status, follow-up owner, and need for further action.

The decisions should match the evidence available in the input. A question alone, for example, does not establish whether someone has already resolved it.

Read Local, SSH, and HDFS Sources

DeciFlow supports:

• Local files.
• Remote files over SSH/SFTP.
• HDFS through the native Hadoop client.
• HDFS through WebHDFS.

File patterns let you select multiple files without merging them first.

Two input formats are available:

• TEXT (Chunk): group text events into configurable chunks.
• JSONL (Line): process one JSON object per line, using a configured top-level text field.

For example, a JSONL source can use consulting_content as its Content Field. Each record remains a separate input, even when several records belong to the same conversation.

HDFS sources support batch reads of UTF-8 TEXT and JSONL files. Connection settings are configured separately from paths such as:

/warehouse/events/part-*.jsonl

Native HDFS and WebHDFS use different connection and authentication settings. See the HDFS documentation for prerequisites and limits.

Save Repeatable Tasks

A Task connects Sources with Criteria, chunk settings, and optional Actions.

Tasks support three execution modes:

• Batch / Manual: read existing files once.
• Batch / Cron: run batches on a schedule with an explicit timezone.
• Realtime: follow new lines and file rotation for local and SSH files.

HDFS currently supports Batch only.

Tasks can be paused and resumed. Execution history, results, and reviewer corrections persist in SQLite rather than only in the browser.

Active runs can recover from saved checkpoints after a server restart, subject to the documented recovery limits. Paused runs remain paused.

Review Predictions in the Browser

The results interface lets you:

• Inspect the original input and each configured decision.
• View decision probabilities and Criteria definitions.
• Filter results by processing status, source, host, review status, and text.
• Find predictions that need review.
• Record human corrections separately from the original predictions.
• Export all results or only results with human corrections as JSON.

Charts let you explore different decisions within the same run. Switching charts or filtering saved results does not rerun inference.

Task History and execution results also have shareable URLs, so you can return to a particular run and its filters.

Probabilities help prioritize inspection; they do not guarantee correctness. Human corrections are review records, not automatic model retraining.

Connect Results to Python Actions

An optional Python Action receives a successful classification result as JSON and runs custom code.

Actions have separately managed Python environments and dependencies. They can connect results to an HTTP endpoint, database, notification system, or another service.

The included Elasticsearch sample demonstrates writing results to an index.

Classification and Action outcomes are tracked separately, so a successful decision and a failed delivery remain distinguishable.

Actions execute code and may produce external effects. Starting a new Batch run rereads the input and can repeat those effects. DeciFlow does not create tickets automatically; that requires an Action you configure.

For the current result schema and setup, see the Actions documentation.

Share One Inference Server Across Nodes

DeciFlow and Laya can run on the same host or on separate hosts:

Log Host A — DeciFlow ─┐
Log Host B — DeciFlow ─┼→ Shared Laya Server
Log Host C — DeciFlow ─┘

DeciFlow nodes do not need a local GPU or PyTorch installation when using a separate Laya server.

Self-hosting Laya lets you keep classification requests within your own infrastructure. Jev remains an optional external service.

Criteria can also be drafted with an optional OpenAI integration. That is separate from classification: it sends the samples you review and submit to OpenAI and incurs API charges. You can define Criteria manually without using it.

Try It With Sample Data

The repository includes sample operational logs and customer inquiry datasets, so you can explore the workflow before connecting your own systems.

For the Docker Compose quick start, use a Linux host with Git and Docker Compose installed:

git clone https://github.com/call518/DeciFlow.git
cd DeciFlow

# Create the configuration only if it does not already exist.
test -f .env || cp .env.example .env

For CPU execution, set these values in .env:

COMPOSE_PROFILES=cpu
DECISION_ENGINE=laya
LAYA_MODEL=multilingual
LAYA_API_URL=http://laya:8000/v1/systemone
PORT=4000

Then start both services:

docker compose up -d
docker compose logs --follow laya-cpu

The first start downloads and loads the model. After checking readiness using the README instructions, open:

http://127.0.0.1:4000

To try the sample access logs:

1. In Sources, edit Sample HTTP access and set its path to /host-logs/sample-logs/access-100.log.
2. Create a Batch / Manual Task.
3. Select that Source and the Sample HTTP access Criteria.
4. Set Events/Chunk to 1 and Action to None.
5. Save, enable the Task, and click Run.

New Tasks are disabled by default; saving alone does not start processing.

Change the trial API key before deployment. The README covers readiness checks, GPU execution, remote inference, and deployment configuration.

Current Limits

DeciFlow is still in early development.

• Validate decision quality against representative data before relying on it.
• TEXT chunking does not automatically detect multiline event boundaries such as stack traces.
• Inputs must fit the selected model's token budget.
• HDFS supports batch processing, not continuous tailing.
• Follow-up judgments require evidence in the source text.
• Actions need care because repeated runs can repeat external effects.

The most useful test is a small dataset you understand well: define the decisions, inspect the mistakes, and assess whether the results help your review process.

Feedback

I would especially appreciate feedback from people working with operational logs, customer inquiry records, or text datasets in HDFS.

Which decisions would save you time? What information would you need to trust—or reject—a prediction? Where would this workflow fit into your existing tools?

Code, samples, and documentation:

https://github.com/call518/DeciFlow