DolphinDB X Kronos: A Smarter Way to Train on Massive K-Line Data

Predicting financial markets from K-line data is a very different challenge from forecasting temperature or traffic. The patterns are noisy, the market is constantly changing, and the same price movement can mean very different things in different conditions.

That's why simply applying a general-purpose time-series model to financial data doesn't always work. Kronos takes a different approach: instead of treating K-lines as continuous numerical sequences, it converts them into discrete tokens, borrowing an idea from large language models to learn the patterns and dynamics of financial markets.

As the first open-source foundation model built specifically for financial K-line data, Kronos has surpassed 20,000 GitHub stars and can be used for tasks such as price forecasting, volatility prediction, and synthetic K-line generation.

But as the dataset grows, efficiently feeding massive amounts of K-line data into the training pipeline becomes a challenge of its own.

We address this challenge by making DolphinDB a direct data source for Kronos, scaling from a simple K-line forecasting example to minute-level, multi-stock, large-scale historical data and multi-GPU DDP training.

Quick Start: Running Kronos on Real K-Line Data

This example was tested with DolphinDB Server 3.00.5, Python 3.12, CUDA 13.2, and PyTorch 2.11.

After installing the required dependencies, load the pretrained Kronos-small model and tokenizer:

tokenizer = KronosTokenizer.from_pretrained("NeoQuasar/Kronos-Tokenizer-base")
model = Kronos.from_pretrained("NeoQuasar/Kronos-small")
predictor = KronosPredictor(model, tokenizer, max_context=512)

Next, use the DolphinDB Python API to query real stock K-line data and prepare it in the format expected by Kronos. We retrieve 5-minute K-lines for a stock from 2025 onward here:

s = ddb.session()
s.connect("localhost", 8848, "admin", "123456")  # Replace with your actual IP
script = '''
    select trade_time as timestamps, open, high, low, close, vol as volume, amount 
    from loadTable("dfs://tushare_minute_db", "stock_5min_k") 
    where trade_date >= 2025.01.01 and code = `000514.SZ
'''
df = s.run(script)
df['timestamps'] = pd.to_datetime(df['timestamps'])

With the data ready, use the previous 400 time points as the input and predict the next 100 time points:

lookback = 400
pred_len = 100
x_df = df.loc[:lookback-1, ['open', 'high', 'low', 'close', 'volume', 'amount']]
x_timestamp = df.loc[:lookback-1, 'timestamps']
y_timestamp = df.loc[lookback:lookback+pred_len-1, 'timestamps']
pred_df = predictor.predict(
    df=x_df,
    x_timestamp=x_timestamp,
    y_timestamp=y_timestamp,
    pred_len=pred_len,
    T=1.0,
    top_p=0.9,
    sample_count=1
)

The complete DolphinDB → Kronos → K-line forecasting pipeline is now up and running.

Scaling Up: How Can Kronos Efficiently Consume K-Line Data?

The example above works well for small-scale K-line forecasting. But training or fine-tuning Kronos on minute-level, multi-stock, and large-scale historical data requires a more scalable data pipeline.

DolphinDB provides three ways to feed data into Kronos, depending on the dataset size:

The first two approaches are straightforward: load small datasets into memory, or batch medium-sized datasets into PKL files for PyTorch DataLoader.

But as the dataset grows, both rely on local memory or files—which is where DDBDataLoader comes in.

DDBDataLoader: Making DolphinDB a Direct Data Source for Training

During training, DDBDataLoader retrieves data from DolphinDB on demand and organizes it into batches for PyTorch. For large-scale K-line training, this approach reduces the need for intermediate files and local data management.

For multi-GPU training, data also needs to be distributed across processes. DDBDataLoader can work with DolphinDB's partitioned storage to assign different data ranges to each GPU based on rank and world_size.

For Kronos workloads involving minute-level, multi-stock, and large-scale K-line data, DDBDataLoader provides a scalable data input layer: read directly for small datasets, batch and save to disk for medium-sized datasets, and let DolphinDB feed data directly to the training pipeline at scale.

Tips

Estimate steps_per_epoch manually. DDBDataLoader loads data on demand and cannot determine the dataset size. Set it slightly higher to avoid leaving training data unused.

Choose partitions carefully. For stock K-line data, partitioning by stock code can reduce data scanning and improve query performance.

Install dolphindb_tools in advance.

Pay attention to data loading. As the dataset grows, efficient data loading is essential for keeping GPUs continuously supplied and maintaining overall training performance.

Got historical market data? Why not give DolphinDB × Kronos a try and see what it can do for your forecasting workflow?

For the full code, configuration, and detailed DDBDataLoader guide, please contact us at [e-mail].

Visit the DolphinDB Website to explore the free trial and get started.