DEV Community: nk_Enuke

Kaggle:Trying to Reproduce the 1st Place NeurIPS Polymer Prediction Solution (Scaled-Down Experiment)

nk_Enuke — Sun, 22 Mar 2026 16:37:24 +0000

A post-competition reproduction attempt by someone with a chemistry background

The NeurIPS Open Polymer Prediction 2025 competition challenged about 2,000 teams to predict five physical properties of polymers from their chemical structure (SMILES strings).

I have a background in chemistry, so after the competition ended I read through James Day's 1st place writeup and tried to reproduce the approach as best I could. With a late submission, I got a Private LB score of 0.08180 — which should be roughly equivalent to 8th place out of 2,240 teams. Though of course, this was done after the competition closed, so it should be taken with a grain of salt.

Disclaimer: This article describes a scaled-down reproduction experiment. All solution design credit goes to James Day. My goal was simply to see how far I could get reproducing his approach with limited resources.

James Day's 1st Place Writeup:
https://www.kaggle.com/competitions/neurips-open-polymer-prediction-2025/writeups/1st-place-solution

nkwork9999 / NeurIP2025_mytrial_following_1st_solution

Reproducing the 1st Place NeurIPS Polymer Prediction 2025 Solution

Top-8 equivalent score (0.08180) on the Private Leaderboard using CodeBERTa-small

This repository contains a reproduction of James Day's 1st place solution for the NeurIPS - Open Polymer Prediction 2025 Kaggle competition.

Disclaimer: This is a reproduction study, not original work. All credit for the solution design goes to the original authors. The purpose is to verify reproducibility and provide a working codebase for the community.

Competition Overview

Task: Predict 5 polymer properties (Tg, FFV, Tc, Density, Rg) from chemical structure (SMILES)
Metric: Weighted Mean Absolute Error (wMAE)
Scale: 2,240 teams, $50,000 prize pool
Ground truth: Averaged from multiple molecular dynamics simulation runs

Results

Rank	Team	Score (wMAE)
1	James Day	0.07536
2	Ezra	0.07722
3	Ghy HUST CS	0.07820
7	CoderGirlM	0.08144
-	This reproduction	0.08180
8	Dmitry Uarov	0.08271

Late submission score of 0.08180 — equivalent to 8th place out…

View on GitHub

Getting Started

Clone the repo and install dependencies to run the pipeline yourself:

git clone https://github.com/nkwork9999/NeurIP2025_mytrial_following_1st_solution.git
cd NeurIP2025_mytrial_following_1st_solution

pip install -r requirements.txt

The training notebooks are designed to run on Google Colab (L4 GPU, ~7 hours). Open the notebooks in order:

Train CodeBERTa — Fine-tune with SMILES augmentation and 5-fold CV
Train AutoGluon — Tabular model on RDKit descriptors
Ensemble & submit — Combine predictions and apply post-processing

Refer to the repo's README for the full walkthrough.

Competition Overview

The Task

Given a polymer's SMILES notation, predict five molecular-dynamics-simulated physical properties:

Property	Description	Unit
Tg	Glass transition temperature	degC
FFV	Fractional free volume	-
Tc	Thermal conductivity	W/(m*K)
Density	Density	g/cm^3
Rg	Radius of gyration	angstrom

The Sparse Label Problem

The defining challenge of this competition was extreme label sparsity. Not every polymer had all five properties measured — the number of available samples varied widely across targets.

Evaluation Metric

Weighted Mean Absolute Error (wMAE), normalized by each property's value range, with higher weights assigned to properties with fewer samples. In other words, the rarest labels matter most.

James Day's 1st Place Solution

Day's winning approach was an ensemble of three models:

ModernBERT-base — Treats SMILES as text, with a regression head for property prediction
AutoGluon — A tabular model using RDKit descriptors and Morgan fingerprints as features
Uni-Mol 2 — A pretrained model that accounts for 3D molecular structure

The most surprising finding: a code-pretrained ModernBERT-base outperformed chemistry-specific models. SMILES notation resembles source code — parentheses, symbols, and repeating structural patterns — so a tokenizer trained on code captures SMILES structure remarkably well. I found this genuinely fascinating. It makes sense that SMILES, being string-based, can benefit from large language models, but the fact that a code-pretrained model specifically has an advantage was unexpected.

Insights from 2nd and 3rd Place

2nd place (Ezra): Simply converting Tg units from Celsius to Fahrenheit dramatically improved the score. Achieved a high rank with ExtraTreesRegressor — a relatively simple model.
3rd place (Hongyu Guo): GATv2Conv (6 layers) + Morgan fingerprints, with post-hoc linear regression calibration.

My Reproduction: Design Decisions

Why CodeBERTa-small?

Day used ModernBERT-base (125M parameters). I went with CodeBERTa-small (84M parameters) — honestly, I simply wanted to try things out easily on Google Colab. Training had to fit within the L4 GPU time limit (~7 hours), and I wanted to get something running quickly without too much setup.

Pipeline Architecture

SMILES --> CodeBERTa-small --> Regression Head --> 5 properties
                                    | (ensemble)
SMILES --> RDKit features --> AutoGluon --> 5 properties

Each target is trained with 5-fold cross-validation. At inference, 30 rounds of test-time augmentation (TTA) are applied.

The Key Technique: Random SMILES Augmentation

This is the single most impactful technique in the pipeline.

SMILES Are Not Unique

The same molecule can be represented by hundreds of different SMILES strings, depending on the atom traversal order:

Example: Ethanol
  Canonical: CCO
  Random 1:  OCC
  Random 2:  C(O)C

RDKit's MolToSmiles(mol, canonical=False, doRandom=True) generates randomized SMILES on the fly.

Training: 10x Augmentation

Each epoch presents the same molecule with a different SMILES representation (10x augmentation). This forces the model to learn representation-invariant molecular features rather than memorizing a specific string.

class SMILESDataset(Dataset):
    def __init__(self, smiles_list, labels, tokenizer, max_len=128, aug_factor=10):
        self.smiles = smiles_list
        self.labels = labels
        self.aug = aug_factor

    def __len__(self):
        return len(self.smiles) * self.aug

    def __getitem__(self, idx):
        ri = idx % len(self.smiles)
        smi = random_smiles(self.smiles[ri])  # Random SMILES each time
        ...

Inference: TTA with Median Aggregation

At test time, 30 random SMILES variants are generated per molecule, and the median prediction is taken as the final output. Median is preferred over mean to suppress outlier predictions.

def predict_tta(model, tokenizer, smiles_list, n_tta=30):
    all_preds = []
    for _ in range(n_tta):
        aug = [random_smiles(s) for s in smiles_list]
        preds = model.predict(aug)
        all_preds.append(preds)
    return np.median(all_preds, axis=0)  # Aggregate with median

Personal note: I find this technique quite intriguing. It feels somewhat analogous to adding noise to images in computer vision augmentation. I would have expected that randomizing the SMILES string might introduce confusing signals, but it actually works surprisingly well.

Post-Processing Tricks

Tg Distribution Shift Correction

There is a known distribution shift between the train and test Tg values. Adding std * 0.5644 to the predictions corrects for this shift. It is a rather analog/manual method, but it turned out to be quite effective.

tg_std = submission['Tg'].std()
tg_shift = tg_std * 0.5644
submission['Tg'] += tg_shift

The coefficient 0.5644 was found via grid search on out-of-fold predictions.

Direct Match

When a test SMILES exactly matches a training SMILES, the known training value is used directly instead of the model prediction. Simple but effective.

for target in TARGETS:
    for _, row in test.iterrows():
        match = train.loc[
            (train['SMILES'] == row['SMILES']) & (train[target].notna()), target
        ]
        if len(match) > 0:
            submission.loc[submission['id'] == row['id'], target] = match.values[0]

Results

Leaderboard Comparison

Rank	Team	Score
1st	James Day	0.07536
2nd	Ezra	0.07722
3rd	Ghy HUST CS	0.07820
7th	CoderGirlM	0.08144
--	This reproduction	0.08180
8th	Dmitry Uarov	0.08271

What I Could Not Reproduce

The biggest missing piece was Uni-Mol 2, the 3D molecular structure model that was part of the 1st place ensemble. I was unable to reproduce this component — I did not have the required software environment set up, and the compute requirements were beyond what I had available. This was a clear limitation of my attempt.

It made me wish there were lighter-weight methods for incorporating 3D molecular structure information. If something like a lightweight 3D-aware featurizer existed that could run on a single GPU, it would make these kinds of reproduction experiments much more practical.

Wrapping Up

There are many summary articles about top competition solutions, but fewer attempts to actually write the code, run the pipeline, and verify the results. I wanted to try doing that here, even in a scaled-down form.

This article may contain errors or misunderstandings — I would appreciate any corrections or feedback. If you notice something off, please feel free to leave a comment or open an issue on the repo.

nkwork9999 / NeurIP2025_mytrial_following_1st_solution

Reproducing the 1st Place NeurIPS Polymer Prediction 2025 Solution

Top-8 equivalent score (0.08180) on the Private Leaderboard using CodeBERTa-small

This repository contains a reproduction of James Day's 1st place solution for the NeurIPS - Open Polymer Prediction 2025 Kaggle competition.

Disclaimer: This is a reproduction study, not original work. All credit for the solution design goes to the original authors. The purpose is to verify reproducibility and provide a working codebase for the community.

Competition Overview

Task: Predict 5 polymer properties (Tg, FFV, Tc, Density, Rg) from chemical structure (SMILES)
Metric: Weighted Mean Absolute Error (wMAE)
Scale: 2,240 teams, $50,000 prize pool
Ground truth: Averaged from multiple molecular dynamics simulation runs

Results

Rank	Team	Score (wMAE)
1	James Day	0.07536
2	Ezra	0.07722
3	Ghy HUST CS	0.07820
7	CoderGirlM	0.08144
-	This reproduction	0.08180
8	Dmitry Uarov	0.08271

Late submission score of 0.08180 — equivalent to 8th place out…

View on GitHub

Fixing DuckDB Extension Deadlock: From Query Strings to LIST Types

nk_Enuke — Wed, 03 Sep 2025 15:00:00 +0000

My Project on Github
https://github.com/nkwork9999/icedduck

The Problem

When implementing a data visualization extension for DuckDB, I encountered a critical issue: executing SQL queries within a scalar function context caused the entire database to deadlock. The original implementation looked like this:

// This causes deadlock!
auto x_result = context.Query(x_query_str, false);
auto y_result = context.Query(y_query_str, false);

The Solution: Using LIST Types

Instead of accepting SQL query strings and executing them within the function, the solution was to accept pre-aggregated LIST types directly. This approach is both safer and more idiomatic for DuckDB extensions.

Key Changes

Function Signature Update

// Before: Accepting query strings
auto bar_chart_func = ScalarFunction(
    "bar_chart",
    {LogicalType::VARCHAR, LogicalType::VARCHAR, LogicalType::VARCHAR},
    LogicalType::VARCHAR,
    BarChartFunction
);

// After: Accepting LIST types
auto bar_chart_func = ScalarFunction(
    "bar_chart",
    {LogicalType::LIST(LogicalType::VARCHAR), 
     LogicalType::LIST(LogicalType::DOUBLE), 
     LogicalType::VARCHAR},
    LogicalType::VARCHAR,
    BarChartArrayFunction
);

Data Extraction Logic

// Extract data from LIST types
auto x_list_value = x_list.GetValue(0);
if (!x_list_value.IsNull() && x_list_value.type().id() == LogicalTypeId::LIST) {
    auto &x_children = ListValue::GetChildren(x_list_value);
    for (idx_t i = 0; i < x_children.size(); i++) {
        if (i > 0) x_data += ",";
        x_data += x_children[i].ToString();
    }
}

How to Use

Prerequisites

Build the Rust chart viewer:

cd rust_hello_duck
cargo build --release
cd ..

Build the DuckDB extension:

make clean
make

Running the Extension

-- Start DuckDB
./build/release/duckdb

-- Load the extension
LOAD 'build/release/extension/quack/quack.duckdb_extension';

-- Create sample data
CREATE TABLE sales (category VARCHAR, amount DECIMAL);
INSERT INTO sales VALUES 
    ('Electronics', 1500.50), 
    ('Clothing', 800.25), 
    ('Food', 1200.00);

-- Generate bar chart using LIST aggregation
SELECT bar_chart(
    LIST(category ORDER BY amount DESC), 
    LIST(amount ORDER BY amount DESC), 
    'Sales Chart'
) FROM sales;

What Happens

The LIST() functions aggregate the data in DuckDB's query engine
The aggregated data is passed to the extension function
The function writes the data to a temporary file
A separate Rust process is spawned to display the chart using Iced GUI

Advanced Usage

You can use more complex aggregations:

-- Group by month and sum revenues
WITH monthly_data AS (
    SELECT 
        month,
        SUM(revenue) as total_revenue
    FROM monthly_sales
    GROUP BY month
    ORDER BY total_revenue DESC
)
SELECT bar_chart(
    LIST(month),
    LIST(total_revenue),
    'Monthly Revenue Analysis'
) FROM monthly_data;

Key Takeaways

Avoid Nested Queries: Never execute SQL queries within DuckDB function implementations
Use Aggregate Types: LIST types are perfect for passing collections of data
Separate Concerns: Data aggregation happens in SQL, visualization in the extension
Process Isolation: GUI applications should run in separate processes to avoid threading issues

Performance Benefits

This approach is not only safer but also more performant:

No query parsing overhead within the function
Direct memory access to aggregated data
Better query optimization by DuckDB's planner

The complete implementation demonstrates how proper function design can avoid common pitfalls in database extension development while providing a clean, efficient interface for users.

Building a Real-time Data Visualization Extension for DuckDB with Rust and Iced

nk_Enuke — Mon, 01 Sep 2025 09:58:18 +0000

Introduction

This article demonstrates how to create a DuckDB extension that generates interactive charts using Rust and the Iced GUI framework. We'll build a system that executes SQL queries and displays the results as bar charts in a native window, bridging the gap between database analytics and visual representation.

Project Setup and Architecture
Creating the Rust Library with FFI
Building the C++ Extension Wrapper
Implementing the Iced Chart Viewer
Handling macOS Threading Constraints
Dynamic Data Visualization
Troubleshooting and Lessons Learned

Chapter 1: Project Setup and Architecture

Overview

Our architecture consists of three main components:

A C++ DuckDB extension that provides SQL functions
A Rust library exposed via FFI (Foreign Function Interface)
A standalone Iced application for rendering charts

Directory Structure

iced_duck/
├── src/                    # C++ extension
│   ├── quack_extension.cpp
│   └── include/
│       └── quack_extension.hpp
├── rust_hello_duck/        # Rust library
│   ├── Cargo.toml
│   ├── src/
│   │   ├── lib.rs
│   │   └── bin/
│   │       └── chart_viewer.rs
├── CMakeLists.txt
└── Makefile

Chapter 2: Creating the Rust Library with FFI

Basic FFI Setup

We started with a simple Rust library exposing C-compatible functions:

use std::ffi::{c_char, CString};

#[no_mangle]
pub extern "C" fn rust_hello_world() -> *const c_char {
    let message = CString::new("Hello from Duck!").unwrap();
    message.into_raw()
}

#[no_mangle]
pub extern "C" fn rust_hello_free(s: *mut c_char) {
    unsafe {
        if !s.is_null() {
            let _ = CString::from_raw(s);
        }
    }
}

Key Learnings

Memory Management: Always provide a free function for allocated strings
Safety Attributes: Modern Rust requires explicit unsafe markers for FFI functions
Cross-platform Compatibility: Handle different library extensions (.dylib, .so, .dll)

Chapter 3: Building the C++ Extension Wrapper

Dynamic Library Loading

The C++ extension loads the Rust library at runtime:

void* rust_lib = dlopen(lib_path.c_str(), RTLD_LAZY);
if (!rust_lib) {
    printf("Warning: Could not load Rust library\n");
    return;
}

// Get function pointers
auto rust_hello_world = (rust_hello_world_fn)dlsym(rust_lib, "rust_hello_world");

Registering SQL Functions

auto rust_hello_func = ScalarFunction("rust_hello", {}, LogicalType::VARCHAR, RustHelloScalarFun);
ExtensionUtil::RegisterFunction(instance, rust_hello_func);

Chapter 4: Implementing the Iced Chart Viewer

Chart Application Structure

We created a separate binary for the chart viewer to handle GUI rendering:

struct ChartApp {
    data: ChartData,
}

impl Application for ChartApp {
    type Message = Message;
    type Theme = Theme;
    type Executor = iced::executor::Default;
    type Flags = ChartData;

    fn view(&self) -> Element<Message> {
        let chart = canvas(self as &Self)
            .width(Length::Fill)
            .height(Length::Fill);

        container(
            column![
                text(&self.data.title).size(24),
                chart,
            ]
            .spacing(10)
        )
        .padding(20)
        .into()
    }
}

Canvas Drawing

The actual chart rendering uses Iced's canvas API:

impl<Message> canvas::Program<Message> for ChartApp {
    fn draw(&self, _state: &Self::State, renderer: &iced::Renderer, 
            _theme: &Theme, bounds: Rectangle, _cursor: iced::mouse::Cursor) -> Vec<Geometry> {
        let mut frame = Frame::new(renderer, bounds.size());

        // Draw bars
        for (i, &value) in self.data.y_data.iter().enumerate() {
            let height = (value / max_value) * chart_height * 0.8;
            frame.fill_rectangle(
                Point::new(x, y),
                Size::new(bar_width, height as f32),
                Color::from_rgb(0.2, 0.6, 0.9),
            );
        }

        vec![frame.into_geometry()]
    }
}

Chapter 5: Handling macOS Threading Constraints

The Problem

On macOS, GUI applications must run on the main thread. Our initial approach using thread::spawn resulted in:

thread '<unnamed>' panicked at:
on macOS, `EventLoop` must be created on the main thread!

The Solution

We launched the chart viewer as a separate process:

pub extern "C" fn rust_show_chart() -> *const c_char {
    match Command::new("rust_hello_duck/target/release/chart_viewer")
        .spawn() {
        Ok(_) => {
            let message = CString::new("Chart viewer launched").unwrap();
            message.into_raw()
        }
        Err(e) => {
            let message = CString::new(format!("Failed: {}", e)).unwrap();
            message.into_raw()
        }
    }
}

Chapter 6: Dynamic Data Visualization

SQL Function Interface

We created a bar_chart function that accepts SQL queries:

SELECT bar_chart(
    'SELECT category FROM sales', 
    'SELECT amount FROM sales', 
    'Sales by Category'
);

Data Transfer

Instead of complex JSON serialization, we used a simple text format:

// Save data to file
std::ofstream file("/tmp/duckdb_chart_data.txt");
file << title_str << "\n";
file << x_data << "\n";  // comma-separated
file << y_data << "\n";  // comma-separated
file.close();

Chapter 7: Troubleshooting and Lessons Learned

Common Issues Encountered

Library Path Resolution: Always try both relative and absolute paths
Deadlocks in Query Execution: The context.Query() method can cause deadlocks when called from within a DuckDB function
Type Compatibility: Careful handling of numeric types between C++ and Rust
Build System Complexity: Managing both CMake and Cargo builds requires coordination

Best Practices

Start Simple: Begin with basic FFI functions before adding complexity
Debug Incrementally: Add print statements at each stage of execution
Handle Errors Gracefully: Always provide fallback behavior
Test Cross-platform Early: Platform-specific issues can be significant

Conclusion

This project demonstrates the power of combining different technologies:

DuckDB for SQL processing
Rust for safe systems programming
Iced for modern GUI development
FFI for language interoperability

While the integration presents challenges, particularly around threading and build systems, the result is a powerful system that can transform SQL query results into interactive visualizations.

The complete source code and additional examples are available in the project repository. Feel free to extend this foundation with additional chart types, improved error handling, or real-time data updates.

Future Enhancements

Support for multiple chart types (line, pie, scatter)
Real-time data streaming
Interactive chart features (zoom, pan, tooltips)
Better error handling and user feedback
Cross-platform installer generation

Happy coding, and may your data always be beautifully visualized!

BigQuery AI - Building the Future of Data:Day1

nk_Enuke — Tue, 19 Aug 2025 15:00:00 +0000

Overview

The goal is to build a prototype that processes unstructured data held by companies (chat logs, PDFs, screenshots, recordings, etc.) using BigQuery's AI capabilities to solve real-world business problems.
I'm currently studying GCP data engineering and thought this would be perfect, so I decided to participate.
The content is unique in that you select from 3 approaches (minimum 1 selection):

Utilizing BigQuery's generative AI features
Vector search: ML.GENERATE_EMBEDDING, VECTOR_SEARCH, etc.
Integrated analysis of structured/unstructured data: Object Tables, ObjectRef, Multimodal DataFrame, etc. Also, the evaluation criteria are different from typical Kaggle competitions:
Technical Implementation (35%): Code quality, effective use of BigQuery AI
Innovation and Creativity (25%): Novelty of solution, business impact
Demo and Presentation (20%): Clarity of problem definition, documentation quality
Assets (20%): Quality of blog/video, public GitHub repository
Bonus (10%): Feedback, survey submission

Competition Without Dataset

The most distinctive feature is that no data is provided
Participants are free to choose and use publicly available datasets...
For example, the following datasets are recommended:
BigQuery public datasets overview page
BigQuery public dataset marketplace
Image datasets provided through Cloud Storage

Code Reading

Code by Dao Sy Duy Minh
https://www.kaggle.com/code/daosyduyminh/simple-tutorial-weather-forecasting
Demonstrates 3 approaches for weather forecasting using BigQuery.
Appears to be using Hanoi weather data.

Python and Prophet It seems you can use Prophet through bigquery.Client().
BigQuery ML You can retrieve datasets with dataset = bigquery.Dataset(dataset_id) and use them directly. Usage is as simple as writing client.query(train_query), requiring relatively little code.
BigQuery Generative AI Large-scale predictions are possible using FROM AI.FORECAST

Kaggle Diary: MITSUI&CO. Commodity Prediction Challenge: Day 1

nk_Enuke — Mon, 18 Aug 2025 08:00:00 +0000

Competition URL

https://www.kaggle.com/competitions/mitsui-commodity-prediction-challenge/rules#7-competition-data

Overview

The objective of this competition is to develop accurate and stable prediction models for commodity prices, which seems relatively straightforward. The data is lightweight, and unlike recent competitions that require high-spec PCs just to participate, I felt this was something I could actually join, so I decided to give it a try.

Data

The dataset consists of multiple time series data obtained from financial markets worldwide, including various financial products such as metals, futures, US stocks, and foreign exchange. The markets include:

LME: London Metal Exchange
JPX: Japan Exchange Group
US: Various US securities exchanges
FX: Foreign Exchange

Key features:

1,977 columns - quite a large-scale dataset
4 markets: LME (London metals), JPX (Japan), US (US stocks), FX (foreign exchange)
424 targets: Single commodity returns and differences between commodity pairs
1-4 day lags: Split into test_labels_lag_[1-4].csv
Need to consider lags due to financial institution holidays and processing time

Prediction

Apparently, the actual leaderboard isn't very useful, as the evaluation is based on how well predictions match actual values after the competition ends.

Code Reading

Reading public code shared by maverick_ss_26.

Examining Code to Verify if the Leaderboard is Really Invalid

https://www.kaggle.com/code/maverickss26/commodity-price-prediction-v1

Preprocessing and Other Tips

Check data quality by summing null values: .isnull().sum()
Display missing values using histograms
Use Prophet time series model
Express price volatility using standard deviation

About Prophet Model Options

changepoint_prior_scale=0.05,  # Trend change flexibility (smaller = more stable)
interval_width=0.95  # 95% confidence interval for predictions

General Example of Prophet Usage

from prophet import Prophet
import pandas as pd

# Data preparation (required: ds and y columns)
df = pd.DataFrame({
    'ds': ['2024-01-01', '2024-01-02', '2024-01-03'],  # Dates
    'y': [100, 120, 110]  # Values to predict
})

# Create and train model
model = Prophet()
model.fit(df)

# Create future dates
future = model.make_future_dataframe(periods=7)  # 7 days ahead

# Execute prediction
forecast = model.predict(future)
print(forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']])

It can be used quite easily like this.
It appears that the test data contains training data, which shows that the leaderboard is meaningless.

Reading CSVs with varying column counts that pandas cannot read using DuckDB

nk_Enuke — Sat, 16 Aug 2025 07:26:27 +0000

What are CSV files with varying column counts?

6, 54
58,81,62
75,84,64,21,55
20,71,55,32

These are what they look like.
For example, there are cases where CSV files output from devices have rows below the first line that contain more columns than expected.

Let's try converting this file to a pandas DataFrame

So, let's mindlessly convert it to a pandas DataFrame with the following code:

import pandas as pd
df = pd.read_csv("./csv_file1.csv")
print(df)

This resulted in the following error:

pandas.errors.ParserError: Error tokenizing data. C error: Expected 3 fields in line 3, saw 5

This seems to mean that line 3 had 5 columns when 3 were expected, causing an error.
It appears that pandas looks at the first few lines (2 lines in this case, though it's unclear if 2 is the default) to determine the number of columns in the DataFrame. So when there are more columns than expected, it results in an error and the data cannot be read.

Solution 1: Find the maximum number of columns and fill the DataFrame with empty values for missing parts

So I thought I could open the CSV file with something other than pandas, find the maximum number of columns, and then fill rows with fewer columns with empty values. Here's the code I created:

import pandas as pd
with open('./csv_file1.csv', 'r', encoding='utf-8') as file:
    lines = file.readlines()
comma_count_max = max(line.count(',') for line in lines)
columns = [f'col_{i+1}' for i in range(comma_count_max + 1)]
data = []
for line in lines:
    row_data = line.strip().split(',')
    # Add empty strings as needed
    if len(row_data) < len(columns):
        row_data += [''] * (len(columns) - len(row_data))
    data.append(row_data)
df = pd.DataFrame(data, columns=columns)
print(df)

Since pandas gives an error when reading, I open the file with Python's open() function and find the line with the most commas among all lines. The number of commas + 1 gives us the maximum number of columns, so then I fill rows with fewer columns with empty strings.
The result is:

  col_1 col_2 col_3 col_4 col_5
0     6    54                  
1    58    81    62            
2    75    84    64    21    55
3    20    71    55    32

This achieves the requirement, but for example...

6, 54
58,81,62,,,,,
75,84,64,21,55
20,71,55,32

For data like this, the result would be:

  col_1 col_2 col_3 col_4 col_5 col_6 col_7 col_8
0     6    54                                    
1    58    81    62                              
2    75    84    64    21    55                  
3    20    71    55    32

This is fine for a few rows, but I thought it would be quite heavy to do the process of opening CSV files and adding maximum comma count data to each row for large files.

Solution 2: Use DuckDB's null_padding option

DuckDB's read_csv has a null_padding option that fills blank parts with null values.

import duckdb
con = duckdb.connect(database=":memory:")
print(con.sql("SELECT * FROM read_csv('csv_file1.csv',null_padding=true)"))

Running these three lines of code gives us...?

┌─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
│ column0 │ column1 │ column2 │ column3 │ column4 │ column5 │ column6 │ column7 │
│  int64  │  int64  │  int64  │  int64  │  int64  │ varchar │ varchar │ varchar │
├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│       6 │      54 │    NULL │    NULL │    NULL │ NULL    │ NULL    │ NULL    │
│      58 │      81 │      62 │    NULL │    NULL │ NULL    │ NULL    │ NULL    │
│      75 │      84 │      64 │      21 │      55 │ NULL    │ NULL    │ NULL    │
│      20 │      71 │      55 │      32 │    NULL │ NULL    │ NULL    │ NULL    │
└─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘

It automatically fills up to the maximum number of columns with nulls and even creates headers for the maximum number of columns.
(Though naturally, the number of columns created matches the maximum number of commas.)
By the way, when you want to convert to a DataFrame:

import duckdb
con = duckdb.connect(database=":memory:")
print(con.sql("SELECT * FROM read_csv('csv_file1.csv',null_padding=true)").df())

Adding .df() at the end like this converts it to a pandas DataFrame.
If you're facing this issue, please give it a try! 🙌

DuckDB 1.20 Released! Exploring the New Features

nk_Enuke — Wed, 30 Apr 2025 11:12:59 +0000

Introduction

DuckDB has been updated to version 1.20. 🙌

It seems that various features have been added, so I tried out some of the new functionality mentioned in the official blog.

Reference

https://duckdb.org/2025/02/05/announcing-duckdb-120.html

Codename for this Update

The codename for this release is "Histrionicus"

This refers to the Harlequin Duck (scientific name: Histrionicus histrionicus) that lives in cold rivers in North America, Greenland, Iceland, and Eastern Russia.

The Japanese name is "Shinorigamo," and they apparently fly to coastal areas of Hokkaido and the Tohoku region in winter.

I looked at some images, but since I've always lived in the western part of Japan, it has an appearance I don't think I've seen before. 🦆

Common Python Code Setup

I'm using the following code to enable DuckDB in Python (the usual setup):

import duckdb

# Create an in-memory DuckDB database
con = duckdb.connect(database=':memory:')

① RANDOM() update now outputs more types of random numbers

In this update, the random output has been changed from 32-bit to 64-bit.

So the previous limit:

32-bit maximum: 4,294,967,295 variations

has become:

64-bit maximum: 18,446,744,073,709,551,615 variations

I tried the following code to test this. I generated 5 billion random numbers (note: this takes about 10 minutes).

# Generate a large number of random numbers
query = """
SELECT 
    COUNT(DISTINCT random_value) AS unique_count,
    MIN(random_value) AS min_value,
    MAX(random_value) AS max_value
FROM (
    SELECT CAST(RANDOM() * (2 ** 63) AS BIGINT) AS random_value
    FROM range(1, 5000000000)
) AS subquery
"""

# Execute the query and get results
result = con.sql(query)
print(result)

max_32bit = (2 ** 32) - 1
print(f"32-bit maximum: {max_32bit}")
max_64bit = (2 ** 64) - 1
print(f"64-bit maximum: {max_64bit}")

Results:

┌──────────────┬────────────┬─────────────────────┐
│ unique_count │ min_value  │      max_value      │
│    int64     │   int64    │        int64        │
├──────────────┼────────────┼─────────────────────┤
│   4999999093 │ 3023944772 │ 9223372031664709632 │
└──────────────┴────────────┴─────────────────────┘

32-bit maximum: 4294967295
64-bit maximum: 18446744073709551615

Looking at the output, it seems the unique values indeed exceed the 32-bit limit of about 4.3 billion types.

(I'm not completely sure if this is the meaning of the update.)

② Map return values changed from lists to single values

SELECT map(['k'], ['v'])['k'];

This creates a dictionary (map) with k and v, and when called with key k, the result should be v.

When running this code previously, the result was apparently:

Old:

['v']

With the update, it's now:

New:

┌────────────────────────────────────────────────────────┐
│ "map"(main.list_value('k'), main.list_value('v'))['k'] │
│                        varchar                         │
├────────────────────────────────────────────────────────┤
│ v                                                      │
└────────────────────────────────────────────────────────┘

This could be a significant change for those who have been using map, as the return value has changed from a list to a single string.

③ Adding primary keys to existing tables

It seems you can now add primary keys to tables you've already created!

data = pd.DataFrame({
    "id": [1, 2, 3, 4, 5],
    "name": ["Alice", "Bob", "Charlie", "David", "Eve"],
    "age": [25, 30, 35, 40, 45]
})

# Create table (initially without primary key)
con.sql("CREATE OR REPLACE TABLE tbl AS SELECT * FROM data")

result_df = con.sql("SELECT * FROM tbl")

print(result_df)

con.sql("ALTER TABLE tbl ADD PRIMARY KEY (id);")

con.sql("INSERT INTO tbl VALUES (5, 'Frank', 50)")

After adding a primary key to the existing table, if you try to insert data with ID 5 which already exists:

con.sql("INSERT INTO tbl VALUES (5, 'Frank', 50)")
duckdb.duckdb.ConstraintException: Constraint Error: Duplicate key "id: 5" violates primary key constraint.

It's rejected due to the primary key constraint that prevents duplicate values.

Also, previously you couldn't insert data with the same ID as a record you had deleted, but that has been improved and is now possible:

con.sql("""
DELETE FROM tbl WHERE id = 1;
""")
con.sql("""
INSERT INTO tbl VALUES (1, 'Dareka',100);
""")
print(con.sql("SELECT * FROM tbl"))

The insertion succeeds as shown below:

┌───────┬─────────┬───────┐
│  id   │  name   │  age  │
│ int64 │ varchar │ int64 │
├───────┼─────────┼───────┤
│     2 │ Bob     │    30 │
│     3 │ Charlie │    35 │
│     4 │ David   │    40 │
│     5 │ Eve     │    45 │
│     1 │ Dareka  │   100 │
└───────┴─────────┴───────┘

④ 🦆🦆🦆

Surprisingly, you can now use 🦆 as a separator when reading CSV files!

This means you can use 4-byte characters (the duck emoji is apparently 4 bytes) as separators.

So if you have a CSV file with this data:

a🦆b
hello🦆world

And run this code:

print(con.sql("SELECT * FROM read_csv('csv_file1.csv',sep = '🦆')"))

You get this result:

┌─────────┬─────────┐
│    a    │    b    │
│ varchar │ varchar │
├─────────┼─────────┤
│ hello   │ world   │
└─────────┴─────────┘

It splits nicely.

However, if you create a CSV like:

a🇯🇵b
hello🇯🇵world

And run:

print(con.sql("SELECT * FROM read_csv('csv_file1.csv',sep = '🇯🇵')"))

You get this error:

duckdb.duckdb.InvalidInputException: Invalid Input Error: The delimiter option cannot exceed a size of 4 bytes.

This is because 🇯🇵 exceeds 4 bytes (apparently it's 6 bytes) and can't be used as a separator. I found it interesting to think about emoji byte sizes, which I'd never considered before.

I also thought it was quite remarkable that the original text used the English word "emoji." Looking it up, I found that emoji were first used in 1999 in NTT DoCoMo's i-mode service for mobile phones. I've been using emoji since the feature phone era but didn't know this history. They were standardized globally and spread as "emoji" in 2011.

⑤ strict_mode to only read CSVs that follow RFC 4180

There are many types of CSV files, including haphazard ones like:

1,2,3
a,b,c,d,e,f,g
h
i,j,k,l

With the update, you can now enforce reading only CSVs that follow the strict CSV standard (RFC 4180).

If you want to read non-compliant CSVs, you can use:

FROM read_csv('rfc_4180-defiant.csv', strict_mode = false);

This could be useful for filtering out messy CSVs when you want to use reliable data sources.

⑥ SQL column and table name aliases can now be set at the beginning without using AS

Normally in SQL, you use AS for aliases:

SELECT 
    some_long_and_winding_expression AS e1,
    t2.a_column_name AS e2
FROM
    long_schema.some_long_table_name AS t1,
    short_s.tbl AS t2;

With this update, you can now set column names like this (which is indeed more readable):

SELECT 
    e1: some_long_and_winding_expression,
    e2: t2.a_column_name 
FROM
    t1: long_schema.some_long_table_name,
    t2: short_s.tbl;

There are many other updates, and more details will apparently be written in a blog post in the near future, which I'm looking forward to. 🙌

A story about prototyping a 'Desktop Directory Structure Visualization Tool' with Tauri

nk_Enuke — Wed, 25 Sep 2024 22:00:20 +0000

[Overview]
You might feel strange hearing about desktop visualization (since the desktop is already visible).
However, after recently trying out BI tools, I felt that the necessary information isn't visualized by default.
So I thought I'd create an app for myself that could visualize the information that comes out of the tree command.

[URL]
https://github.com/nkwork9999/tree_viz/tree/vizonly

[Purpose]
Create and publish a personal application.
Visualize folder structure in a tree format.
Open folders when clicked.

[Frameworks Used]
Tauri (Rust × TypeScript): Allows writing desktop application GUI with React.
ReactFlow: Create flowchart diagrams with React.
MUI: Used for screen creation.

[What I'm Happy About]
Although it was my first time properly creating a desktop application as a hobby, Tauri is so excellent that the app launches at lightning speed with a double-click.
Backend processing is also fast. 🙌

[Learnings]
I was able to work with Rust, which increased the number of things I want to learn and frameworks I want to try next (creating my own OS, WASM, data analysis with Rust, etc.)
Learned how to output data from the backend using the ReactFlow library. (Can it be used for ER diagrams too?)
Being able to write desktop application GUIs in React means not having to learn specific syntax for GUI libraries for each language. (If Tauri can establish itself...)

[Unclear Points]
Implementation of the tree command in the backend... How to handle security and application publishing (publishing to App Store, how to write terms of service, etc.)
Apparently, a dialog asking for permission appears by default when launching the tool.
Also, logic (DTM application files) seems to be recognized as directories, and the cause of this is unclear.
There are many other unclear points remaining.
In conclusion, for the time being, I won't publish it on the App Store, but will make it public on GitHub.
I would appreciate any comments or feedback. 🙏

Tauri dialog instead of window.confirm

nk_Enuke — Mon, 19 Aug 2024 15:04:28 +0000

[Background]
I used window.confirm to confirm actions before calling backend processes. But, Tauri’s dialog are also available,so I tried to use it.

[Conclusion]
window.confirm has minimal customization.
But Tauri’s dialog has various customizations.
This could be useful when you don't need many customization just like MUI.

Before

typescript

import { invoke } from "@tauri-apps/api/tauri";

const executeCommands = () => {
    invoke<string>("command");
};

...

const handleCommand = async () => {
    const Confirm = await window.confirm("Do you want to execute the hello command?");
    if (Confirm) {
        executeCommands();
    }
};

...

return (
  <div>
      <button onClick={handleCommand}>Execute</button>
  </div>
);

After

typescript

import { invoke } from "@tauri-apps/api/tauri";
import { dialog } from "@tauri-apps/api";

const executeCommands = () => {
    invoke<string>("command");
};

...

const handleCommand = async () => {
    const Confirm = await dialog.ask("Do you want to execute the hello command?", "Ask Dialog");
    if (Confirm) {
        // yes
        executeCommands();
    }
};

...

return (
  <div>
      <button onClick={handleCommand}>Execute</button>
  </div>
);

In tauri.conf.json

json

{
  "tauri": {
    "allowlist": {
      "all": false,
      "dialog": {
        "confirm": true,
        "open": true,
        "ask": true
      }
    }
  }
}

[Rust]How to make string handing to frontend on tauri app

nk_Enuke — Thu, 25 Jul 2024 21:58:04 +0000

#[tauri::command]
pub fn simple_command()->String{
    "String".to_string()
}

If you want to return a string in a Tauri command,add to_string() to the end of the string just like this.
This allows you to pass any string to the frontend like React.

[Rust]get file name

nk_Enuke — Mon, 22 Jul 2024 22:09:25 +0000

Mypath.file_name().unwrap().to_string_lossy().into_owned();

Mypath:type of object
file_name():method of retreaving tail of path and return Option<&OsStr>
unwrap(): method of retreaving Option type
to_string_lossy():method of changing to string(lossy is just like forced)
.into_owned():method of changing to string have Cow

[Rust]Making struct for getting directory path

nk_Enuke — Sun, 21 Jul 2024 22:10:17 +0000

#[derive(Serialize, Deserialize)]
struct DirNode{
    name:String,
    children:Vec<DirNode>,
}

1:#[derive(Serialize, Deserialize)]
Making serialize and deserialize for struct to json
2:struct DirNode{}
Making struct named DirNode
3:children:Vec
'children' retain subdirectory's list.It means to make recursively storing a list of DirNode.
DirNode can contain multiple DirNode.
So 'children' can have subdirectories.
4:Vec
Making array can have multiple values by using heap allocation.

DEV Community: nk_Enuke

Kaggle:Trying to Reproduce the 1st Place NeurIPS Polymer Prediction Solution (Scaled-Down Experiment)

A post-competition reproduction attempt by someone with a chemistry background

nkwork9999 / NeurIP2025_mytrial_following_1st_solution

Reproducing the 1st Place NeurIPS Polymer Prediction 2025 Solution

Competition Overview

Results

Getting Started

Competition Overview

The Task

The Sparse Label Problem

Evaluation Metric

James Day's 1st Place Solution

Insights from 2nd and 3rd Place

My Reproduction: Design Decisions

Why CodeBERTa-small?

Pipeline Architecture

The Key Technique: Random SMILES Augmentation

SMILES Are Not Unique

Training: 10x Augmentation

Inference: TTA with Median Aggregation

Post-Processing Tricks

Tg Distribution Shift Correction

Direct Match

Results

Leaderboard Comparison

What I Could Not Reproduce

Wrapping Up

nkwork9999 / NeurIP2025_mytrial_following_1st_solution

Reproducing the 1st Place NeurIPS Polymer Prediction 2025 Solution

Competition Overview

Results

Fixing DuckDB Extension Deadlock: From Query Strings to LIST Types

The Problem

The Solution: Using LIST Types

Key Changes

How to Use

Prerequisites

Running the Extension

What Happens

Advanced Usage

Key Takeaways

Performance Benefits

Building a Real-time Data Visualization Extension for DuckDB with Rust and Iced

Introduction

Table of Contents

Chapter 1: Project Setup and Architecture

Overview

Directory Structure

Chapter 2: Creating the Rust Library with FFI

Basic FFI Setup

Key Learnings

Chapter 3: Building the C++ Extension Wrapper

Dynamic Library Loading

Registering SQL Functions

Chapter 4: Implementing the Iced Chart Viewer

Chart Application Structure

Canvas Drawing

Chapter 5: Handling macOS Threading Constraints

The Problem

The Solution

Chapter 6: Dynamic Data Visualization

SQL Function Interface

Data Transfer

Chapter 7: Troubleshooting and Lessons Learned

Common Issues Encountered

Best Practices

Conclusion

Future Enhancements

BigQuery AI - Building the Future of Data:Day1

Overview

Competition Without Dataset

Code Reading

Kaggle Diary: MITSUI&CO. Commodity Prediction Challenge: Day 1

Competition URL

Overview

Data

Prediction

Code Reading

Examining Code to Verify if the Leaderboard is Really Invalid