Build a daily stock data pipeline in Python: yfinance or EODHD APIs into SQLite

jun yin — Tue, 13 Oct 2026 03:56:43 +0000

Disclosure: I run aiquantnav.com, which is an affiliate partner of EODHD APIs. The link to their plans below is an affiliate link: if you sign up through it I may earn a commission, at no extra cost to you. The yfinance version of this pipeline is free and needs no account.

What you end up with

One SQLite file, one table of daily bars, and one script of about 80 lines that you can run every day. It only downloads the days it is missing, it never creates duplicate rows, and changing the data source means changing one function. That last part matters: free sources break, and you do not want that to break your backtests too.

The code is on GitHub: tomyin1314/daily-stock-data-pipeline.

You need Python 3.10 or later and three packages:

pip install yfinance pandas requests

The whole script

Save this as pipeline.py. The sections below explain the parts that are easy to get wrong.

"""Daily price pipeline: fetch daily bars and store them in SQLite.

Safe to run as often as you like: it only asks for the days it is missing,
and the primary key stops duplicate rows.
"""
import sqlite3
import sys

import pandas as pd
import requests
import yfinance as yf

DB = "prices.db"
SYMBOLS = ["AAPL", "MSFT"]
START = "2024-01-01"  # how far back to go on the first run
COLUMNS = ["symbol", "date", "open", "high", "low", "close", "adj_close", "volume", "source"]


def fetch_yfinance(symbol: str, start: str) -> pd.DataFrame:
    """Daily bars from Yahoo Finance via yfinance."""
    raw = yf.download(symbol, start=start, auto_adjust=False, progress=False,
                      multi_level_index=False)
    if raw.empty:
        return pd.DataFrame(columns=COLUMNS)
    df = raw.reset_index().rename(columns={
        "Date": "date", "Open": "open", "High": "high", "Low": "low",
        "Close": "close", "Adj Close": "adj_close", "Volume": "volume",
    })
    df["date"] = df["date"].dt.strftime("%Y-%m-%d")
    df["symbol"], df["source"] = symbol, "yfinance"
    return df[COLUMNS]


def fetch_eodhd(symbol: str, start: str, token: str = "demo") -> pd.DataFrame:
    """Daily bars from EODHD APIs. The "demo" token works for AAPL only."""
    resp = requests.get(
        f"https://eodhd.com/api/eod/{symbol}.US",  # US tickers need the .US suffix
        params={"api_token": token, "fmt": "json", "from": start},
        timeout=30,
    )
    resp.raise_for_status()
    df = pd.DataFrame(resp.json())
    if df.empty:
        return pd.DataFrame(columns=COLUMNS)
    df = df.rename(columns={"adjusted_close": "adj_close"})
    df["symbol"], df["source"] = symbol, "eodhd"
    return df[COLUMNS]


def init_db(conn: sqlite3.Connection) -> None:
    conn.execute("""
        CREATE TABLE IF NOT EXISTS daily (
            symbol TEXT NOT NULL, date TEXT NOT NULL,
            open REAL, high REAL, low REAL, close REAL, adj_close REAL,
            volume INTEGER, source TEXT,
            PRIMARY KEY (symbol, date)
        )""")


def last_date(conn: sqlite3.Connection, symbol: str) -> str | None:
    return conn.execute("SELECT MAX(date) FROM daily WHERE symbol = ?", (symbol,)).fetchone()[0]


def update(conn: sqlite3.Connection, symbol: str, fetch) -> int:
    # Start again from the last stored day: if the job ran before the close,
    # that bar was incomplete, and INSERT OR REPLACE overwrites it.
    start = last_date(conn, symbol) or START
    df = fetch(symbol, start)
    conn.executemany(
        "INSERT OR REPLACE INTO daily VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
        df.itertuples(index=False, name=None),
    )
    conn.commit()
    return len(df)


if __name__ == "__main__":
    fetch = fetch_eodhd if "--eodhd" in sys.argv else fetch_yfinance
    symbols = ["AAPL"] if fetch is fetch_eodhd else SYMBOLS
    with sqlite3.connect(DB) as conn:
        init_db(conn)
        for symbol in symbols:
            n = update(conn, symbol, fetch)
            total = conn.execute("SELECT COUNT(*) FROM daily WHERE symbol = ?", (symbol,)).fetchone()[0]
            print(f"{symbol}: fetched {n} rows, {total} in the database, latest {last_date(conn, symbol)}")

Run it twice

The first run downloads everything since the start date:

$ python pipeline.py
AAPL: fetched 696 rows, 696 in the database, latest 2026-10-09
MSFT: fetched 696 rows, 696 in the database, latest 2026-10-09

Run it again and it asks only for the last stored day onwards. The row count does not change, because the primary key on (symbol, date) plus INSERT OR REPLACE turns a repeat download into an overwrite:

$ python pipeline.py
AAPL: fetched 1 rows, 696 in the database, latest 2026-10-09
MSFT: fetched 1 rows, 696 in the database, latest 2026-10-09

Re-fetching the last day is deliberate. If the job ever runs before the market closes, the bar it stores is incomplete, and the next run quietly replaces it with the final one.

(Output from 12 October 2026, yfinance 1.7.0 and pandas 2.3.3.)

Two sources, one shape

Both fetch_yfinance and fetch_eodhd return the same nine columns in the same order. Everything after the fetch step, the table, the incremental logic and whatever reads the database later, has no idea which source was used.

You can see that by pointing the same script at the other source. EODHD APIs publishes a demo token that works for AAPL without an account, so this runs as written:

$ python pipeline.py --eodhd
AAPL: fetched 1 rows, 696 in the database, latest 2026-10-09

symbol       date      close  adj_close   volume   source
  AAPL 2026-10-09 336.640000 336.640000 37855000    eodhd
  AAPL 2026-10-08 340.420013 340.420013 35332400 yfinance
  AAPL 2026-10-07 336.670013 336.670013 34147900 yfinance

The newest row now comes from the second source, and the close and volume match what yfinance had stored for the same day.

For anything beyond the demo ticker you need your own token. The free plan allows 20 API calls a day, which covers a small watchlist once a day, but it only returns the past year of history: with START = "2024-01-01" a free key gives you about one year of bars, not the full range. Paid plans lift both limits. See the EODHD APIs plans for current limits.

Close and adjusted close are different numbers

close is the price the stock actually traded at that day. adj_close is rewritten after splits and dividends so that returns come out right. Use adj_close to compute returns in a backtest, and close when you care about the price level that was on the screen.

Two details to know. With yfinance you must pass auto_adjust=False, as the script does, or the adjusted price replaces the close and you lose the raw one. And adjusted history changes over time: after a new dividend, older adjusted prices are restated. The incremental update above does not go back and refresh them, so plan a full reload now and then, by deleting the file and running the script again.

You may also notice values such as 340.420013 from yfinance. That is floating-point noise from the source, not extra precision. Round when you display, not when you store.

Schedule it

US markets close at 16:00 New York time. Give the data a few hours to settle and run the job once a day after that. On Linux or macOS, a cron entry for 22:30 UTC on weekdays:

30 22 * * 1-5 cd /path/to/project && python pipeline.py >> pipeline.log 2>&1

On Windows, the same thing with Task Scheduler:

schtasks /Create /SC WEEKLY /D MON,TUE,WED,THU,FRI /ST 22:30 /TN PricePipeline /TR "python C:/path/to/pipeline.py"

Task Scheduler uses your local time, so adjust the hour. Running on a market holiday is harmless: the source returns nothing new and the script stores nothing new.

Where it breaks

The free source stops answering. yfinance reads Yahoo Finance's public endpoints and can fail without notice. The script prints the latest stored date for every symbol; if that date stops moving, you know the same day.
Ticker suffixes. EODHD APIs wants AAPL.US, not AAPL, which is why the fetch function adds the suffix for you.
Rate limits. One request per symbol per day is modest, but a first run over hundreds of symbols is not. Add a short pause between symbols before you scale up.
Terms of use. Yahoo's data is for personal use. If the pipeline feeds anything commercial, use a source whose terms allow it. That applies to paid sources too: EODHD APIs' standard plans are for personal use, and it sells separate Commercial plans.

Next step

If you are deciding which source to build on, I compared four of them here: yfinance vs AKShare vs OpenBB vs EODHD APIs.

Nothing in this post is investment advice. Prices from any source can be wrong or late; check them before you rely on them.

DEV Community: jun yin