DEV Community: Stock Expert AI

Building a Daily Data Pipeline That Survives Market Holidays

Stock Expert AI — Sun, 24 May 2026 23:15:37 +0000

So, I had to build this data pipeline that grabs financial market data. Daily. Sounds simple, right?

Not so fast. Market holidays threw a wrench in the works. The data source wouldn't update, and my pipeline couldn't just choke on those days. Pipeline failures are annoying.

I needed the pipeline to either run smoothly, processing the data available (even if it was the same as yesterday), or skip the run entirely — no errors, no alerts.

Here's the simple stupid strategy I landed on:

Get a holiday calendar.
Check the date at the start.
Run or exit based on that check.
Log the decision.

Let's look at some Python code.

First, the holiday calendar. Hardcoding? No thanks. I'm not into maintenance nightmares. Instead, I generate it with code:

from datetime import date
# Assume 'market_holidays' is a function to generate a list of holidays
# based on the current year.
holiday_list = market_holidays(date.today().year)

Now, before the actual processing, check if today's a holiday:

today = date.today()
if today in holiday_list:
    print("Today is a market holiday. Skipping data processing.")
    # Optionally, log this decision to a file or database.
    # Here's a basic logging example:
    with open("pipeline_log.txt", "a") as log_file:
        log_file.write(f"{today}: Skipped due to holiday\n")
    exit()  # Gracefully exit the script

If it's not a holiday, the script just keeps going.

But what about the next business day? How do we ensure the pipeline runs then and processes the data correctly? Initially, I let the scheduler handle it. It worked, assuming the scheduler retried on the next available day.

But sometimes the data source only provided data for trading days. That's when I added a "lookback" mechanism. If today's a holiday, check the previous business day.

This meant tweaking the date checking:

from datetime import date, timedelta

today = date.today()
data_date = today  # Default to today

if today in holiday_list:
    print("Today is a market holiday. Checking previous business day.")
    data_date = today - timedelta(days=1)
    # Keep subtracting days until we find a non-holiday
    while data_date in holiday_list:
        data_date -= timedelta(days=1)

    print(f"Using data from {data_date}")
    # Log this decision
    with open("pipeline_log.txt", "a") as log_file:
        log_file.write(f"{today}: Using data from {data_date} (holiday fallback)\n")

# Now, fetch data using the 'data_date' variable
# data = fetch_market_data(data_date)

This ensures the pipeline grabs data from the most recent trading day, even on holidays. This is obvious.

A few more things to remember.

Error handling is vital. What if the data source is down for other reasons? Add retry logic and logging, so you can diagnose issues quickly.

And monitoring? Gotta track performance and spot anomalies. If the pipeline keeps skipping runs, that's a problem.

Test everything. Simulate holidays to see if it behaves as expected.

Building a robust data pipeline that can handle market holidays takes thought and planning. A holiday calendar, conditional execution, and a lookback mechanism? This is the way.

Now I can sleep soundly, knowing my pipeline won't bother me on Christmas.

Building a Daily Data Pipeline That Survives Market Holidays

Stock Expert AI — Sun, 24 May 2026 21:52:58 +0000

I recently put together a daily data pipeline to pull and process financial data. Simple stupid, right? Grab data, clean it, load it.

Of course, the devil's in the details. And one of the biggest headaches? Market holidays.

The pipeline needed to run daily, grabbing data from the prior day's trading. Weekdays are easy. Weekends? Obvious. But what about holidays when the markets are closed? If the pipeline blindly tries to grab data for a non-trading day, it errors out. The whole thing grinds to a halt. Nobody wants that.

Here's how I tackled this, focusing on making the pipeline solid and avoiding those failures.

First, I needed a way to determine if a given date was a market holiday. Hardcoding a list? Brittle. Constant updating? No thanks. Instead, I decided to calculate market holidays dynamically.

There are ways to do this, depending on the markets you care about. I wrote a function that checks if a given date is a business day. If it isn't, it flags it as a holiday.

The core of this function? Checking the day of the week (weekends are obviously holidays) and then iterating through a list of known holidays for the current year. This list is built based on the country and market.

Here's a simplified Python snippet:

import datetime

def is_market_holiday(date):
 """
 Checks if a given date is a market holiday.
 """
 # Check if it's a weekend
 if date.weekday() >= 5: # 5 and 6 are Saturday and Sunday
 return True

 # Placeholder for a more sophisticated holiday calculation.
 # In a real implementation, you'd generate a list of holidays
 # for the given year based on the specific market.
 # This example just checks for a hardcoded Independence Day.
 if date.month == 7 and date.day == 4:
 return True

 return False

# Example usage:
today = datetime.date.today()
if is_market_holiday(today):
 print(f"{today} is a market holiday.")
else:
 print(f"{today} is a trading day.")

This is basic. Expand it to include your actual holiday calculation logic. The key is to wrap this logic in a function that's easily called from the pipeline.

Next, I dropped this holiday check into the pipeline's scheduling logic. Instead of running every day no matter what, the pipeline now checks if the target date is a trading day. If it's a holiday, the pipeline skips the data-grabbing step. It either waits until the next scheduled run or backfills the data for the previous trading day.

This "backfilling" strategy is important. If the pipeline skips a holiday, you don't want to lose that day's data. Instead, you grab the data for the last actual trading day.

Here's how this might look in a pipeline orchestration tool:

daily_pipeline:
 schedule: "0 0 * * *" # Run every day at midnight
 steps:
 - name: Check if today is a market holiday
 task: is_market_holiday
 date: "{{ execution_date }}"
 output: holiday_flag

 - name: Fetch data
 task: fetch_market_data
 date: "{{ execution_date - 1 }}" # Fetch data for *yesterday*
 when: "{{ holiday_flag == False }}" # Only run if not a holiday

 - name: Process data
 task: process_data
 input_data: "{{ fetch_market_data.output_data }}"

 - name: Load data
 task: load_data
 input_data: "{{ process_data.output_data }}"

execution_date is the date the pipeline is running for. The is_market_holiday task decides if that date is a holiday. If it's not, the fetch_market_data task grabs data for the previous day (which should be a trading day).

This isn't perfect. It assumes that the market is always open the day before a holiday. Not always true. You might need to tweak the logic to handle multiple consecutive holidays.

Finally, I set up monitoring and alerting. The pipeline sends notifications if it hits an unexpected error or if the data quality checks fail. This lets me quickly spot and fix issues, including those related to holiday handling.

Building a solid data pipeline? It means thinking about edge cases like market holidays. By handling these proactively, you make sure your pipeline runs reliably. And it provides good data, even when the markets are closed.

Sentiment Scoring for Finance News: A Practical Intro

Stock Expert AI — Thu, 21 May 2026 13:00:45 +0000

So you want to play around with sentiment analysis of finance news? Awesome. It's an interesting field, and frankly, it's not as scary as it sounds. I'll walk you through a basic but working example to get you started. This isn't an academic exercise; we're getting our hands dirty.

Why bother? News articles can move markets. Quantify the tone of news—is it positive, negative, or neutral about a company or asset?—and you might find predictive power. Note that "might." There are zero guarantees in this game.

Let's begin with the basics: text. Let's say we have this headline:

headline = "Company X announces record profits, stock price surges"

The simplest method is a pre-trained sentiment lexicon. Think of it as a dictionary with pre-assigned sentiment scores for words. A common one is VADER (Valence Aware Dictionary and sEntiment Reasoner). I can't tell you where to find it, but they exist.

Imagine the lexicon looks like this (totally fabricated, BTW):

lexicon = {
 "record": 0.7,
 "profits": 0.8,
 "surges": 0.9,
 "announces": 0.2, # Could be positive or neutral
 # ... lots more words ...
}

Scores show how positive or negative the word is, perhaps on a scale from -1 to +1.

Here's some code:

def score_headline(headline, lexicon):
 words = headline.lower().split() # Lowercase and split into words
 total_score = 0
 for word in words:
 if word in lexicon:
 total_score += lexicon[word]
 return total_score

headline = "Company X announces record profits, stock price surges"
sentiment_score = score_headline(headline, lexicon)
print(f"Sentiment score: {sentiment_score}")

This is dead simple. It sums the scores of headline words found in the lexicon. This yields a number. A positive number means positive sentiment, and the reverse is also true.

Why is this too simple?

Tons of reasons. It doesn't handle negation. "Not good" should be negative, but this code treats "good" as positive and ignores the "not." It also ignores context. "Bankrupt" is bad, but "avoided bankruptcy" is different. It certainly doesn't handle sarcasm; good luck with that one. And finally, it treats all words equally, which is not ideal.

Simple improvements:

Negation Handling: Look for "not," "never," or "no" before a sentiment word. If you find one, flip the sentiment score's sign.

def score_headline_with_negation(headline, lexicon):
 words = headline.lower().split()
 total_score = 0
 negate = False # Track negation
 for word in words:
 if word in ["not", "never", "no"]:
 negate = True
 elif word in lexicon:
 score = lexicon[word]
 if negate:
 score *= -1
 negate = False # Reset negation
 total_score += score
 return total_score

N-grams: Consider pairs or triplets of words (bigrams, trigrams). The phrase "not good" is a bigram. You'd need to grow your lexicon to include these phrases. This gets complicated fast, I promise.
Weighting: Give different words different weights. Maybe "record" and "surges" matter more than "announces." Adjust the lexicon scores or use multipliers in your code to do this.

Real-World Stuff

Lexicons are almost never perfect, and you'll probably need to customize yours. This means manually checking headlines and changing scores, or adding terms. Tedious, but needed.

The subject area matters. Finance news sentiment differs from movie review sentiment. A general lexicon won't cut it.

Don't just use headlines. Article bodies have more info, but processing them costs a lot more.

Sentiment analysis is just one piece. Don't expect to get rich scoring headlines. It's a weak signal.

Beyond Lexicons

Lexicon methods are simple, but limited. Advanced methods use machine learning models trained on big text datasets. These models learn more complex patterns. But they need lots of data and compute power.

Sentiment scoring is fun and hard. Start simple, iterate, and don't expect miracles.

Storing and Querying Time-Series Price Data Efficiently

Stock Expert AI — Wed, 20 May 2026 18:00:59 +0000

Okay, let's get real about time-series data, especially the kind that represents prices. I've butted heads with this problem enough times to have a few tricks up my sleeve. These can seriously save you from wanting to throw your computer out the window later. I'm talking about how to store and query data like stock prices, sensor readings, or anything else that morphs over time without wanting to pull your hair out.

The first thing most people try is just dumping everything into a relational database. Sure, it works. But it's like trying to run a marathon in flip-flops; you'll quickly hit a wall. Imagine logging every single price change of a stock, every second, for years. You're staring down billions of rows, and even basic queries will take an eternity.

So, what's the better play? It really boils down to what you need, but here's what I've found actually works, along with some code to show you what I mean.

First, get to know your data. Is it high-frequency data, like every tick of a stock? Or is it lower frequency, such as daily closing prices? This one question will shape how you store and query the data. What kind of questions will you be asking? Are you figuring out moving averages? Spotting trends? Something else?

For high-frequency data, compression is your new best friend. Storing raw, uncompressed data? That's just burning money. Delta encoding is a neat trick. Instead of saving the actual price, you save the difference from the price before. If the prices are close, these deltas will be small, so you can store them using fewer bits.

Here's a little Python to show you what I'm talking about:

def delta_encode(prices):
 deltas = []
 last_price = prices[0]
 deltas.append(prices[0]) # Store the first price
 for price in prices[1:]:
 delta = price - last_price
 deltas.append(delta)
 last_price = price
 return deltas

def delta_decode(deltas):
 prices = []
 last_price = deltas[0]
 prices.append(deltas[0])
 for delta in deltas[1:]:
 price = last_price + delta
 prices.append(price)
 last_price = price
 return prices

prices = [10.0, 10.1, 10.2, 10.1, 10.3, 10.5]
deltas = delta_encode(prices)
print(f"Original prices: {prices}")
print(f"Delta encoded: {deltas}")
print(f"Decoded prices: {delta_decode(deltas)}")

That's a basic example, but you get the gist. You can compress these deltas even further with variable-length encoding or fancier algorithms.

Data aggregation is another big win. If you're swimming in high-frequency data, you probably don't need to hoard every single tick forever. Aggregate the data into bigger chunks of time – say, 1-minute bars, hourly bars, or daily bars. This slashes the amount of data you're storing, and it makes common queries way faster.

Check out this Python snippet for a basic aggregation:

import pandas as pd

def aggregate_data(data, frequency='1H'): # Default to hourly
 df = pd.DataFrame(data)
 df['timestamp'] = pd.to_datetime(df['timestamp'])
 df = df.set_index('timestamp')
 aggregated = df['price'].resample(frequency).agg(['first', 'last', 'min', 'max', 'mean'])
 return aggregated

# Example usage (assuming 'data' is a list of dictionaries with 'timestamp' and 'price')
# data = [{'timestamp': '2024-01-01 00:00:00', 'price': 10.0}, ...]
# aggregated_data = aggregate_data(data)
# print(aggregated_data)

pandas makes this kind of thing a breeze. You can chop up your data into different timeframes and calculate all sorts of stats: open, close, high, low, average, and whatever else you need.

And don't sleep on indexing. A standard index on the timestamp column is a start, but you might need more firepower depending on what you're asking of your data. If you're always hunting for data within a certain price range, think about a spatial index or something even more specialized.

Beyond all this, your storage engine matters, a lot. Relational databases are okay for smaller datasets, but purpose-built time-series databases are often the way to go when you're dealing with serious scale. They're built for time-based queries and often have compression and aggregation baked right in.

To recap: Know your data. Compress it. Aggregate it smartly. Pick the right storage engine and index it well. Do these things, and you might actually enjoy working with time-series data.

MoonshotScore: Compressing Stock Analysis Into One AI Number (0–100)

Stock Expert AI — Sun, 17 May 2026 13:24:55 +0000

If you've ever researched a stock, you know the problem: you open ten tabs — financials, charts, news, analyst takes — and an hour later you're more confused than when you started.

We built MoonshotScore to fix exactly that. It's a single number from 0 to 100 that answers one question: how much upside potential does this stock have right now?

What goes into the score

MoonshotScore isn't one metric dressed up — it's a blend of signals our models weigh together:

Fundamentals — revenue growth, margins, debt load, free cash flow
Technical structure — trend, momentum, support and resistance
Sentiment — news flow and how the market is positioned
Valuation — is the price ahead of the story, or behind it?

Each stock is re-scored as new data arrives, so the number reflects now — not last quarter.

The Legends Council

Numbers alone miss nuance, so we layered on perspective. The Legends Council runs each stock through the lens of investing legends — Buffett's quality focus, Dalio's macro caution, Lynch's growth-at-a-reasonable-price, Simons' quant rigor. You see where they'd agree, and where they'd argue.

Why a single number?

Because decisions need an anchor. MoonshotScore doesn't tell you to buy — it tells you where to look first. A 78 means "dig in." A 31 means "probably skip it." It turns a two-hour research session into a two-minute filter.

We cover 35,000+ stocks, and it's free to use.

👉 Try it: stockexpertai.com

If you were designing a score like this, what would you want it to factor in? Curious to hear takes in the comments.