276 days ago, I started building EventHorizon.
I wanted to create software that could solve real problems using machine learning. I started with more ambition than experience, and over time, I learned that building a model is only one part of building a useful product.
Since then, I've experimented with demand forecasting, cryptocurrency prediction, validation tools, and customer outreach.
Here's what I learned—including what didn't work.
1. Building the Models Was Only the Beginning
One of my main projects became EventHorizon-AI, a demand forecasting platform designed for small businesses.
The idea was straightforward: help businesses make better decisions about how much stock to prepare or order by predicting future demand.
I also experimented with short-term BTC/USDT price direction prediction.
These projects had different goals, but they taught me the same lesson: a model that produces predictions isn't automatically useful.
You need to understand what the numbers actually tell you, how the model was evaluated, and whether its predictions can create value in the real world.
2. Demand Forecasting: A Measurable Improvement
For demand forecasting, I experimented with the M5 Walmart dataset.
My model used features such as historical sales, lagged values, seasonality, prices, and promotions.
The validation results were:
| Model | WAPE |
|---|---|
| Baseline | 44.4% |
| EventHorizon model | 31.03% |
That's approximately a 30% reduction in WAPE relative to the baseline.
WAPE, or Weighted Absolute Percentage Error, measures total absolute prediction error relative to total actual demand.
This was an encouraging result. In my evaluation, the improvement appeared across all seven stores.
However, a benchmark result is not the same as a successful deployment in a real business.
The M5 dataset provides a useful environment for experimentation, but it doesn't prove that the same performance will transfer to a particular bakery, florist, or other small business. Real-world validation is a separate challenge.
3. Cryptocurrency Prediction: Better Accuracy Wasn't Enough
I also built a short-term BTC/USDT directional prediction system.
The experiment used a Transformer-based architecture combined with a BiLSTM and cross-attention. It was trained on approximately 1.5 million samples, with predictions across multiple time horizons.
In my evaluation, the model achieved around 60% directional accuracy, compared with a baseline of approximately 52%.
At first glance, that looked promising. But there was a problem:
Statistical performance isn't the same as economic profitability.
Once I considered transaction fees and execution-related costs, the apparent edge wasn't economically viable under the conditions I tested.
That changed how I thought about machine learning projects.
A model can outperform a baseline and still fail at the task that actually matters. The evaluation objective must reflect the intended use of the system.
For a trading model, directional accuracy alone isn't enough. For a demand forecasting model, lower prediction error alone doesn't guarantee better business outcomes.
The metric must connect to the real-world objective.
4. Validation Deserves Its Own Attention
As I worked on these projects, I became increasingly interested in how to determine whether an apparent improvement was actually meaningful.
I developed an honest-validation toolkit with utilities for:
- Walk-forward validation splits.
- Block bootstrap and gap bootstrap procedures.
- Permutation testing.
- WAPE and MASE evaluation metrics.
These tools helped me approach model evaluation more systematically.
I learned to pay attention to temporal structure, possible information leakage, baseline selection, and the difference between an observed improvement and evidence that a model will generalize.
A good metric alone isn't sufficient evidence. The evaluation process matters just as much as the result.
5. Building Something Useful Requires Talking to People
Technical experiments were only one side of the project.
I also started reaching out to small businesses, including bakeries, florists, and other potential customers. I wanted to understand how they made decisions about demand, inventory, and daily production.
I learned that the challenge isn't simply building a prediction model. Businesses already have their own methods, experience, constraints, and sources of information.
A technically interesting solution isn't automatically one that someone will adopt.
My outreach hasn't produced a paying customer or a real pilot so far. That's an important part of the story, too.
I still need to validate whether the platform can provide enough practical value for a business to use it. Better model metrics don't answer that question by themselves.
6. What I Would Do Differently
Looking back, I'd put even more emphasis on a few things from the beginning:
- Define the baseline first. A model's performance means little without a meaningful comparison.
- Design validation around the data. Time-dependent data requires particular care with splits, leakage, and evaluation methodology.
- Separate technical progress from product progress. A better model doesn't necessarily mean a better product.
- Document limitations honestly. Failed experiments and negative results can be as informative as successful ones.
- Talk to potential users earlier. It's possible to spend months solving a technically interesting problem without establishing that the proposed solution fits how people actually work.
These lessons are changing how I approach my next experiments.
7. What Comes Next
I'm continuing to work on EventHorizon with a stronger focus on reliable evaluation, practical usefulness, and understanding where forecasting can genuinely help small businesses.
I want to build systems that do more than produce impressive-looking numbers. I want to understand why they work, where they fail, and under which conditions their results can be trusted.
There's still a lot to learn, both technically and commercially. And that's okay.
Final Thoughts
I'm 14, and I spent 276 days building and experimenting with an AI platform.
I haven't built a successful business yet. I haven't demonstrated that every model I've developed is useful in production. Some experiments showed promising results; others taught me why an apparently good result isn't enough.
But I now have a much better understanding of model evaluation, time-series forecasting, statistical validation, and the gap between a working experiment and a useful product.
For me, the biggest lesson has been this:
Building a model is an engineering task. Determining whether you can trust it—and whether it solves a real problem—is a much bigger challenge.
I'm continuing to learn, experiment, and document what I find.
If you're working on forecasting, machine learning validation, or independent software projects, I'd be interested to hear what you've learned from your own experiments.
Explore EventHorizon
If you'd like to explore the project or follow my work:
I'm continuing to experiment with demand forecasting, model validation, and practical applications of machine learning. Feedback and technical discussions are welcome
Top comments (0)