SDET by day. On the side I build tools with AI, including qa-audit.com, which runs automated SEO, accessibility, link, and performance checks on side-project launches, with a free preview scan.
You asked people to break it, so here's a small one. I tried a made-up path on the beta and it still rendered as a normal success page instead of a clear not-found. Soft 404s like that make bad links look "fine" to users and let junk URLs stick in search. The site scored 94/100 (A) in a pass that also surfaced 8 other critical/warning issues. I'm a Staff SDET, so happy to dig into more of the four-task flow if that's useful.
Thank you for sharing this feedback, This is a high value catch that offers bad UX and harm SEO health.
The Strategy i thought of to resolve this issue for the frontend and backend is to implement a dedicated responsive component with clear HTTP 404 semantics and a direct CTA back to the dashboard while for the backend, I will ensure that any unknow api routes and static asset routes return an explicit HTTP status with a standardized JSON error payload...
Yes, that’s very useful. Please dig into the remaining four-task flow and let me know the eight other critical issues.
SDET by day. On the side I build tools with AI, including qa-audit.com, which runs automated SEO, accessibility, link, and performance checks on side-project launches, with a free preview scan.
Glad it helped, Victor. Your plan sounds right. One thing to watch: on Vercel a not-found component can still come back as a 200 if the app shell catches the route, so after you ship it, hit a made-up path with curl -I and make sure the status line really says 404.
Small correction on my end: the soft 404 was the only critical one, the other 8 are warnings. The two I'd do next are adding an H1 heading to the home page, since search engines lean on it to understand what the page is about, and putting one clear "call to action" (button or text) above the fold so a new visitor knows the first step.
Thanks for the heads up about how Vercel catches route! Testing a random-path with curl -I is a smart safety check, and I'll add that to my deployment phase.
Thanks for clarifying the warnings. I am already thinking and going through the codebase to try and locate part of the critical issues. Adding the H1 to the homepage and optimizing the above-the-fold CTA make perfect sense. What are the other 6 warnings on the list? I'd love to get them on my radar so I can clean them up next. Is there anything you can say about the UI and UX of the web app ? Thanks Valencia
SDET by day. On the side I build tools with AI, including qa-audit.com, which runs automated SEO, accessibility, link, and performance checks on side-project launches, with a free preview scan.
Nice, putting the curl -I check into your deploy step means the 404 can't quietly come back.
Of the six left, two are worth doing:
I couldn't find a privacy policy linked anywhere on the site. When people are typing in their income and spending, a short privacy page linked from the footer goes a long way for trust.
The llms.txt one is really part of the 404 issue. There's no llms.txt file yet, so that address hands back your normal app page, and the scan read it as a file with no title. If you want one, add a real llms.txt with a first line like # Vicquant. If not, don't worry about it.
The rest you can mostly skip: three are email DNS records (SPF, DMARC, MX) that don't apply while you're on a vercel.app address, and one is a phone-number check meant for local businesses.
The full report has every issue with a fix, plus a re-scan to confirm once you've fixed things.
On UI and UX, the scan doesn't judge that, so I won't guess. The cheapest test I know: sit a friend who's never seen it in front of the four tasks, say nothing, and watch where they pause. That usually shows you more than any checklist.
I build Canli Capital: free, open-source tools that check whether a backtest is real (deflated Sharpe, overfitting and data-snooping tests), plus SEC fundamentals for AI agents over MCP.
The "no win rate until 30 resolved signals" rule is a good instinct, but 30 is still a thin sample, so I'd show the interval next to the number. I computed Wilson 95% intervals for n=30: 18 wins (60%) gives 42%-75%, and 20 wins (67%) gives 49%-81%. The chance of 20 or more wins out of 30 from a coin-flip strategy is 4.9%, and 21 or more is 2.1%. So a tester can see a 67% "win rate" at the first moment it is displayed and it still can't rule out luck once you have tried more than a few rules. To pin the half-width at about 5 points around 55% you'd need roughly 380 resolved signals.
Two things that would make the proof ledger stronger: record the rule version in each hash-chained entry (so changing a threshold later is visible), and show the count of rules or parameter sets tried before this one went live, since that count is what inflates the best one.
Also worth saying on the Sharpe/Sortino side of the app: with a few months of daily returns those ratios have wide error bars as well, so a "no finance degree" display should probably carry the same caveat. I built canli-mcp (free, MIT, no key), npx -y canli-mcp, and github.com/arhancanli/canlicapital has the free overfitting checks if you want something to test the ledger against.
This is top-tier feedback, thank you for taking the time to run the exact numbers. You’re completely right across all three fronts:
For the Wilson 95% Intervals & Coin-Flip Luck:
You nailed it on the thin sample illusion. Showing a standalone "67% win rate" at $n=30$ without the $[49\%, 81\%]$ interval creates a false sense of certainty, especially when a fair coin gives 20+ wins nearly 5% of the time. Because of that, I’ve updated the headline display so that the win rate never appears without its Wilson 95% interval right beside it (e.g., 66.7% [48.8%–80.8%] (95% CI)), along with the coin-flip luck probability and an explicit sample size indicator showing that ~380 resolved signals are needed to pin the margin of error down to $\pm 5\%$.
For the Rule Versioning & Parameter Sets Tried in the Ledger:
Adding rule_version directly into the canonical payload and SHA-256 hash chain is brilliant, it prevents silent threshold creeping after the fact. More importantly, recording the count of rules sets tried ($M$) before going live tackles selection bias head-on. If you try 15 – 20 parameter variations, the multiple-testing false positive rate balloons even if a single trial looks nominally significant.
For the Sharpe/Sortino Error Bars & "No Finance Degree" Caveats:
Spot-on. A 3-month annualized Sharpe of 1.50 has an enormous standard error (~2.0 under Lo's asymptotic formulation), meaning the true 95% interval easily spans from negative territory to over 3.0+. I also added the standard error / confidence intervals alongside the ratios, plus a clear, plain-language caveat that short-term risk metrics are preliminary noise until at least 2–3 years (500+ trading days) of history accumulate.
I Really appreciate the pointer to canli-mcp. I will definitely test the ledger and overfitting guards against your framework
You asked people to break it, so here's a small one. I tried a made-up path on the beta and it still rendered as a normal success page instead of a clear not-found. Soft 404s like that make bad links look "fine" to users and let junk URLs stick in search. The site scored 94/100 (A) in a pass that also surfaced 8 other critical/warning issues. I'm a Staff SDET, so happy to dig into more of the four-task flow if that's useful.
Thank you for sharing this feedback, This is a high value catch that offers bad UX and harm SEO health.
The Strategy i thought of to resolve this issue for the frontend and backend is to implement a dedicated responsive component with clear HTTP 404 semantics and a direct CTA back to the dashboard while for the backend, I will ensure that any unknow api routes and static asset routes return an explicit HTTP status with a standardized JSON error payload...
Yes, that’s very useful. Please dig into the remaining four-task flow and let me know the eight other critical issues.
Thank you very much.
Glad it helped, Victor. Your plan sounds right. One thing to watch: on Vercel a not-found component can still come back as a 200 if the app shell catches the route, so after you ship it, hit a made-up path with curl -I and make sure the status line really says 404.
Small correction on my end: the soft 404 was the only critical one, the other 8 are warnings. The two I'd do next are adding an H1 heading to the home page, since search engines lean on it to understand what the page is about, and putting one clear "call to action" (button or text) above the fold so a new visitor knows the first step.
Thanks for the heads up about how Vercel catches route! Testing a random-path with curl -I is a smart safety check, and I'll add that to my deployment phase.
Thanks for clarifying the warnings. I am already thinking and going through the codebase to try and locate part of the critical issues. Adding the H1 to the homepage and optimizing the above-the-fold CTA make perfect sense. What are the other 6 warnings on the list? I'd love to get them on my radar so I can clean them up next. Is there anything you can say about the UI and UX of the web app ? Thanks Valencia
Nice, putting the curl -I check into your deploy step means the 404 can't quietly come back.
Of the six left, two are worth doing:
# Vicquant. If not, don't worry about it.The rest you can mostly skip: three are email DNS records (SPF, DMARC, MX) that don't apply while you're on a vercel.app address, and one is a phone-number check meant for local businesses.
Here's the page with your score and problem areas: qa-audit.com/teaser/759e205d-7b82-...
The full report has every issue with a fix, plus a re-scan to confirm once you've fixed things.
On UI and UX, the scan doesn't judge that, so I won't guess. The cheapest test I know: sit a friend who's never seen it in front of the four tasks, say nothing, and watch where they pause. That usually shows you more than any checklist.
Thanks. You have been helpful
The "no win rate until 30 resolved signals" rule is a good instinct, but 30 is still a thin sample, so I'd show the interval next to the number. I computed Wilson 95% intervals for n=30: 18 wins (60%) gives 42%-75%, and 20 wins (67%) gives 49%-81%. The chance of 20 or more wins out of 30 from a coin-flip strategy is 4.9%, and 21 or more is 2.1%. So a tester can see a 67% "win rate" at the first moment it is displayed and it still can't rule out luck once you have tried more than a few rules. To pin the half-width at about 5 points around 55% you'd need roughly 380 resolved signals.
Two things that would make the proof ledger stronger: record the rule version in each hash-chained entry (so changing a threshold later is visible), and show the count of rules or parameter sets tried before this one went live, since that count is what inflates the best one.
Also worth saying on the Sharpe/Sortino side of the app: with a few months of daily returns those ratios have wide error bars as well, so a "no finance degree" display should probably carry the same caveat. I built canli-mcp (free, MIT, no key),
npx -y canli-mcp, and github.com/arhancanli/canlicapital has the free overfitting checks if you want something to test the ledger against.This is top-tier feedback, thank you for taking the time to run the exact numbers. You’re completely right across all three fronts:
For the Wilson 95% Intervals & Coin-Flip Luck:
You nailed it on the thin sample illusion. Showing a standalone "67% win rate" at $n=30$ without the $[49\%, 81\%]$ interval creates a false sense of certainty, especially when a fair coin gives 20+ wins nearly 5% of the time. Because of that, I’ve updated the headline display so that the win rate never appears without its Wilson 95% interval right beside it (e.g.,
66.7% [48.8%–80.8%] (95% CI)), along with the coin-flip luck probability and an explicit sample size indicator showing that ~380 resolved signals are needed to pin the margin of error down to $\pm 5\%$.For the Rule Versioning & Parameter Sets Tried in the Ledger:
Adding
rule_versiondirectly into the canonical payload and SHA-256 hash chain is brilliant, it prevents silent threshold creeping after the fact. More importantly, recording the count of rules sets tried ($M$) before going live tackles selection bias head-on. If you try 15 – 20 parameter variations, the multiple-testing false positive rate balloons even if a single trial looks nominally significant.For the Sharpe/Sortino Error Bars & "No Finance Degree" Caveats:
Spot-on. A 3-month annualized Sharpe of 1.50 has an enormous standard error (~2.0 under Lo's asymptotic formulation), meaning the true 95% interval easily spans from negative territory to over 3.0+. I also added the standard error / confidence intervals alongside the ratios, plus a clear, plain-language caveat that short-term risk metrics are preliminary noise until at least 2–3 years (500+ trading days) of history accumulate.
I Really appreciate the pointer to
canli-mcp. I will definitely test the ledger and overfitting guards against your frameworkbudget-planner-advisor-rho.vercel....
Thanks Veysal. I will fix this issue and do you see any other issue or bug. I really appreciate
Light mode, is not ok:
Thank you for going through the frontend. I will work on it and were you able to understand the whole app