Last month I answered the same question for the fourth time. A friend from my statistics program sent me a screenshot of their thesis dataset and asked: "Do I use a t-test or ANOVA here?"
They had already defended the data collection. They just could not figure out which test to run on it.
Then I started counting. On Cross Validated (stats.stackexchange.com), 181 separate questions share the exact phrasing in their titles: "Which statistical test should I use?" On r/AskStatistics, the same question appears weekly, attached to screenshots of thesis datasets.
When I looked at tutoring platforms like Stuvia, hourly rates for introductory statistics help ranged between $55 and $85 per hour. In many of those sessions, the student is not paying for deep mathematical theory. They are paying someone with a graduate degree to look at a small spreadsheet, ask five basic questions about the study design, and point them toward the right textbook test.
Most universities provide decision trees in PDF format or introductory statistics textbooks. The problem is that static flowcharts gloss over edge cases, and online calculators tend to either lock features behind paywalls or push people toward running tests blindly without checking assumptions.
I wanted a zero-friction alternative. I built Stat Test Selector, an interactive, client-side web tool that walks users through a structured decision path to find the appropriate statistical test, outlines the assumptions they need to check, and gives them an APA 7 reporting template.
The code is open source and hosted on GitHub:
https://github.com/FauziMatakupan/stat-test-selector
The tool runs in the browser as a single HTML file with vanilla JavaScript. There are no accounts, no analytics tracking, no server calls, and no cookies.
Writing the logic looked straightforward at the start. It was only when I began testing all 19 terminal paths and reviewing the output that I realized how many subtle traps exist in standard textbook advice. Here are five decisions and edge cases that forced me to rethink the implementation.
1. Checking independence before touching a distribution
Most introductory statistics flowcharts open with the question: "Is your variable normally distributed?"
This ordering is backwards. If you have repeated measures on the same participants, students nested within classrooms, or longitudinal daily diary entries, your observations are correlated. Running an ordinary two-sample t-test or a standard one-way ANOVA on clustered data violates the fundamental assumption of independent and identically distributed observations.
When independence fails, checking for normality is irrelevant. You need a repeated measures design, a paired test, or a linear mixed-effects model.
In my initial draft, independence was treated as a secondary check. During testing, I moved it to the front. If an observation is dependent or clustered, the selector flags that immediately before letting the user worry about skewness or kurtosis.
2. The log-transform trap in one-sample tests
When continuous data is right-skewed, such as response times or household income, a common textbook remedy is to apply a logarithmic transformation.
Suppose a researcher wants to run a one-sample t-test comparing log-transformed response times against a known benchmark of 500 milliseconds. If they transform their sample data using log(y) but compare it against the raw benchmark value of 500, the resulting test statistic is meaningless. The null comparison value must also be log-transformed to log(500).
In the companion Assumption Checklist, this became an explicit warning. If you transform the scale of your observations, you must transform the null hypothesis value on the other side of the equation.
3. Nagelkerke pseudo-R² is not explained variance
When students move from ordinary least squares regression to binary logistic regression, they naturally look for an equivalent to the R² coefficient of determination. Statistical software packages report Cox and Snell as well as Nagelkerke pseudo-R² values.
In an early draft of the reporting template for binary logistic regression, my output text read: "The model explained 24% of the variance."
That statement is inaccurate. Ordinary R² represents the proportion of variance in a continuous outcome accounted for by linear predictors. Binary outcomes do not have variance in that same sense. Nagelkerke pseudo-R² is a scaled ratio comparing the log-likelihood of the fitted model to an intercept-only null model, adjusted so its theoretical upper bound reaches 1.
Reporting pseudo-R² as "variance explained" in a manuscript is an immediate red flag for peer reviewers and thesis examiners. Correcting that template to describe relative model improvement rather than explained variance was one of the clearest fixes from the review pass.
4. Skipping normality questions for categorical outcomes
In standard statistics software, dialog boxes frequently ask users to specify parameters that make no sense for their chosen data type.
In the first prototype of the selector, Question 4 asked: "Is your continuous outcome roughly normally distributed?"
If a user selected a binary outcome (such as passed versus failed) or an unordered categorical outcome (such as choice of transportation mode) in Question 1, Question 4 still appeared. It confused users because binary and categorical distributions cannot be normally distributed.
The solution was to make the decision path conditional. The moment a user selects a binary, count, or categorical outcome, Question 4 is bypassed entirely. The logic branches directly to logistic regression, chi-square tests of independence, or Poisson regression paths without asking users to judge the distribution shape of categorical categories.
5. Non-parametric tests do not get you off the hook for assumptions
Textbooks often treat the Mann-Whitney U test as a universal escape hatch: if your data is not normal, switch to Mann-Whitney.
That advice is incomplete. The Mann-Whitney U test evaluates stochastic dominance based on mean ranks. If a researcher wants to claim that the test demonstrates a difference in medians between two groups, both groups must have distributions with identical shape and spread.
If two groups have unequal variances or asymmetric shapes, a significant Mann-Whitney result does not prove that their medians differ. When variances are unequal, Welch's t-test is often more robust for comparing means than Mann-Whitney is for medians. When both non-normality and unequal variances are present, rank-based alternatives like the Brunner-Munzel test provide a more reliable inference.
The selector notes this distinction in the assumption output rather than suggesting that switching to a non-parametric test solves every distributional violation.
Architecture: keeping the entire tool in one file
The selector is implemented as a single index.html file containing HTML, CSS, and a vanilla JavaScript decision tree of roughly 40 lines.
Because the core logic is a pure function taking five inputs and returning an object from a 17-item test database, it could be tested in Node.js. All 19 terminal paths were evaluated with automated unit tests to confirm that every combination of inputs resolves to a defined test and valid reporting template.
By avoiding build frameworks and external package dependencies, the tool loads in under 50 milliseconds, runs on any mobile or desktop browser, and requires no ongoing maintenance.
Live tools and repositories
Both tools are free, run locally in the browser, and are open source:
- Stat Test Selector (interactive test picker)
- Assumption Checklist (17-test assumption reference)
Source code on GitHub:
- Selector repository: https://github.com/FauziMatakupan/stat-test-selector
- Checklist repository: https://github.com/FauziMatakupan/assumption-checklist
If you teach applied statistics or consult on thesis research, test the selector with your common edge cases. If you find a data configuration where the tree recommends an inappropriate model or where an assumption threshold needs adjustment, open an issue on the GitHub repository.
Top comments (0)