DEV Community: Milena Monteiro

Comparing Machine Learning Algorithms Using Friedman Test and Critical Difference Diagrams in Python

Milena Monteiro — Thu, 06 Mar 2025 21:00:00 +0000

When you're working with machine learning, deciding which algorithm performs best across multiple datasets can be quite challenging. Simply looking at performance metrics might not be enough, you need statistical methods to be sure. That's when the Friedman Test and Critical Difference (CD) Diagrams can help.

My classmates and I faced this challenge firsthand when preparing a project presentation. We struggled to find a clear way to generate the diagram, so after finally figuring it out, I decided to share this guide to save others time.

In this article, you'll find a Python code that performs this evaluation and visualization. You can also access the complete code on my GitHub gist.

I'll also show you how to modify the code to use accuracy instead of the error rate. The script has been tested on Python 3.8 and above.

The Python script does three main things:

Performs the Friedman Test to statistically evaluate performance differences.
Creates a ranking table comparing the algorithm scores.
Generates and saves a PNG image of the Critical Difference Diagram and the ranking table.

Critical Difference Diagram generated

In the diagram, algorithms connected by a horizontal bar are not significantly different from each other based on the statistical test. Lower-ranked algorithms (positioned further right) generally performed better. The ranking remains the same whether using error rate or accuracy as the performance metric.

Ranking table generated

This table shows the error rates of each algorithm across all datasets. Each cell contains the error rate along with its ranking in parentheses (where 1 is the best and 13 is the worst, since I used 12 datasets).

At the bottom, you’ll find the rank sums and average rankings for each algorithm, for better overall comparison.

I edited the original image and removed some columns for better readability here.

Why Use the Friedman Test and Critical Difference Diagram?

The Friedman test is a non-parametric statistical test designed to detect differences between multiple algorithms across various datasets. It ranks algorithms based on their performance, helping you understand if differences in performance are genuinely significant or just due to chance.

The Critical Difference Diagram visually presents these rankings. It clearly shows which algorithms perform similarly and which are significantly better or worse, making it easy to interpret results at a glance. This diagram is particularly useful when comparing numerous algorithms across multiple datasets.

Preparing Your Data

Now for the implementation, you'll need your data structured like this:

Datasets: Names of your datasets (e.g., MNIST, Fashion-MNIST).
Algorithms: Names of the algorithms you're evaluating. Keep these ordered consistently.
Performance (Error): Lists of error rates for each algorithm per dataset, aligned with your Algorithms list.

For example:

data = {
    'Datasets': ['MNIST', 'Fashion-MNIST', ...],
    'Algorithms': ['NaiveBayes', 'IBk', ..., 'RandomForest'],
    'Performance (Error)': [
        ['30.34%', '3.09%', ..., '88.65%'],   # MNIST
        ['36.72%', '14.35%', ..., '90%'],     # Fashion-MNIST
        # Other datasets...
    ]
}

Make sure the error rates are listed in the same order as their corresponding algorithms. For example, if NaiveBayes is the first algorithm in the list, its performance values should always appear first in each dataset's row.

If you prefer to use accuracy instead of error rates, you can either replace the values in the Performance field with accuracy scores or simply subtract the error rates from 1. I'll also demonstrate how to do this right after the implementation.

Python Implementation

Here's the Python code, also available in this gist on GitHub:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import friedmanchisquare
from aeon.visualisation import plot_critical_difference

data = {
    'Datasets': [
      'MNIST', 'Fashion-MNIST', 
      'e1-Spiral', 'e1-Android',
      'e2-andorinhas', 'e2-chinese',
      'e3-user', 'e3-ecommerce',
      'e4-wine', 'e4-heart',
      'e5-mamiferos', 'e5-titanic'
    ],
    'Algorithms': [
        'NaiveBayes', 'IBk', 'J48', 'RandomForest', 'LMT', 
        'XGBoost', 'SVM', 'LGBM', 'Bagging', 'AdaBoost', 
        'KStar', 'M5P', 'MLP', 'HC', 'E-M'
    ],
    'Performance (Error)': [
        [ # MNIST -ok
            '30.34%', '3.09%', '10.67%', '3.51%', '5.70%', 
            '2.05%', '2.61%', '2.26%', '5.07%', '11.74%', 
            '89.80%', '47.31%', '0%', '44.96%', '88.65%'
        ],
        [ # Fashion-MNIST -ok
            '36.72%', '14.35%', '18.27%', '11.92%', '13.62%', 
            '8.58%', '9.47%', '9.50%', '12.20%', '19.00%',
            '90%', '46.78%', '0.52%', '51.45%', '90%'
        ],
        [ # e1-Spiral -ok
            '29.125%', '0.38%', '2.25%', '1.75%', '2.375%',
            '3.12%', '1.88%', '3.12%', '0%', '4.37%',
            '5.51%', '1.43%', '0%', '49.75%', '72.50%'
        ],
        [ # e1-Android -ok
            '8.1317%', '7.7285%', '4.7491%', '4.5475%', '4.3683%',
            '4.03%', '4.37%', '3.47%', '6.38%', '5.60%', 
            '8.94%', '7.61%', '1.67%', '49.98%', '38.95%'
        ],
        [   '8.1%', '6.65%', '5.60%', '4.90%', '4.60%', 
            '4.00%', '3.75%', '4.25%', '3.50%', '4.75%',
            '4.36%', '2.85%', '3.92%', '48.60%', '49.25%'
        ],
        [ # e2-chinese -ok
            '27.1589%', '12.8911%', '34.9186%', '7.5094%', '10.6383%',
            '7.50%', '6.25%', '5.63%', '16.25%', '34.38%', 
            '0%', '1.21%', '0%', '87.36%', '78.22%'
        ],
        [   '0%', '4.8571%', '0%', '0%', '0%', '0.1429%',
            '2.14%', '0%', '0%', '0%', '0%', '0%',
            '0%', '0.39%', '0%', '79.14%', '4.57%'
        ],
        [ # e3-ecommerce -ok
            '11.37%', '11.15%', '2.39%', '2.07%', '2.42%', 
            '0.90%', '8.80%', '0.70%', '10.35%', '2.85%', 
            '0.02%', '7.56%', '3.96%', '22.11%', '41.49%'
        ],
        [ # e4-wine -ok
            '44.96%', '35.21%', '38.59%', '29.89%', '39.65%',
            '48.95%', '56.56%', '46.85%', '43.94%', '50.99%', 
            '39.23%', '50.82%', '36.51%', '57.34%', '77.98%'
        ],
        [ # e4-heart -ok
            '43.51%', '46.61%', '35.82%', '37.20%', '35.88%',
            '45.71%', '34.51%', '45.73%', '44.16%', '46.1%', 
            '46.15%', '64.18%', '49.22%', '88.1962%', '69.94%'
        ],
        [ # e5-mamiferos -ok
            '0%', '0%', '0%', '0%', '0%', '0%',
            '0%', '0%', '0%', '0%', '0%', '0%',
            '1.57%', '0%', '0.20%', '31.20%', '44.80%'
        ],
        [ # e5-titanic -ok
            '21.3244%', '22.7834%', '22.5589%', '28.3951%', '19.7531%', 
            '16.76%', '27.56%', '14.85%', '7.48%', '10.79%', 
            '27.18%', '61.62%', '26.76%', '38.16%', '38.50%'
        ]
    ]
}

# Convert the data into a DataFrame
datasets = data['Datasets']
algorithms = data['Algorithms']
performance_data = data['Performance (Error)']

# Create a list of dictionaries for each dataset
rows = []
for dataset, performance in zip(datasets, performance_data):
    row = {'Dataset': dataset}
    row.update({alg: perf for alg, perf in zip(algorithms, performance)})
    rows.append(row)

# Create the DataFrame
df = pd.DataFrame(rows)

# Convert string percentages to floats
for alg in algorithms:
    df[alg] = df[alg].str.replace(',', '.').str.rstrip('%').astype(float) / 100

# Calculate the ranking of each algorithm for each dataset
rankings_matrix = df[algorithms].rank(axis=1, method='min', ascending=True)

# Format the results
formatted_results = df[algorithms].copy()
for col in formatted_results.columns:
    formatted_results[col] = formatted_results[col].round(3).astype(str) + " (" + rankings_matrix[col].astype(int).astype(str) + ")"

# Add a row for the sum of ranks and average of ranks
sum_ranks = rankings_matrix.sum().round(3).rename('Sum Ranks')
average_ranks = rankings_matrix.mean().round(3).rename('Average Ranks')

# Add the rows to the formatted DataFrame using concat
formatted_results = pd.concat([formatted_results, sum_ranks.to_frame().T, average_ranks.to_frame().T])

# Add the 'Dataset' column to the formatted DataFrame
formatted_results.insert(0, 'Dataset', df['Dataset'].tolist() + ['Sum Ranks', 'Average Ranks'])

# Display the table
print("Error Table (%) with Ranking:")
print(formatted_results)

# Save the formatted table as an image
fig, ax = plt.subplots(figsize=(14, 8))
ax.axis('tight')
ax.axis('off')
table = ax.table(cellText=formatted_results.values, colLabels=formatted_results.columns, cellLoc='center', loc='center')
table.auto_set_font_size(False)
table.set_fontsize(12)
table.scale(2.5, 2.5)
plt.subplots_adjust(left=0.2, bottom=0.2, right=0.8, top=1, wspace=0.2, hspace=0.2)
plt.savefig('table_with_rankings.png', format="png", bbox_inches="tight", dpi=300)
plt.show()

print("Table saved as 'table_with_rankings.png'")

# Perform the Friedman Test
friedman_stat, p_value = friedmanchisquare(*rankings_matrix.T.values)
print(f"Friedman test statistic: {friedman_stat}, p-value = {p_value}")

# Convert the accuracy matrix into a NumPy array for the critical difference diagram
scores = df[algorithms].values
classifiers = df[algorithms].columns.tolist()

print("Algorithms:", classifiers)
print("Errors:", scores)

# Set the figure size before plotting
plt.figure(figsize=(16, 12))  # Adjust the figure size as needed

# Generate the critical difference diagram
plot_critical_difference(
    scores, 
    classifiers, 
    lower_better=True,
    test='wilcoxon',  # or nemenyi
    correction='holm', # or bonferroni or none
)

# Get the current axes
ax = plt.gca()

# Adjust font size and rotation of x-axis labels
for label in ax.get_xticklabels():
    label.set_fontsize(14) 
    label.set_rotation(45) 
    label.set_horizontalalignment('right')

# Increase padding between labels and axis
ax.tick_params(axis='x', which='major', pad=20)

# Adjust margins to provide more space for labels
plt.subplots_adjust(bottom=0.35)

# Optionally adjust y-axis label font size
ax.tick_params(axis='y', labelsize=12)

# Save and display the plot
plt.savefig('critical_difference_diagram.png', format="png", bbox_inches="tight", dpi=300)
plt.show()

Changing to accuracy instead of error

The code gist already includes a full working script that uses accuracy as the performance metric.

If you're already using the error percentage in the data variable, you can add this last line to convert the values to accuracy:

# Convert string percentages to floats
for alg in algorithms:
    df[alg] = df[alg].str.replace(",", ".").str.rstrip("%").astype(float) / 100
    # Convert to accuracy
    df[alg] = 1 - df[alg]

Then change the rankings_matrix to set the rank ascending value to False:

# Calculate the ranking of each algorithm for each dataset (higher accuracy = better rank)
rankings_matrix = df[algorithms].rank(axis=1, method="min", ascending=False)

Finally, when plotting the Critical Difference diagram, change the lower_better value to False.

plot_critical_difference(
    scores,
    classifiers,
    lower_better=False,  # False for accuracy (higher is better)
    test="wilcoxon",
    correction="holm",
)

Conclusion

I hope this guide helps someone out :) If you have any suggestions or questions, feel free to leave a comment or reach out, and I'll do my best to get back to you!

How to easily load React apps into an ASP.NET project using webpack

Milena Monteiro — Sun, 08 Jan 2023 20:08:13 +0000

In this post I'll walk through the way I found of implementing a React app into an ASP.NET project using webpack, and also show how to connect both projects and make the ASP.NET application work as a back-end for the React pages adding a simple Web API to the existing project.

Note: I've used ASP.NET Web Forms, but this information is adaptable to every ASP.NET application.

The source code for this tutorial is available here.

1. Creating the project

Select your preferred ASP.NET template to start, or skip this part if you're using an existing application. Check out this article if your Visual Studio isn't showing the option to create an ASP.NET Web Application.

I'm leaving the Web API option unselected as I'll show how to add it later.

2. Add your React scripts into a folder and add your package.json file to the project

I'm using the React template app for the sake of simplicity, and pasting only the files contained in the /src folder (only the actual app).

3. Install webpack and webpack-cli

Go to View > Terminal to open the command line and enter these commands to install your project's packages and webpack:

cd <ProjectName>
npm install webpack webpack-cli

4. Setup a webpack.config.js file and a babel.config.js file

I've used this tutorial to configure the webpack file, feel free to adapt and install the loaders according to your project.

The entry property determines the folder where your React app is located and its main file, and the output properties determine the filename of the generated bundled file, and where it'll be located. In my case, it'll be in the Scripts folder.

webpack.config.js file:

const path = require("path");
const webpack = require("webpack");
const MiniCssExtractPlugin = require("mini-css-extract-plugin");

const ROOT = path.resolve(__dirname, "app");
const DESTINATION = path.resolve(__dirname, "Scripts");

module.exports = function (_env, argv) {
    const isProduction = argv.mode === "production";
    const isDevelopment = !isProduction;

    return {
        context: ROOT,
        entry: {
            MyApp: "../my-app/index.js",
        },

        output: {
            filename: "myapp.bundle.js",
            path: DESTINATION,
        },
        module: {
            rules: [
                {
                    test: /\.(jsx|js)$/,
                    exclude: /node_modules/,
                    use: [
                        {
                            loader: "babel-loader",
                            options: {
                                presets: [
                                    "@babel/preset-env",
                                    ["@babel/preset-react", { "runtime": "automatic" }]
                                ]
                            },
                        },
                    ],
                },
                {
                    test: /\.css$/i,
                    use: [
                        isProduction ? MiniCssExtractPlugin.loader : "style-loader",
                        "css-loader",
                    ],
                },
                {
                    test: /\.(png|jpg|gif)$/i,
                    use: {
                        loader: "url-loader",
                        options: {
                            limit: 8192,
                            name: "static/media/[name].[hash:8].[ext]",
                        },
                    },
                },
                {
                    test: /\.svg$/,
                    use: ["@svgr/webpack", "url-loader"],
                },
            ],
        },
        resolve: {
            extensions: [".js", ".jsx"],
        },
        plugins: [
            // fix "process is not defined" error:
            // (do "npm install process" before running the build)
            isProduction &&
            new MiniCssExtractPlugin({
                filename: "assets/css/[name].[contenthash:8].css",
                chunkFilename: "assets/css/[name].[contenthash:8].chunk.css",
            }),

            new webpack.ProvidePlugin({
                process: "process/browser",
            }),
        ].filter(Boolean),
    };
};

//stats.errorDetails: true

babel.config.js file:

module.exports = {
    presets: [
        [
            "@babel/preset-env",
            {
                modules: false
            }
        ],
        "@babel/preset-react"
    ],
    plugins: [
        "@babel/plugin-transform-runtime",
        "@babel/plugin-syntax-dynamic-import",
        "@babel/plugin-proposal-class-properties"
    ],
    env: {
        production: {
            only: ["my-app"],
            plugins: [
                [
                    "transform-react-remove-prop-types",
                    {
                        removeImport: true
                    }
                ],
                "@babel/plugin-transform-react-inline-elements",
                "@babel/plugin-transform-react-constant-elements"
            ]
        }
    }
};

5. Install the needed loaders

For my project, I've installed the babel loader, css loader, svg loader and url loader.

npm install @babel/core babel-loader

From the cited tutorial above, I'm installing these packages to configure the babel loader:

npm install @babel/preset-env @babel/preset-react @babel/runtime @babel/plugin-transform-runtime @babel/plugin-syntax-dynamic-import @babel/plugin-proposal-class-properties babel-plugin-transform-react-remove-prop-types @babel/plugin-transform-react-inline-elements @babel/plugin-transform-react-constant-elements

CSS loader:

npm install css-loader style-loader mini-css-extract-plugin

URL loader:

npm install url-loader

SVG loader:

npm install @svgr/webpack

Install process to avoid the "process is not defined" error:

npm install process

6. Change your build script in package.json to load your webpack configuration

This will create a bundled file when you run the command npm run build, pointing to where the webpack.config.js file is located and setting the mode as development or production.

"scripts": {
    "build": "webpack --mode=development --config ./webpack.config.js"
  }

When running this command build in the terminal it shows:

And it has successfully created a bundled file in my Scripts folder:

7. Running the npm command when the project builds

Now that your React app if fully loaded into your ASP.NET project, you can set it up to run the build command when you build and run your application, so if any changes were made to the React app, they'll be built automatically when you click run.

Right in the end of your <ProjectName>.csproj file, add:

<Project>
    ...
    <Target Name="BeforeBuild">
        <Exec Command="npm install" />
            <Exec Command="npm run build" />
        </Target> 
</Project>

With this configuration, every time you run your project, it'll make sure every package is installed and the bundled file was built. Change the commands according to your project.

8. Call your react app in an ASP.NET page

According to the id set in your index.js, add a div with the same id in the desired .aspx page. In my case, the id is content.

In my index.js page:

const root = ReactDOM.createRoot(document.getElementById('content'));
root.render(
    <React.StrictMode>
        <App />
    </React.StrictMode>
);

I'll load the React app into the About page adding this code to the inside the BodyContent in the About.aspx file to create the div and load the script:

<asp:Content ID="BodyContent" ContentPlaceHolderID="MainContent" runat="server">
    <div id="content"></div>
    <script type="text/javascript" src="../Scripts/myapp.bundle.js"></script>
</asp:Content>

All done! The React app is properly running in the ASP.NET page.

9. Connecting both applications adding a Web API to the ASP.NET project

Create a new item in the project, and select Web API Controller Class:

It'll give you a template file for a ApiController like that:

This API will be your main connection between the ASP.NET project and the React app, so it works as a proper back-end for it.

Now we just need to configure the API route to get it running.

Add an API route to your RouteConfig.cs:

RouteTable.Routes.MapHttpRoute(
                 name: "DefaultApi",
                 routeTemplate: "api/{action}/{id}",
                 defaults: new { id = System.Web.Http.RouteParameter.Optional, controller = "Values" }
            );

I've used this article in the ASP.NET documentation to customize my API route according to what my project needed. In my case all the requests will be made from the /api route, with the additional parameters being the action (method name in the controller) and the optional id. I set the default controller to be the ValuesController since it's the only controller for this application.

Testing the API:

I went to /api/get/1 as I configured, and it got the value, so it's working! Now it's time to pass this value to the React app so both projects are connected.

10. Making a request to your Web API in the React app

I'll add a simple GET request into my React app, set the value I got from the API in the state, and display it in the main page.

In my App.js:

import logo from './logo.svg';
import React, { Component } from "react";
import './App.css';

class App extends Component {
    constructor() {
        super();
        this.state = { value: [] };
    }

    async getData() {
        let resp = await fetch('/api/get/1',
            {
                method: "GET",
                headers: {
                    'Content-Type': 'application/json'
                }
            });
        if (resp.status === 200) {
            let value = await resp.json();
            return value;
        }
    }

    async componentDidMount() {
        let value = await this.getData();
        this.setState({ value: value });
    }

    render() {
        return (
            <div className="App">
                <header className="App-header">
                    <img src={logo} className="App-logo" alt="logo" />
                    <p>
                        Edit <code>src/App.js</code> and save to reload.
                    </p>
                    <a
                        className="App-link"
                        href="https://reactjs.org"
                        target="_blank"
                        rel="noopener noreferrer"
                    >
                        Learn React
                    </a>
                    <p>{this.state.value}</p>
                </header>
            </div>
        );
    }
}

export default App;

Done! The value I got from the ASP.NET Web API is showing in the React page.

Conclusion

This is my first article ever, I hope it helps people in some way, like I needed when I got the task to do this. Comment any suggestions or questions, feel free to reach out and I'll do my best to get back to you ;)