DEV Community: Gaurav Rai

Data visualization: Creating charts using REST API's in React.js

Gaurav Rai — Mon, 14 Jun 2021 05:05:09 +0000

In our previous post of the series we have created the REST API's in Mojolicious

The swagger ui is available on https://localhost/api (port:443) for development environment. If you look at the swagger ui mentioned here, we can see we have 2 API's endpoint

/api/v1/multi-line-chart
/api/v1/stacked-column-chart

We will be querying these two endpoint in our react app.
Also I expect you to have a little understating in React.js.
So, lets get started.

Install react and other dependencies

There are many place where you can get info about how to install react, so I will not go through the whole process in details.

Install Node.js from official site
Install create-react-app

npx create-react-app react-app

It will take some time. After installation is successful, you will see react-app dir created. Go inside it and run

npm start

It will open your default browser and you can see your home page on http://localhost:3000/.

We have to install some more dependencies.
We will add these to package.json. Here is the snapshot of mine.

...
    "dependencies": {
        "mdbreact": "^4.27",
        "@amcharts/amcharts4": "^4.10.18",
        "react": "^16.13.1",
        "react-dom": "^16.13.1",
        "react-scripts": "^3.3.1"
    },
...

We will be using Material Design for Bootstrap and there MDBReact for our react app. For now we will be using v4 as it is stable but they have v5 also release few month back.
For charting we will be using AMcharts v4
The react version we are using is 16.13.1. The current version is 17.0.2. If you are writing any thing from scratch better to go ahead with newer version. My objective here is to show the chart usage and scope is very limited, hence using this version. Also you will creating function component instead of class component in newer version and a lot of complexity can be avoided.

After updating package.json, run

npm install

It will install all the dependencies in node_modules

Also our back-end server is running on https://localhost, we will add this to package.json so that we don't have to add the whole path in fetch.

{
...
   "proxy": "https://localhost",
...
}

Modifying the application

We will be creating few trivial things which every website has - header, footer, body, different pages etc.
Before that we will be removing/modifying few items. If you look at the your dir structure various files and dir is already created by you.

index.html is the entry point. Let update index.js which is actually doing all the work to

import React from "react";
import ReactDOM from "react-dom";

import "@fortawesome/fontawesome-free/css/all.min.css";
import "bootstrap-css-only/css/bootstrap.min.css";
import "mdbreact/dist/css/mdb.css";

import ReactApp from "./ReactApp";

ReactDOM.render(
    <React.StrictMode>
        <ReactApp />
    </React.StrictMode>,
    document.getElementById("root")
);

Here I have imported mdb and other dependencies. I have also renamed the App.js to ReactApp.js and included that.

Creating header

We will creating a component in react-app\src\components\layouts\Header.jsx. We will be using Bootstrap Navbar for it where we ill be creating navigation for different pages.

import React, { Component } from "react";
import {
    MDBNavbar,
    MDBNavbarBrand,
    MDBNavbarNav,
    MDBNavbarToggler,
    MDBCollapse,
    MDBNavItem,
    MDBNavLink,
} from "mdbreact";
import { withRouter } from "react-router";

class Header extends Component {
    constructor(props) {
        super(props);
        this.state = {
            collapse: false,
        };
        this.onClick = this.onClick.bind(this);
    }

    onClick() {
        this.setState({
            collapse: !this.state.collapse,
        });
    }

    render() {
        return (
            <React.Fragment>
                <header>
                    <MDBNavbar color="default-color" dark expand="md" scrolling fixed="top">
                        <MDBNavbarBrand href="/">
                            <strong>Mojo React App</strong>
                        </MDBNavbarBrand>
                        <MDBNavbarToggler onClick={this.onClick} />
                        <MDBCollapse isOpen={this.state.collapse} navbar>
                            <MDBNavbarNav left>
                                <MDBNavItem active={this.props.location.pathname === "/"}>
                                    <MDBNavLink to="/">Home</MDBNavLink>
                                </MDBNavItem>
                                <MDBNavItem active={this.props.location.pathname === "/chart1"}>
                                    <MDBNavLink to="/chart1">LineChart</MDBNavLink>
                                </MDBNavItem>
                                <MDBNavItem active={this.props.location.pathname === "/chart2"}>
                                    <MDBNavLink to="/chart2">ColumnChart</MDBNavLink>
                                </MDBNavItem>
                            </MDBNavbarNav>
                        </MDBCollapse>
                    </MDBNavbar>
                </header>
            </React.Fragment>
        );
    }
}

export default withRouter(Header);

We will be changing the tab highlight based on the this.props.location.pathname value which will be passed from parent component.
This will create a header similar to

Creating footer

Create a component in react-app\src\components\layouts\Footer.jsx. We will be using Bootstrap footer and modifying it as per our need.

import React, { Component } from "react";
import { MDBContainer, MDBFooter } from "mdbreact";

class Footer extends Component {
    render() {
        return (
            <MDBFooter color="default-color" className="font-small pt-4 mt-4">
                <div className="text-center py-3">
                    <MDBContainer fluid className="text-center">
                        <a href="/">Home</a> | <a href="/chart1">LineChart</a>| <a href="/chart2">ColumnChart</a>
                    </MDBContainer>
                </div>
                <div className="footer-copyright text-center py-3">
                    <MDBContainer fluid>
                        &copy; {new Date().getFullYear()} Copyright:{" "}
                        <a href="https://www.mdbootstrap.com"> MDBootstrap.com </a>
                    </MDBContainer>
                </div>
            </MDBFooter>
        );
    }
}

export default Footer;

Creating Home page.

Lets create a small home landing page. Inside react-app\src\components\Home.jsx

import React, { Component } from "react";

class Home extends Component {
    render() {
        return (
            <React.Fragment>
                <h2>This is home page</h2>
                <h5>Welcome to Mojolicious React application</h5>
            </React.Fragment>
        );
    }
}

export default Home;

Simple. Also lets update our ReactApp.js(renamed from App.js) and ReactApp.css(renamed from App.css) to inculded the newly created header and footer.

import React, { Component } from "react";
import { BrowserRouter, Route, Switch } from "react-router-dom";
import "./ReactApp.css";

import Header from "./components/layouts/Header";
import Footer from "./components/layouts/Footer";
import Home from "./components/Home";
import { MDBContainer } from "mdbreact";

class ReactApp extends Component {
    render() {
        return (
            <React.Fragment>
                <BrowserRouter>
                    <Header location={this.props.location} />
                    <main className="site-content">
                        <MDBContainer className="text-center my-5">
                            <Switch>
                                <Route exact path="/" component={Home} />
                                {/* <Route exact path="/chart1" component={Chart1} />
                                <Route exact path="/chart2" component={Chart2} /> */}
                            </Switch>
                        </MDBContainer>
                    </main>
                    <Footer />
                </BrowserRouter>
            </React.Fragment>
        );
    }
}

export default ReactApp;

I have commented the charting components as we haven't created those now.
We have imported the Header and Footer components and on request of / we are rendering the Homecomponent.
There are certain keywords here which has special meaning in react(e.g. Switch etc.). I encourage you to look at official react document to understand them.
If you look, closely we have created our web page skeleton her. Inside BrowserRouter tag you can see - Header at top, main content in the middle and Footer at bottom.

In ReactApp.css

.site-content {
    padding-top: 25px;
}

Let run this and see it in action.

So far so good.

Creating charts

Now lets try to create the chart components and uncomment those lines in ReactApp.js
We will be creating 2 charts for 2 API endpoints on 2 separate pages.

Chart1

I am using this name but better to use some meaningful name here.
Inside react-app\src\components\Chart1.jsx

import React, { Component } from "react";
import LineChart from "./Charts/LineChart";

class Chart1 extends Component {
    constructor(props) {
        super();
        this.state = {
            error: null,
            isLoaded: false,
            chartData: [],
        };
    }
    getChartData = () => {
        fetch("/api/v1/multi-line-chart")
            .then((response) => response.json())
            .then(
                (result) => {
                    this.setState({
                        isLoaded: true,
                        chartData: result.chart_data,
                    });
                },
                (error) => {
                    this.setState({
                        isLoaded: true,
                        error,
                    });
                }
            );
    };

    componentDidMount() {
        this.getChartData();
    }
    render() {
        if (this.state.error) {
            return <div>Error: {this.state.error.message}</div>;
        } else if (!this.state.isLoaded) {
            return (
                <div className="spinner-border" role="status">
                    <span className="sr-only">Loading...</span>
                </div>
            );
        } else {
            return (
                <React.Fragment>
                    <LineChart
                        chartId="chart1"
                        data={this.state.chartData.data}
                        axisNames={{
                            xAxis: [this.state.chartData.label.domainAxis],
                            yAxis: [this.state.chartData.label.rangeAxis],
                        }}
                        lineForXAxis="Date"
                        linesForFirstAxis={["Ford", "Honda", "Renault", "Toyota"]}
                        chartTitle={this.state.chartData.title}
                    />
                </React.Fragment>
            );
        }
    }
}

export default Chart1;

The above code is similar to what available on AJAX and APIs section on react doc.
We are querying our API endpoint /api/v1/multi-line-chart which will return the JSON response which we will be passing to LineChart component for creating multi-line chart.
During the process of request and getting the response we will be using a Loading spinner.
In case of any error in response the same will be available on UI.
The one thing which is of interest is LineChart component. I have created this component some time back and the objective of this article is to showcase its power. This component is created in such a way you can create a single line chart, a multi-line chart or a multi-axle chart. You can also create percentage chart. It doesn't matter whether your x-axis is Date axis or not it will work for both. Just pass the parameter in props, and will be create the chart based on it it on fly. We will look into it. The LineChart component provide you with the layer of abstraction and it can act as a base component for all your Line charts.

Chart2

Inside react-app\src\components\Chart2.jsx

import React, { Component } from "react";
import StackedClusteredColumnChart from "./Charts/StackedClusteredColumnChart";

class Chart2 extends Component {
    constructor(props) {
        super();
        this.state = {
            error: null,
            isLoaded: false,
            chartData: [],
        };
    }
    getChartData = () => {
        fetch("/api/v1/stacked-column-chart")
            .then((response) => response.json())
            .then(
                (result) => {
                    this.setState({
                        isLoaded: true,
                        chartData: result.chart_data,
                    });
                },
                (error) => {
                    this.setState({
                        isLoaded: true,
                        error,
                    });
                }
            );
    };

    componentDidMount() {
        this.getChartData();
    }
    render() {
        if (this.state.error) {
            return <div>Error: {this.state.error.message}</div>;
        } else if (!this.state.isLoaded) {
            return (
                <div className="spinner-border" role="status">
                    <span className="sr-only">Loading...</span>
                </div>
            );
        } else {
            return (
                <React.Fragment>
                    <StackedClusteredColumnChart
                        chartId="chart2"
                        data={this.state.chartData.data}
                        axisNames={{
                            xAxis: [this.state.chartData.label.domainAxis],
                            yAxis: [this.state.chartData.label.rangeAxis],
                        }}
                        columnForXAxis="Year"
                        columnsForYAxis={["Africa", "America", "Antartica", "Asia", "Australia", "Europe"]}
                        chartTitle={this.state.chartData.title}
                    />
                </React.Fragment>
            );
        }
    }
}

export default Chart2;

We are querying our API endpoint /api/v1/stacked-column-chart which will return the JSON response which we will be passing to StackedClusteredColumnChart component for creating column chart.
Again this is similar to LineChart component and powerful too. Just pass the proper params in props and it will do all the work for you.

Before creating the line and column chart component let update the ReactApp.css for loading spinner and chart css

.site-content {
    padding-top: 25px;
}

.chart-display {
    width: 1000px;
    height: 500px;
}

.loader {
    border: 16px solid #f3f3f3;
    border-top: 16px solid #3498db;
    border-radius: 50%;
    width: 120px;
    height: 120px;
    animation: spin 2s linear infinite;
}

@keyframes spin {
    0% {
        transform: rotate(0deg);
    }
    100% {
        transform: rotate(360deg);
    }
}

Creating LineChart.jsx

This is quite a big component.
Amcharts comes with lot of good example and documentation. I encourage you to look at the series doc and multi-axes example to understand more. I have modifies those default configurations and used it as per my need. Each of these are covered in there documentation. I have also added comments in between for understanding.

Inside react-app\src\components\Charts\LineChart.jsx

import React, { Component } from "react";
import * as am4core from "@amcharts/amcharts4/core";
import * as am4charts from "@amcharts/amcharts4/charts";
import am4themes_animated from "@amcharts/amcharts4/themes/animated";

class LineChart extends Component {
    constructor(props) {
        super(props);
        this.state = {
            chartId: this.props.chartId,
            chartdata: this.props.data,
            axisNames: this.props.axisNames,
            lineForXAxis: this.props.lineForXAxis,
            linesForFirstAxis: this.props.linesForFirstAxis,
            linesForSecondAxis: this.props.linesForSecondAxis
                ? this.props.linesForSecondAxis
                : null,
            legendNames: this.props.legendNames
                ? this.props.legendNames
                : this.props.linesForFirstAxis.concat(this.props.linesForSecondAxis),
            isPercentageChart: this.props.isPercentageChart ? true : false,
            isDateAxis: this.props.isDateAxis ? true : false,
        };
    }

    componentDidMount() {
        am4core.useTheme(am4themes_animated);
        const chart = am4core.create(this.state.chartId, am4charts.XYChart);
        this.createChart(chart);
        this.chart = chart;
    }

    componentWillUnmount() {
        if (this.chart) {
            this.chart.dispose();
        }
    }

    createDateAxis = (chart, xAxisName) => {
        let dateAxis = chart.xAxes.push(new am4charts.DateAxis());
        dateAxis.title.text = xAxisName;
        dateAxis.baseInterval.timeUnit = "minute";
        dateAxis.baseInterval.count = 1;
        let axisTooltip = dateAxis.tooltip;
        axisTooltip.background.strokeWidth = 0;
        axisTooltip.background.cornerRadius = 3;
        axisTooltip.background.pointerLength = 0;
        axisTooltip.dy = 5;
        dateAxis.tooltipDateFormat = "MMM dd HH:mm:ss";
        dateAxis.cursorTooltipEnabled = true;
        //dateAxis.renderer.minGridDistance = 50;
        //dateAxis.renderer.grid.template.disabled = true;
        dateAxis.renderer.line.strokeOpacity = 1;
        dateAxis.renderer.line.strokeWidth = 2;
        dateAxis.skipEmptyPeriods = true;
        return dateAxis;
    };
    createCategoryAxis = (chart, xAxisName) => {
        let categoryAxis = chart.xAxes.push(new am4charts.CategoryAxis());
        categoryAxis.dataFields.category = this.state.lineForXAxis;
        categoryAxis.title.text = xAxisName;

        categoryAxis.renderer.grid.template.location = 0;
        categoryAxis.renderer.minGridDistance = 20;
        categoryAxis.renderer.cellStartLocation = 0.1;
        categoryAxis.renderer.cellEndLocation = 0.9;
        return categoryAxis;
    };
    createValueAxisRange = (valueAxis, value, color, guideLabel) => {
        let axisRange = valueAxis.axisRanges.create();
        axisRange.value = value;
        axisRange.grid.stroke = am4core.color(color);
        axisRange.grid.strokeOpacity = 0.7;
        // https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/stroke-dasharray
        axisRange.grid.strokeDasharray = "4 5";
        axisRange.grid.opacity = 0.8;
        axisRange.grid.strokeWidth = 2;
        axisRange.label.inside = true;
        axisRange.label.text = guideLabel;
        axisRange.label.fill = axisRange.grid.stroke;
        axisRange.label.verticalCenter = "bottom";
        axisRange.label.horizontalCenter = "middle";
        return axisRange;
    };
    createValueAxis = (chart, yAxisName, opposite) => {
        let valueAxis = chart.yAxes.push(new am4charts.ValueAxis());
        valueAxis.title.text = yAxisName;
        valueAxis.min = 0;
        valueAxis.ghostLabel.disabled = true;
        valueAxis.extraMax = 0.1;
        valueAxis.numberFormatter = new am4core.NumberFormatter();
        valueAxis.numberFormatter.numberFormat = "# a";
        if (typeof opposite !== "undefined") {
            valueAxis.renderer.opposite = opposite;
        }
        if (this.state.linesForSecondAxis) {
            valueAxis.renderer.grid.template.disabled = true;
        }
        valueAxis.renderer.line.strokeOpacity = 1;
        valueAxis.renderer.line.strokeWidth = 2;
        valueAxis.renderer.ticks.template.disabled = false;
        valueAxis.renderer.ticks.template.strokeOpacity = 1;
        valueAxis.renderer.ticks.template.strokeWidth = 2;
        return valueAxis;
    };

    createAxis = (chart, xAxisName, yAxisName) => {
        // Create x-axes
        let xAxis;
        if (this.state.isDateAxis) {
            xAxis = this.createDateAxis(chart, xAxisName);
        } else {
            xAxis = this.createCategoryAxis(chart, xAxisName);
        }
        // Create y-axes
        let valueAxis = this.createValueAxis(chart, yAxisName);
        if (this.state.isPercentageChart) {
            // This is to create horizontal 'red' (on 80%) and 'green'(on 100%) lines
            this.createValueAxisRange(valueAxis, 80, "#ff0000", "Threshold");
            this.createValueAxisRange(valueAxis, 100, "#00b33c", "Goal");
        }
        return [xAxis, valueAxis];
    };

    createTrendLine = (chart, value, name, yAxisId, bulletType, fillOpacity) => {
        let series = chart.series.push(new am4charts.LineSeries());
        series.name = name;
        series.dataFields.valueY = value;
        if (this.state.isDateAxis) {
            series.dataFields.dateX = this.state.lineForXAxis;
        } else {
            series.dataFields.categoryX = this.state.lineForXAxis;
        }
        series.strokeWidth = 2;
        series.strokeOpacity = 0.8;
        series.tensionX = 0.7;
        series.yAxis = yAxisId;
        series.fillOpacity = fillOpacity;
        if (this.state.isPercentageChart) {
            series.tooltipText = "{name}: [bold]{valueY}%[/]";
        } else {
            series.tooltipText = "{name}: [bold]{valueY}[/]";
        }
        series.tooltip.background.cornerRadius = 13;
        series.tooltip.background.fillOpacity = 0.8;
        series.tooltip.exportable = false;
        series.minBulletDistance = 15;
        // Enable the number in the legend on hovering over the graph
        if (this.state.isPercentageChart) {
            series.legendSettings.itemValueText = "[bold]{valueY}%[/]";
            series.legendSettings.valueText =
                "(Avg: [bold]{valueY.average.formatNumber('#.##')}%[/])";
        } else {
            series.legendSettings.itemValueText = "[bold]{valueY}[/]";
        }
        // Add a drop shadow filter on columns
        //let shadow = series.filters.push(new am4core.DropShadowFilter());
        //shadow.dx = 10;
        //shadow.dy = 10;
        //shadow.blur = 5;
        let bullet;
        let hoverState;
        switch (bulletType) {
            case "rectangle":
                bullet = series.bullets.push(new am4charts.Bullet());
                let square = bullet.createChild(am4core.Rectangle);
                square.strokeWidth = 1;
                square.width = 7;
                square.height = 7;
                square.stroke = am4core.color("#fff");
                square.horizontalCenter = "middle";
                square.verticalCenter = "middle";
                hoverState = square.states.create("hover");
                hoverState.properties.scale = 1.7;
                break;
            case "triangledown":
            case "triangleup":
                bullet = series.bullets.push(new am4charts.Bullet());
                let triangle = bullet.createChild(am4core.Triangle);
                triangle.strokeWidth = 1;
                triangle.width = 7;
                triangle.height = 7;
                if (bulletType === "triangleup") {
                    triangle.direction = "top";
                } else {
                    triangle.direction = "bottom";
                }
                triangle.stroke = am4core.color("#fff");
                triangle.horizontalCenter = "middle";
                triangle.verticalCenter = "middle";
                hoverState = triangle.states.create("hover");
                hoverState.properties.scale = 1.7;
                break;
            case "circle":
            case "hollowcircle":
                bullet = series.bullets.push(new am4charts.CircleBullet());
                bullet.strokeWidth = 1;
                bullet.circle.radius = 3.5;
                bullet.fillOpacity = 1;
                if (bulletType === "circle") {
                    bullet.stroke = am4core.color("#fff");
                    bullet.circle.fill = series.stroke;
                } else {
                    bullet.stroke = series.stroke;
                    bullet.circle.fill = am4core.color("#fff");
                }
                hoverState = bullet.states.create("hover");
                hoverState.properties.scale = 1.7;
                break;
            default:
                break;
        }
        this.addEvents(series);
        return series;
    };
    addEvents = (series) => {
        // Enable interactions on series segments
        let segment = series.segments.template;
        segment.interactionsEnabled = true;

        // Create hover state
        let hoverState = segment.states.create("hover");
        hoverState.properties.strokeWidth = 4;
        hoverState.properties.strokeOpacity = 1;
    };
    createLegend = (chart) => {
        chart.legend = new am4charts.Legend();
        chart.legend.maxWidth = 400;
        chart.legend.markers.template.width = 40;
        chart.legend.markers.template.height = 10;
        // Use this to change the color of the legend label
        //chart.legend.markers.template.disabled = true;
        //chart.legend.labels.template.text = "[bold {color}]{name}[/]";
        chart.legend.itemContainers.template.paddingTop = 2;
        chart.legend.itemContainers.template.paddingBottom = 2;
        chart.legend.labels.template.maxWidth = 130;
        chart.legend.labels.template.truncate = true;
        chart.legend.itemContainers.template.tooltipText = "{name}";
        chart.legend.numberFormatter = new am4core.NumberFormatter();
        chart.legend.numberFormatter.numberFormat = "#.## a";
        chart.legend.itemContainers.template.events.on("over", (ev) => {
            let lineSeries = ev.target.dataItem.dataContext.segments.template;
            lineSeries.strokeOpacity = 1;
            lineSeries.strokeWidth = 4;
        });
        chart.legend.itemContainers.template.events.on("out", function (ev) {
            let lineSeries = ev.target.dataItem.dataContext.segments.template;
            lineSeries.strokeOpacity = 0.8;
            lineSeries.strokeWidth = 2;
        });
        chart.legend.valueLabels.template.adapter.add("textOutput", function (text, target) {
            if (text === "(Avg: [bold]%[/])" || text === "(Total: [bold][/])") {
                return "N/A";
            } else if (text === "[bold]%[/]" || text === "[bold][/]") {
                return "";
            }
            return text;
        });
    };

    createExportMenu = (chart, title) => {
        chart.exporting.menu = new am4core.ExportMenu();
        chart.exporting.menu.verticalAlign = "bottom";
        chart.exporting.filePrefix = title + " LineChart";
    };

    createCursor = (chart) => {
        chart.cursor = new am4charts.XYCursor();
    };

    createScrollBar = (chart, series) => {
        chart.scrollbarX = new am4core.Scrollbar();
        chart.scrollbarX.thumb.background.fill = am4core.color("#66c9ff");
        chart.scrollbarX.startGrip.background.fill = am4core.color("#0095e6");
        chart.scrollbarX.endGrip.background.fill = am4core.color("#0095e6");
        chart.scrollbarX.stroke = am4core.color("#66c9ff");
        chart.scrollbarX.height = "20";
        chart.scrollbarX.exportable = false;
        // Add simple vertical scrollbar
        // chart.scrollbarY = new am4core.Scrollbar();
        // chart.scrollbarY.thumb.background.fill = am4core.color("#66c9ff");
        // chart.scrollbarY.startGrip.background.fill = am4core.color("#0095e6");
        // chart.scrollbarY.endGrip.background.fill = am4core.color("#0095e6");
        // chart.scrollbarY.stroke = am4core.color("#66c9ff");
        // chart.scrollbarY.width = "20";
        // chart.scrollbarY.exportable = false;
    };

    addChartTitle = (chart, titleText) => {
        let title = chart.titles.create();
        title.text = titleText;
        title.fontSize = 25;
        title.marginBottom = 30;
    };

    createChart = (chart) => {
        chart.data = this.state.chartdata;
        chart.colors.step = 4;
        // This will change the background color of chart
        //chart.background.fill = "#fff";
        //chart.background.opacity = 0.5;
        this.createLegend(chart);

        this.createCursor(chart);

        // Use this to change bullet type in lines if needed
        //let bulletsType = ["circle", "triangleup", "triangledown", "hollowcircle", "rectangle"];
        let axis = this.createAxis(
            chart,
            this.state.axisNames.xAxis[0],
            this.state.axisNames.yAxis[0]
        );
        for (let i = 0; i < this.state.linesForFirstAxis.length; i++) {
            //if (typeof bulletsType[i] !== "undefined") {
            this.createTrendLine(
                chart,
                this.state.linesForFirstAxis[i],
                this.state.legendNames[i],
                axis[1],
                "circle"
            );
            //} else {
            //    this.createTrendLine(chart, this.state.linesForFirstAxis[i], axis[1]);
            //}
        }

        if (this.state.linesForSecondAxis) {
            let yAxis = this.createValueAxis(chart, this.state.axisNames.yAxis[1], "true");
            for (let i = 0; i < this.state.linesForSecondAxis.length; i++) {
                let series;
                let fillOpacity = 0.2;
                //if (typeof bulletsType[this.state.linesForSecondAxis.length - i] !== "undefined") {
                series = this.createTrendLine(
                    chart,
                    this.state.linesForSecondAxis[i],
                    this.state.legendNames[this.state.linesForFirstAxis.length + i],
                    yAxis,
                    "circle",
                    fillOpacity
                );
                //} else {
                //    series = this.createTrendLine(chart, this.state.linesForSecondAxis[i], yAxis);
                //}
                if (this.state.linesForSecondAxis.length === 1) {
                    yAxis.renderer.line.stroke = series.stroke;
                    yAxis.renderer.ticks.template.stroke = series.stroke;
                }
            }
        }
        this.createScrollBar(chart);
        if (this.props.chartTitle) {
            this.addChartTitle(chart, this.props.chartTitle);
            this.createExportMenu(chart, this.props.chartTitle);
        } else {
            this.createExportMenu(chart, "");
        }
    };

    componentDidUpdate(prevProps) {
        if (this.chart !== null) {
            if (JSON.stringify(prevProps.data) !== JSON.stringify(this.props.data)) {
                this.chart.data = this.props.data;
            }
        }
    }

    render() {
        return (
            <div>
                <div id={this.state.chartId} className="chart-display" />
            </div>
        );
    }
}

export default LineChart;

Creating StackedClusteredColumnChart.jsx

Again please have a look at amcharts doc and demo for more understanding. For starter you can look at this example
Inside react-app\src\components\Charts\StackedClusteredColumnChart.jsx

import React, { Component } from "react";
import * as am4core from "@amcharts/amcharts4/core";
import * as am4charts from "@amcharts/amcharts4/charts";
import am4themes_animated from "@amcharts/amcharts4/themes/animated";

class StackedClusteredColumnChart extends Component {
    constructor(props) {
        super(props);
        this.state = {
            chartId: this.props.chartId,
            chartdata: this.props.data,
            axisNames: this.props.axisNames,
            columnForXAxis: this.props.columnForXAxis,
            columnsForYAxis: this.props.columnsForYAxis,
            legendNames: this.props.legendNames
                ? this.props.legendNames
                : this.props.columnsForYAxis,
            showDummyData: this.props.showDummyData ? true : false,
            isPercentageChart: this.props.isPercentageChart ? true : false,
            isDateAxis: this.props.isDateAxis ? true : false,
        };
    }
    componentDidMount() {
        am4core.useTheme(am4themes_animated);
        const chart = am4core.create(this.state.chartId, am4charts.XYChart);
        this.createChart(chart);
        this.chart = chart;
    }
    componentWillUnmount() {
        if (this.chart) {
            this.chart.dispose();
        }
    }
    getLinearGradientModifier = () => {
        // Adding greadient to create a round bar effect
        let fillModifier = new am4core.LinearGradientModifier();
        fillModifier.brightnesses = [0, 1, 1, 0];
        fillModifier.offsets = [0, 0.45, 0.55, 1];
        fillModifier.gradient.rotation = 0;
        return fillModifier;
    };
    getLinearGradient = (color1, color2) => {
        let gradient = new am4core.LinearGradient();
        gradient.addColor(color1);
        if (typeof color2 !== "undefined") {
            gradient.addColor(color2);
        } else {
            gradient.addColor("#66c9ff");
            gradient.addColor(color1);
        }
        gradient.rotation = 90;
        return gradient;
    };
    createLegend = (chart) => {
        chart.legend = new am4charts.Legend();
        chart.legend.maxWidth = 400;
        chart.legend.markers.template.width = 20;
        chart.legend.markers.template.height = 20;
        chart.legend.itemContainers.template.paddingRight = 2;
        chart.legend.itemContainers.template.paddingLeft = 2;
        chart.legend.labels.template.maxWidth = 100;
        chart.legend.labels.template.truncate = true;
        chart.legend.valueLabels.template.align = "left";
        chart.legend.valueLabels.template.textAlign = "end";
        chart.legend.itemContainers.template.tooltipText = "{name}";

        chart.legend.itemContainers.template.events.on("over", (ev) => {
            let seriesColumn = ev.target.dataItem.dataContext.columns.template;
            seriesColumn.fillOpacity = 1;
        });
        chart.legend.itemContainers.template.events.on("out", function (ev) {
            let seriesColumn = ev.target.dataItem.dataContext.columns.template;
            seriesColumn.fillOpacity = 0.7;
        });
        chart.legend.valueLabels.template.adapter.add("textOutput", function (text, target) {
            if (text === "(Avg: [bold]%[/])" || text === "(Total: [bold][/])") {
                return "N/A";
            } else if (text === "[bold]%[/]" || text === "[bold][/]") {
                return "";
            }
            return text;
        });
    };
    createScrollBar = (chart) => {
        chart.scrollbarX = new am4core.Scrollbar();
        chart.scrollbarX.background.fillOpacity = 0.7;

        let gradient = this.getLinearGradient("#0095e6");
        chart.scrollbarX.thumb.background.fill = gradient;
        chart.scrollbarX.thumb.background.fillOpacity = 0.7;
        chart.scrollbarX.startGrip.background.fill = am4core.color("#0095e6");
        chart.scrollbarX.endGrip.background.fill = am4core.color("#0095e6");
        chart.scrollbarX.stroke = am4core.color("#66c9ff");
        chart.scrollbarX.height = "20";
        chart.scrollbarX.exportable = false;
    };
    createExportMenu = (chart, title) => {
        chart.exporting.menu = new am4core.ExportMenu();
        chart.exporting.menu.verticalAlign = "bottom";
        chart.exporting.filePrefix = title + " StackedColumnChart";
    };
    createCursor = (chart) => {
        chart.cursor = new am4charts.XYCursor();
    };
    createDateAxis = (chart, xAxisName) => {
        let dateAxis = chart.xAxes.push(new am4charts.DateAxis());
        dateAxis.title.text = xAxisName;
        dateAxis.cursorTooltipEnabled = true;
        dateAxis.renderer.minGridDistance = 30;
        dateAxis.renderer.cellStartLocation = 0.1;
        dateAxis.renderer.cellEndLocation = 0.9;
        dateAxis.skipEmptyPeriods = true;
        dateAxis.renderer.grid.template.location = 0;
        dateAxis.renderer.axisFills.template.disabled = false;
        dateAxis.renderer.axisFills.template.fill = am4core.color("#b3b3b3");
        dateAxis.renderer.axisFills.template.fillOpacity = 0.2;
        return dateAxis;
    };
    createCategoryAxis = (chart, xAxisName) => {
        let categoryAxis = chart.xAxes.push(new am4charts.CategoryAxis());
        categoryAxis.dataFields.category = this.state.columnForXAxis;

        categoryAxis.title.text = xAxisName;
        categoryAxis.renderer.grid.template.location = 0;
        categoryAxis.renderer.minGridDistance = 20;
        categoryAxis.renderer.cellStartLocation = 0.1;
        categoryAxis.renderer.cellEndLocation = 0.9;
        categoryAxis.renderer.axisFills.template.disabled = false;
        categoryAxis.renderer.axisFills.template.fillOpacity = 0.2;
        categoryAxis.renderer.axisFills.template.fill = am4core.color("#b3b3b3");
        return categoryAxis;
    };
    createValueAxis = (chart, yAxisName) => {
        let valueAxis = chart.yAxes.push(new am4charts.ValueAxis());
        valueAxis.title.text = yAxisName;
        valueAxis.min = 0;
        valueAxis.ghostLabel.disabled = true;
        valueAxis.extraMax = 0.1;
        valueAxis.renderer.line.strokeOpacity = 1;
        valueAxis.renderer.line.strokeWidth = 2;
        valueAxis.renderer.ticks.template.disabled = false;
        valueAxis.renderer.ticks.template.strokeOpacity = 1;
        valueAxis.renderer.ticks.template.strokeWidth = 2;
        return valueAxis;
    };
    createValueAxisRange = (valueAxis, value, color, guideLabel) => {
        let axisRange = valueAxis.axisRanges.create();
        axisRange.value = value;
        axisRange.grid.stroke = am4core.color(color);
        axisRange.grid.strokeOpacity = 0.7;
        // https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/stroke-dasharray
        axisRange.grid.strokeDasharray = "4 5";
        axisRange.grid.opacity = 0.8;
        axisRange.grid.strokeWidth = 2;
        axisRange.label.inside = true;
        axisRange.label.text = guideLabel;
        axisRange.label.fill = axisRange.grid.stroke;
        axisRange.label.verticalCenter = "bottom";
        axisRange.label.horizontalCenter = "middle";
        return axisRange;
    };

    createAxis = (chart, xAxisName, yAxisName) => {
        // Create x-axes
        let xAxis;
        if (this.state.isDateAxis) {
            xAxis = this.createDateAxis(chart, xAxisName);
        } else {
            xAxis = this.createCategoryAxis(chart, xAxisName);
        }
        // Create y-axes
        let valueAxis = this.createValueAxis(chart, yAxisName);
        if (this.state.isPercentageChart) {
            // This is to create horizontal 'red' (on 80%) and 'green'(on 100%) lines
            this.createValueAxisRange(valueAxis, 80, "#ff0000", "Threshold");
            this.createValueAxisRange(valueAxis, 100, "#00b33c", "Goal");
        }
        return [xAxis, valueAxis];
    };

    createSeries = (chart, field, name, stacked, showDummyData) => {
        // For normal coloums
        let series = chart.series.push(new am4charts.ColumnSeries());
        // For 3D coloums
        //let series = chart.series.push(new am4charts.ColumnSeries3D());
        series.name = name;
        series.dataFields.valueY = field;
        if (this.state.isDateAxis) {
            series.dataFields.dateX = this.state.columnForXAxis;
        } else {
            series.dataFields.categoryX = this.state.columnForXAxis;
        }
        if (showDummyData && !this.state.isPercentageChart) {
            series.columns.template.propertyFields.dummyData = field + "_breakdown";
            series.columns.template.tooltipText =
                "[bold]{name} #{categoryX}\n[bold]Total:[/] {valueY}\n[#00cc44 bold]Pass:[/] {dummyData.pass}\n[#ff0000 bold]Fail:[/] {dummyData.fail}\n[#ff471a bold]Error:[/] {dummyData.error}\n[#ff9900 bold]Terminated:[/] {dummyData.terminated}[/]";
        } else if (this.state.isPercentageChart) {
            series.columns.template.tooltipText = "{name}: [bold]{valueY}%[/]";
        } else {
            series.columns.template.tooltipText = "{name}: [bold]{valueY}[/]";
        }
        series.strokeWidth = 2;
        series.tooltip.background.fillOpacity = 0.9;
        series.tooltip.exportable = false;
        series.stacked = stacked;
        series.columns.template.width = am4core.percent(90);
        series.columns.template.fillOpacity = 0.7;
        series.tooltip.getFillFromObject = false;
        series.tooltip.background.fill = am4core.color("#ffffff");
        series.tooltip.background.stroke = chart.colors.getIndex(
            chart.colors.currentStep - chart.colors.step
        );
        series.tooltip.background.strokeWidth = 2;
        series.tooltip.label.fill = am4core.color("#000000");

        let fillModifier = this.getLinearGradientModifier();
        series.columns.template.fillModifier = fillModifier;
        if (this.state.isPercentageChart) {
            series.legendSettings.itemValueText = "[bold]{valueY}%[/]";
            series.legendSettings.valueText =
                "(Avg: [bold]{valueY.average.formatNumber('#.##')}%[/])";
        } else {
            series.legendSettings.itemValueText = "[bold]{valueY}[/]";
            series.legendSettings.valueText = "(Total: [bold]{valueY.sum.formatNumber('#.')}[/])";
        }
        series.cursorTooltipEnabled = false;
        this.addEvents(series);
    };

    addChartTitle = (chart, titleText) => {
        let title = chart.titles.create();
        title.text = titleText;
        title.fontSize = 25;
        title.marginBottom = 30;
    };

    addEvents = (series) => {
        let hoverState = series.columns.template.states.create("hover");
        hoverState.properties.fillOpacity = 1;
    };

    preZoomChart = (chart, xAxis) => {
        chart.events.on("ready", (a) => {
            // different zoom methods can be used - zoomToIndexes, zoomToDates, zoomToValues
            if (this.state.isDateAxis) {
                xAxis.start = 0.4;
                xAxis.end = 1;
            } else {
                xAxis.zoomToIndexes(chart.data.length - 9, chart.data.length, false, true, true);
            }
        });
    };

    createChart = (chart) => {
        chart.data = this.state.chartdata;
        chart.colors.step = 3;
        if (this.props.isDateAxis) {
            chart.dateFormatter.inputDateFormat = "yyyy-MM-ddThh";
        }
        this.createLegend(chart);
        this.createCursor(chart);
        // Fow now its single axis hence '0'
        let axis = this.createAxis(
            chart,
            this.state.axisNames.xAxis[0],
            this.state.axisNames.yAxis[0]
        );

        this.createScrollBar(chart);
        if (this.props.chartTitle) {
            this.addChartTitle(chart, this.props.chartTitle);
            this.createExportMenu(chart, this.props.chartTitle);
        } else {
            this.createExportMenu(chart, "");
        }
        for (let i = 0; i < this.state.columnsForYAxis.length; i++) {
            this.createSeries(
                chart,
                this.state.columnsForYAxis[i],
                this.state.legendNames[i],
                false,
                this.state.showDummyData
            );
        }

        // Prezoom only one we have some big dataset (equal or more than 10 points on xaxis)
        if (chart.data.length > 9) {
            this.preZoomChart(chart, axis[0]);
        }
        // Extending the axisFills to axis labels
        chart.plotContainer.adapter.add("pixelHeight", function (value, target) {
            return value + 40;
        });
    };
    render() {
        return (
            <div>
                <div id={this.state.chartId} className="chart-display" />
            </div>
        );
    }
}

export default StackedClusteredColumnChart;

I have tried to create a proper function name so that it will be easy for you to understand what I am doing in chart. Also, I have added comments in between for your understanding.

Lets run in and see it in action.
Hit the 'LineChart' on Navbar.

Similary for ColumnChart

Let's see the action in real time.

Note - If the above gif is not working you can take a look directly at - https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6d836lpxie5c24rxs49f.gif

Takeaway

There are certain places where I haven't explained that much. The article is getting too big and there are numerous other places where you get that info. Also my aim was to show the usage of amcharts library in react.js. We have already done the similar thing many times in past (if you are following my article). The only difference right now is jsx.
LineChart and StackedClusteredColumnChart components are the 2 key takeaway. You can use them as independent components in your code or modify it as per your need.

Conclusion

With this we are done with our series.
In the past few months I have gone through different charting libraries and ways to use them. I have created different article based on that.

and the current one ofcourse.

I hope it will be helpful for you in future. All these libraries are quite poweful and you can use any one of them to create elegenet charts.

The above example is also available at github.

References

Amcharts logo taken from here
React logo taken from here

A fusion of Makefile and Docker

Gaurav Rai — Sun, 16 May 2021 17:48:23 +0000

In our previous article we have created a Dockerfile for our Mojolicious application. In there I mentioned that we have to run several docker command and when we map it across different environment we have to take care of different permutations and combinations.
What if we can automate it more. What if we can abstract the complexity and as a user I just have to run minimum command to get it working.
Here comes the Makefile. We will be using this to make our process more easier. For windows you can use gmake.exe utility instead of make. It comes with Strawberry Perl.
Lets create a Make file in mojo_react_app/Makefile

DOCKER_TAG ?= mojo_react_app:development
NO_CACHE ?= false
PULL ?= false
BUILD_TYPE ?= development
CONTAINER_NAME ?= mojo_react_app
HTTPS_OUT_PORT ?= 443
HTTPS_IN_PORT ?= 443
HTTP_OUT_PORT ?= 80
HTTP_IN_PORT ?= 80
DOCKER_REPO ?= <docker_repo_url>
ADDRESS ?= 0.0.0.0
DOCKER_CFG = <path to docker config on machine>

.DEFAULT_GOAL := help

build:
    # Build the dockerfile
    docker build --pull=${PULL} --no-cache=$(NO_CACHE) -t $(DOCKER_TAG) .

create:
    # Create the container and copy the certificates
    echo 'Creating container for ${ADDRESS}'
    docker create --name ${CONTAINER_NAME}_${BUILD_TYPE} -t \
        -p ${ADDRESS}:$(HTTPS_OUT_PORT):$(HTTPS_IN_PORT) -p ${ADDRESS}:$(HTTP_OUT_PORT):$(HTTP_IN_PORT) --env mode=${BUILD_TYPE} \
        ${DOCKER_TAG}
    docker cp ${DOCKER_CFG}/mojo_react_app/apache-certificate/apache_certificate_${BUILD_TYPE}.crt ${CONTAINER_NAME}_${BUILD_TYPE}:/etc/ssl/certs/server.crt
    docker cp ${DOCKER_CFG}/mojo_react_app/apache-certificate/apache_certificate_${BUILD_TYPE}.key ${CONTAINER_NAME}_${BUILD_TYPE}:/etc/ssl/private/server.key

start:
    # Start a container
    docker start ${CONTAINER_NAME}_${BUILD_TYPE}

run:
    # Create and start the container
    make create -e ADDRESS=${ADDRESS} HTTPS_OUT_PORT=${HTTPS_OUT_PORT} HTTPS_IN_PORT=${HTTPS_IN_PORT} BUILD_TYPE=${BUILD_TYPE} DOCKER_TAG=${DOCKER_TAG}
    make start -e BUILD_TYPE=${BUILD_TYPE}
    make show

stop:
    # Stop a running container
    docker stop ${CONTAINER_NAME}_${BUILD_TYPE};

clean_container:
    # remove previous container
    docker rm -f ${CONTAINER_NAME}_${BUILD_TYPE} 2>/dev/null \
    && echo 'Container for "${CONTAINER_NAME}_${BUILD_TYPE}" removed.' \
    || echo 'Container for "${CONTAINER_NAME}_${BUILD_TYPE}" already removed or not found.'

clean_image:
    # remove created image
    docker rmi ${DOCKER_TAG} 2>/dev/null \
    && echo 'Image(s) for "${DOCKER_TAG}" removed.' \
    || echo 'Image(s) for "${DOCKER_TAG}" already removed or not found.'

dev: 
    make build -e PULL=${PULL} NO_CACHE=${NO_CACHE}
    make clean_container
    make create
    make start
    make show

stag:
    $(eval override BUILD_TYPE=staging)
    $(eval override DOCKER_TAG=${CONTAINER_NAME}:${BUILD_TYPE})
    # This will return only the IP address associated with the domain name ans assign it to ADDRESS
    $(eval override ADDRESS=$(shell dig +short <your staging URL e.g. mojo-react-app-staging.com>))

    make build -e PULL=true NO_CACHE=true DOCKER_TAG=${DOCKER_TAG}
    make clean_container -e BUILD_TYPE=${BUILD_TYPE}
    make create -e ADDRESS=${ADDRESS} BUILD_TYPE=${BUILD_TYPE} DOCKER_TAG=${DOCKER_TAG}
    make start -e BUILD_TYPE=${BUILD_TYPE}
    make show

prod:
    $(eval override BUILD_TYPE=production)
    $(eval override DOCKER_TAG = ${CONTAINER_NAME}:${BUILD_TYPE})
    # This will return only the IP address associated with the domain name ans assign it to ADDRESS
    $(eval override ADDRESS=$(shell dig +short <your production URL e.g. mojo-react-app.com))

    make build -e PULL=true NO_CACHE=true DOCKER_TAG=${DOCKER_TAG}
    make clean_container -e BUILD_TYPE=${BUILD_TYPE}
    make create -e ADDRESS=${ADDRESS} BUILD_TYPE=${BUILD_TYPE} DOCKER_TAG=${DOCKER_TAG}
    make start -e BUILD_TYPE=${BUILD_TYPE}
    make show

show:
    # show running containers
    docker ps | grep ${CONTAINER_NAME}

rebuild:
    # rebuilt the dockerfile
    make clean_container -e BUILD_TYPE=${BUILD_TYPE}
    make build -e PULL=true NO_CACHE=true DOCKER_TAG=${DOCKER_TAG}

up:
    # Run container on port
    make build -e PULL=true NO_CACHE=true DOCKER_TAG=${DOCKER_TAG}
    make run -e HTTPS_OUT_PORT=${HTTPS_OUT_PORT} HTTPS_IN_PORT=${HTTPS_IN_PORT} BUILD_TYPE=${BUILD_TYPE} DOCKER_TAG=${DOCKER_TAG}

login:
    # run as a service and attach to it
    docker exec -it ${CONTAINER_NAME}_${BUILD_TYPE} bash

release:
    make build -e PULL=true NO_CACHE=true DOCKER_TAG=${DOCKER_TAG}
    make push -e VERSION=${VERSION}

push:
    docker push $(DOCKER_REPO)/$(CONTAINER_NAME):$(VERSION)

pull:
    docker pull $(DOCKER_REPO)/$(CONTAINER_NAME):$(VERSION)

# Docker tagging
tag:
    ## Generate container tags for the `{version}`
    @echo 'create tag $(VERSION)'
    docker tag $(DOCKER_TAG) $(DOCKER_REPO)/$(CONTAINER_NAME):$(VERSION)

help:
    @echo ''
    @echo 'Usage: make [TARGET] [EXTRA_ARGUMENTS]'
    @echo 'Targets:'
    @echo '  build      build docker --image--'
    @echo '  rebuild    rebuild docker --image--'
    @echo '  dev        run docker --container-- in development mode => $(DOCKER_TAG)'
    @echo '  stag       run docker --container-- in staging mode'
    @echo '  prod       run docker --container-- in production mode'
    @echo '  login      run as service and login --container--'
    @echo '  clean_image        remove docker --image-- '
    @echo ''
    @echo 'Extra arguments:'
    @echo 'CONTAINER_NAME=:     make clean_container -e CONTAINER_NAME=my_app (no need to provide this param, it will be set by default)'
    @echo 'BUILD_TYPE=:     make clean_container -e CONTAINER_NAME=my_app BUILD_TYPE=staging (whether the build type is 'development', 'staging' or 'production')'
    @echo 'DOCKER_TAG=:     make build -e DOCKER_TAG=my_app:staging'
    @echo 'HTTPS_OUT_PORT=:     make create -e HTTPS_IN_PORT=8080 (port from which the request will come- outside world)'
    @echo 'HTTPS_IN_PORT=:      make create -e HTTPS_IN_PORT=443 (port to which the request will be forwarded)'

I know its overwhelming. Lets go through each one of them.

Initial 12 lines are the default parameters. In case you will not provide any parameter it will take those arguments. By default everything is in development mode. You will have to overwrite it for different modes.

DOCKER_TAG     =>  Name of your app with the env name (e.g. mojo_react_app:development)
NO_CACHE       =>  Whether to start the build from scratch (true|false). Default - false
PULL           =>  Whether to pull a newer version of the image (true|false). Default - false
BUILD_TYPE     =>  Build environment (development|staging|production). Default - development
CONTAINER_NAME =>  Name of the container (e.g. mojo_react_app)
HTTPS_OUT_PORT =>  HTTPS host port visible to outside world while publishing the container(443)
HTTPS_IN_PORT  =>  HTTPS container port on which traffic will be coming inside the container(443)
HTTP_OUT_PORT  =>  HTTP host port visible to outside world while publishing the container(80)
HTTP_IN_PORT   =>  HTTP container port on which traffic will be coming inside the container(80)
DOCKER_REPO    =>  URL of the docker repo where you want to push or pull image.
ADDRESS        =>  IP address of the container from where the outside word can access it. Default - 0.0.0.0
DOCKER_CFG     =?  Path to docker config files on host machine. This will contain the apache certificate and will be private to you.

build will build your dockerfile
create will create the container and copy the apache certificates
start will start the container
run will create and start the container. It will internally call the create and start command.
stop will stop the running container.
clean_container will remove/delete the remove previous container if exist.
clean_image will remove/delete created image if exist.
show will show the info about running container.
rebuild will rebuilt the dockerfile. It will just call clean_container and build the image again.
up will run the container on given port. It will always pull the new image.
login will login to the container. Using this you can see the traffic logs and what is going on inside the container.
release push pull tag - These commands you will be using when you want to create tag and push it to docker repo url(somewhere where you can access it and you don't have to create the image again). This is handful in Jenkins deployments, otherwise in normal dev work its not that important and will not be used much.
help will give you the info about different param and what they do.
Now 3 important commands which we will be using more often is -

    make dev
    make stag
    make prod

make dev

dev is written in such a way that you don't have to worry about your container name and param. Just update the default value with your project specific value one time and your dev environment will be ready in no time.
It will build the container, create it and run it with single command.

make dev

Run this command.
Here are the screenshot at different point of time for this command. For the first time this command will take few minutes but subsequent run will be completed within few seconds.

I have highlighted different section for better understanding.
You can access your app on https://localhost/

make stag

This command you will be running when you want to deploy your application in staging environment.
If you look closely we are overriding the default config for staging in initial few lines.

    $(eval override BUILD_TYPE=staging)
    $(eval override DOCKER_TAG=${CONTAINER_NAME}:${BUILD_TYPE})
    # This will return only the IP address associated with the domain name ans assign it to ADDRESS
    $(eval override ADDRESS=$(shell dig +short <your staging URL e.g. mojo-react-app-staging.com>))

The 4th lines is where you have to make your changes as per your requirement. We are overriding the default value of address(0.0.0.0) with your domain name of staging environment. We are using the dig utility to generate the IP address from that domain name and assigning it to ADDRESS.
Just run

make stag

and you can access your staging environment.
E.g.
https://mojo-react-app-staging.com/
or
https://staging-ip-add/

make prod

This command you will be running when you want to deploy your application in production environment.
Again, you can see we are overriding the default config for production in initial few lines.

    $(eval override BUILD_TYPE=production)
    $(eval override DOCKER_TAG = ${CONTAINER_NAME}:${BUILD_TYPE})
    # This will return only the IP address associated with the domain name ans assign it to ADDRESS
    $(eval override ADDRESS=$(shell dig +short <your production URL e.g. mojo-react-app.com))

Here again in 4th lines is where you have to make your changes as per your requirement. Update it to your prod environment hostname. We are using the dig utility to generate the IP address from that domain name and assigning it to ADDRESS.
Just run

make prod

and you can access your staging environment.
E.g.
https://mojo-react-app.com/
or
https://production-ip-add/

Stop the container

make stop

By default it will stop the dev container. Pass different param for stag or prod env.

Login to container

make login

Again it will login to dev one by default. You can pass param to override that behavior.

Conclusion

This Makefile is just a wrapper around different docker commands.
Especially during development you can enable live reload and your changes in host will be available in docker container also . More info here.
You can use it for any Dockerfile. Here I am using a Mojolicious application. But the same Makefile can be used for a Django application or any other.

I hope this is useful for you and make your life more smoother.
The Makefile is also available on github

Perl onion logo taken from here
Mojolicious logo taken from here
Docker logo taken from here
GNU logo taken from here

Getting ready for production (Apache + uWSGI + Mojo)

Gaurav Rai — Sun, 09 May 2021 09:41:55 +0000

In previous article we have created few REST API's.
Now lets try to deploy it to production. It will be a long article so brace your self.
For production deployment we will be using few things -

Mojolicious already comes with hypnotoad which is a production grade server. You can use it and you will be ready in no time. This way is already mentioned in Deploying a Mojolicious Application using Hypnotoad and Apache. If you are interested you can read it. When I first deployed my mojo app to production I followed that approach (thank you perlmaven 😃).

There are several other ways which we will see. But before that I want to thank a unknown person. While exploring more about this deployment process and what to choose I stumble upon this link - https://codeday.me/es/qa/20190709/1032129.html
This link is not working as of today(Don't try to run it on 'http' as it is redirecting to some malicious site). I am not sure who is the owner of this website. But this website contains a lot of useful information regarding Perl. Even though it was in Japanese language (AFAIK), which you can translate easily, I found a great amount of knowledge here. Its a shame its not working anymore. This is also a motivation for me to write this article as I don't want that knowledge to get lost on the internet. I hope someday it will come up and people can see lot of interesting article on it. The thing which I am going to write next is inspired from one of the article mentioned there(the uWSGI part). Right now even I am not sure whether that is the correct link or not, but that is the only one I have. So a big thanks to that unknown person.

I encourage you to look at this stackoverflow answer to understand the basic concepts. It talks about Nginx, PSGI, Plack, Starman and Dancer. It has one of the best explanation and it should be added in hall of fame if possible. 😁

So lets get started.

Why uWSGI ?

According to uWSGI project -

The uWSGI project aims at developing a full stack for building hosting services.
Versatility, performance, low-resource usage and reliability are the strengths of the project.
The "WSGI" part in the name is a tribute to the namesake Python standard, as it has been the first developed plugin for the project.

Please note that uWSGI(highly-performant WSGI server implementation) is not any language specific. I am taking an example for Perl language, but it is almost similar to other language (Python, Ruby,PHP etc.). I will provide the option in between for other languages too.

Sample Web App Architecture :

Perl

Code <-> Web framwork (Mojolicious, Dancer, Catalyst etc.) <-> PSGI <-> uWSGI <-> Web Server(apache, nginx) <-> Clients

Python

Code <-> Web framwork (Django, Flask etc.) <-> WSGI <-> uWSGI <-> Web Server(apache, nginx) <-> Clients

Ruby

Code <-> Web framwork (Rails etc.) <-> RACK <-> uWSGI <-> Web Server(apache, nginx) <-> Clients

Please keep in mind that PSGI/WSGI/RACK is not Yet Another web application framework. PSGI/WSGI/RACK is a specification to decouple web server environments from web application framework code. Nor is PSGI a web application API. Web application developers (end users) will not run their web applications directly using the PSGI interface, but instead are encouraged to use frameworks that support PSGI.

uWSGI is toolkit that contains PSGI/WSGI/RACK middleware, helpers and adapters to web servers. In other words, they are the implementation of PSGI/WSGI/RACK specification.

There are several ways to setup the particular architecture mentioned above.

Using Gunicorn or 'Green Unicorn' (inspired from Ruby 'Unicorn') for Python
OR
Using Plack(inspired from Ruby 'Rack') and Starman / Starlet / Gazelle for Perl.
Using (WSGI + uWSGI) for Python
OR
(PSGI + uWSGI) for Perl
OR
(RACK + uWSGI) for Ruby.

All these WSGI/PSGI/RACK are plugin provided by uWSGI which extend across almost all languages.

So the question is which option is best or which has more advantage over other - I will try to explain with help of Perl but I hope it is true across other language as everyone is inspired from each other.

The PSGI 'protocol' (like WSGI) is essentially a calling convention for a subroutine. A request enters the application as a subroutine call with a hash as an argument. The application responds through the return value of the subroutine: an arrayref that contains an HTTP status code, HTTP headers and body. There is more than that, but those are the essential elements.
What this means is that a process can only implement PSGI if the process contains a Perl interpreter. To achieve this, the process can be implemented in Perl or implemented in a language like C that can be loaded by the libperl.so shared library. Similarly, a process can only implement WSGI if it contains a Python interpreter.
In reality the PSGI application is within the Starman process. So there are really only two parts (although both parts are multi-process containers).
When we say that "nginx has uWSGI directly integrated", this does not mean that a WGSI application runs within the Nginx process. It means that the WSGI application runs in a separate uwsgi process and Nginx communicates with that process through a TCP socket using the uWSGI protocol. This is essentially the same model as Nginx with Starman behind, but with the distinction that the socket connection to Starman will use the HTTP protocol:

    .----------------------.          .-----------.
    |       Starman        |          |   Nginx   |
    |                      |   HTTP   |      /    |   HTTP
    | .------------------. |<---------|   Apache  |<-------(internet)
    | | PSGI Application | |          |           |
    | '------------------' |          |           |
    '----------------------'          '-----------'

The HTTP protocol has higher overhead than the uWSGI protocol(remember OSI Model - HTTP at Application Layer(7) while TCP at Transport Layer(4)), so you can get better performance by running an application server that speaks the uWSGI socket protocol and can load libperl.so to implement the PSGI interface. uWSGI can do that :

    .----------------------.           .----------.
    |        uWSGI         |           |  Nginx   |
    |                      |   uWSGI   |     /    |   HTTP
    | .------------------. |<----------|  Apache  |<-------(internet)
    | | PSGI Application | |           |          |
    | '------------------' |           |          |
    '----------------------'           '----------'

Hence it is encouraged to use uWSGI over any language specific implementation.

All few thing to note here is that-

uWSGI implementation is available in almost all language (no more mod_perl or mod_python (language specific))
It can be implemented across CGI script also even mason too.
Applicable across different Web server. So if tomorrow you want nginx instead of Apache, its 5 min of work. Even some has out of box support for it (e.g. Nginx).
Scalability
Speed

How to use uWSGI ?

First install 'uWSGI'. It is available as package in several OS/distributions. So at most you have to do

sudo apt-get install uwsgi

Install the plugin specific to language you are using. Each language has a plugin associated with it -

    uwsgi-plugin-psgi    -> Perl
    uwsgi-plugin-python3 -> Python3
    uwsgi-plugin-python  -> Python2.7
    uwsgi-plugin-ruby    -> Ruby

You can install these plugin using apt-get(which I prefer).
OR

curl http://uwsgi.it/install | bash -s psgi /tmp/uwsgi

Web server(Apache, Nginx) specific changes needed

For Apache - mod_proxy_uwsgi. More info here

sudo apt-get install libapache2-mod-proxy-uwsgi

Nginx includes uwsgi protocol support out of the box. More info here

How to Run

Running on terminal (without web server)-

Python

uwsgi --http-socket :8080 --wsgi-file <Application Script>

Perl

uwsgi_psgi --http-socket :8080 --psgi script/my_app

uwsgi --plugins http,psgi --http :8080 --http-modifier1 5 --psgi script/my_app

Please note that 'http-modifier' tag in option.

uWSGI supports various languages and platform. When the server receives a request it has to know where to ‘route’ it.
Each uWSGI plugin has an assigned number (the modifier), the perl/psgi one has the 5. So –http-modifier1 5 means “route to the psgi plugin”.
ruby/rack has 7.
lua has 6.

If using socket (through Web Server)-

Python

uwsgi --socket 127.0.0.1:8080 -w wsgi

Perl

uwsgi_psgi --socket :8080 --protocol=http --psgi script/my_app

There are various command line parameters. Have a look at Options to understand them and use them as per your need.
Since it support a large range of options, we will be using the config file instead of command line params as it will be easy to read and maintain.

Whatever we have read till now is generic introductions of uWSGI.
Now lets try to use it for our mojo app.

Creating Apache configuration

You can use any web server you like. I am just using Apache(I have a soft corner for it 😅).
Create a file in etc\apache2.conf

<VirtualHost *:80>
    ServerAdmin grai@gmail.com
    ServerName  mojo-react-app.com

    RewriteEngine on
    # This checks to make sure the connection is not already HTTPS
    RewriteCond %{HTTPS} off [OR]
    RewriteCond %{HTTP:X-Forwarded-Proto} !https

    # Redirect http (port 80) to https (port 443)
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [NC,R=301,L]

</VirtualHost>

<VirtualHost *:443>
    ServerAdmin grai@gmail.com
    ServerName  mojo-react-app.com

    SSLEngine on
    SSLProxyEngine on
    SSLProxyVerify none 
    SSLProxyCheckPeerCN off
    SSLProxyCheckPeerName off
    SSLProxyCheckPeerExpire off
    SSLCertificateFile /etc/ssl/certs/server.crt
    SSLCertificateKeyFile /etc/ssl/private/server.key

    DocumentRoot /var/www/html
    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>
    ProxyRequests Off
    ProxyPreserveHost On
    ProxyPass / uwsgi://127.0.0.1:6363/ keepalive=On
    ProxyPassReverse / uwsgi://127.0.0.1:6363/
    RequestHeader set X-Forwarded-Ssl on
    RequestHeader set X-Forwarded-Proto "https"

    ErrorLog ${APACHE_LOG_DIR}/mojo-react-app-error.log
    LogLevel warn
    CustomLog ${APACHE_LOG_DIR}/mojo-react-app-access.log combined
</VirtualHost>

We have two host listening on 80(http) and 443(https). While the first one is doing nothing except redirecting to 443. It means if we try to access out website from browser as plain http it will redirect the connection to https and will force us to go through that path. We will look into it in real time. But its a good addition to a website. With just few lines of code we are supporting both http and https.
I really love that when a web site owner does this (redirect).

The configuration is pretty standard. You can find more details here and here about what each one of these means.
One thing I would like to point out is ProxyPass and ProxyPassReverse. Here we just doing the forwarding. 6363 is the port where our uWSGI app is listening. We are forwarding the incoming request to it which will forward it to back-end server. More details - Reverse Proxy.
SSLCertificateFile and SSLCertificateKeyFile contains the path to CA signed certificate file on your system.
Also, we have created a separate error and access log for our application(which you should do in case you have multiple application running). All the incoming request hitting Apache and error if any will be logged to those logs.

Creating uWSGI configuration

As I mentioned before we will be using config file for uWSGI instead of command line, so lets create that file. I am creating a ini format file but other formats are also acceptable(xml, json, yaml)

[uwsgi]
project = MojoReactApp
chdir = /home/mojo_react_app
# spawn the specified number of workers/processes
workers = 4
# enable master process
master = true

[uwsgi_psgi]
ini = :uwsgi
# Currently the module lacks the ability to set modifiers, though this will be fixed soon.
# An alternative is to set the plugin you want to use as the first one (0)
plugins = 0:psgi
psgi = script/mojo_react_app
# set uwsgi protocol modifier1 (perl/psgi is 5)
http-socket-modifier1 = 5
# do not catch $SIG{__DIE__}
perl-no-die-catch = true

[development]
# This will load the uwsgi section below
ini = :uwsgi_psgi
# set environment variable
env = PLACK_ENV=development
# bind to the specified UNIX/TCP socket using uwsgi protocol
socket = 127.0.0.1:6363
logto = log/uwsgi_development.log

[staging]
# This will load the uwsgi section below
ini = :uwsgi_psgi
# set environment variable
# staging is similar to production, there is no separate DB for staging
env = PLACK_ENV=production
# bind to the specified UNIX/TCP socket using uwsgi protocol
socket = 127.0.0.1:6363
logto = log/uwsgi_staging.log

[production]
# This will load the uwsgi section below
ini = :uwsgi_psgi
# set environment variable
env = PLACK_ENV=production
# bind to the specified UNIX/TCP socket using uwsgi protocol
socket = 127.0.0.1:6363
logto = log/uwsgi_production.log

Please have a look at Configuration for more details about each items. Check the ini-files section for creating the ini config.

By default, uWSGI uses the [uwsgi] section, but you can specify another section name while loading the INI file with the syntax filename:section, that is:

uwsgi --ini myconf.ini:app1

Alternatively, you can load another section from the same file by omitting the filename and specifying just the section name.

In our [uwsgi] section we have created the config which we want globally available.
[uwsgi_psgi] section contain items specific to psgi plugin. If you are using some other plugin (e.g. ruby) you can update it for that. I have added the comment on each line to understand it better.
I have created 3 more section - [development], [staging] and [production]. This is something which I generally do for all my project. In all 3 I have loaded the [uwsgi_psgi] section and in [uwsgi_psgi] I have loaded [uwsgi] meaning all those items are available in these 3 section.
You can add the params which you think is specific to particular environment(e.g. PLACK_ENV).
socket contains the address where the request coming from Apache will be forwarded (remember ProxyPass). This is the address where your mojolicious app will be running.
logto contains the filename where the logs will be generated.

The reason I have created 3 sections - dev, stag and prod because it make everything so smooth. In just few min you have your specific environment ready. We will look into it more and what I meant by this statement.

Creating the signed certificate

For now I am creating a self-signed certificate
but for actual production use you should get a CA signed certificate. If your website is going to be in a public space accessible across world, better to get a proper certificate.
A self-signed certificate is ok till your usage is limited (intranet site etc.).

There are plenty of place where you can learn how to create it. One of them is - How To Create a Self-Signed SSL Certificate
Also, you should not be committing this certificate in your repo(git, svn etc). I have just committed it for demo.
This is private to you, use it with utmost care.

Creating the startup script

Lets create a wrapper script which will start our app.
Inside script\start_mojo_react_app.sh

#!/bin/bash
# This script can be run by docker or directly by provide the parameter.
# Move it outside and run -
# ./start_mojo_react_app.sh -m "development" or
# ./start_mojo_react_app.sh -m "production"

# Restart Web server (here Apache)
service apache2 restart

# start Mojo React App
while getopts "m:" opt; do
    case ${opt} in
        m)
            mode="${OPTARG}"
            echo "Running in '$mode' mode"
            if [ "$mode" == "development" ]; then
                # Since we are using uwsgi, we have commented the morbo
                # exec morbo -l "https://*:6363" script/mojo_react_app
                exec uwsgi --ini etc/uwsgi.conf:$mode
            elif [[ "$mode" = "production" || "$mode" = "staging" ]]; then
                # exec hypnotoad -f script/mojo_react_app
                exec uwsgi --ini etc/uwsgi.conf:$mode
            else 
                echo "Wrong mode provided. Accepted value - development, staging or production" 1>&2
            fi
            ;;
        : )
            echo "Invalid option: $OPTARG requires an argument" 1>&2
    esac
done

In my case I have merged the staging and production environment. But you can segregate them based on your need.
I have commented the morbo(development server) and hypnotoad(HTTP and WebSocket production server) line. The reason being we are using uWSGI. In case you don't want to use uWSGI you can uncomment those and remove uWSGI line. Also don't forget to add the hypnotoad specific config and update the ProxyPass and ProxyPassReverse in apache2.conf 😏

    ProxyPass / http://127.0.0.1:6363/ keepalive=On
    ProxyPassReverse / http://127.0.0.1:6363/

Creating the Dockerfile

Now lets wrap our application with all the configuration file created above.
Create a Dockerfile in mojo_react_app\Dockerfile

# build environment
FROM ubuntu:18.04

# By default will run in 'dev' mode
ENV mode=development

# Needed dependencies
RUN apt-get update && apt-get install -y \
    build-essential \
    gnupg \
    curl \
    vim \
    less \
    openssl=1.1.1-1ubuntu2.1~18.04.9 \
    libssl-dev \
    zlib1g-dev \
    apache2 \
    uwsgi=2.0.15-10.2ubuntu2.1 \
    uwsgi-plugin-psgi=2.0.15-10.2ubuntu2.1 \
    libapache2-mod-proxy-uwsgi=2.0.15-10.2ubuntu2.1

# Needed dependencies specific to project
RUN curl -L https://cpanmin.us | perl - -M https://cpan.metacpan.org -n Mojolicious@9.17
RUN curl -L https://cpanmin.us | perl - -M https://cpan.metacpan.org -n Mojolicious::Plugin::OpenAPI@4.03
RUN curl -L https://cpanmin.us | perl - -M https://cpan.metacpan.org -n Mojolicious::Plugin::SwaggerUI@0.0.4
RUN curl -L https://cpanmin.us | perl - -M https://cpan.metacpan.org -n IO::Socket::SSL@2.070
RUN curl -L https://cpanmin.us | perl - -M https://cpan.metacpan.org -n JSON::XS@4.03

# Apache specific configuration
RUN \
    a2enmod headers && \
    a2enmod proxy && \
    a2enmod proxy_http && \
    a2enmod proxy_uwsgi && \
    a2enmod rewrite && \
    a2enmod ssl

# Disable the default apache home page on port 80
RUN a2dissite 000-default.conf

# Copy your codebase from local to inside container
COPY . /home/mojo_react_app
RUN mkdir -p /home/mojo_react_app/log/

# These will be used while setting virtual host in apache.conf
ENV APACHE_LOG_DIR /var/log/apache2
ENV APACHE_LOCK_DIR /var/lock/apache2
ENV APACHE_PID_FILE /var/run/apache2.pid

# Copy Apache config file
COPY etc/apache2.conf /etc/apache2/sites-available/mojo_react_app.conf
RUN ln -s /etc/apache2/sites-available/mojo_react_app.conf /etc/apache2/sites-enabled/mojo_react_app.conf

WORKDIR /home/mojo_react_app

# Expose both http and https port
EXPOSE 80 443

COPY script/start_mojo_react_app.sh /start_mojo_react_app.sh
RUN chmod a+x /start_mojo_react_app.sh

ENTRYPOINT /start_mojo_react_app.sh -m "$mode"

I have added comment for better understanding.

We are using the ubuntu(18.04) image. You can use the recent one (20.04) but still some plugins are missing for this version hence I have used 18.04. You can also use the official Perl docker image instead if you want (e.g. perl:5.32).
By default we will running in development mode. You can override it by providing different value at runtime (e.g. staging or production). This is the one which will cause the the different section to pick in uwsgi.conf.
Next we are installing some tools(e.g. vim). In case you don't want it you can remove those lines. Few notable plugins/tools are - openssl(for https), apache2, uwsgi, uwsgi-plugin-psgi(for perl), libapache2-mod-proxy-uwsgi(for apache2).
I have used some version which are specific to ubuntu(18.04), you can update that as per your image or not use at all.
Next I have installed some dependencies specific to our Mojolicious app. I am using cpanminus for this. I have added the version number also to get that specific version. There are again multiple way to achieve this. One of them is using cpanm(with cpanfile) from command line as mentioned here and here. Another is using Carton which is dependency manager for Perl(similar to Bundler in Ruby).
Since our scope is limited I have used it like that. In big project you may want to follow one of above mentioned approach.
Next is some Apache specific configuration. We are enabling some modules in Apache. More info at a2enmod.
We are enabling the ability of handling HTTP proxy requests, handling the uWSGI protocol, ssl etc. Also we disable the default Apache homepage which is available on port 80.
After that we are copying our mojo_react_app code base and placing it at /home/mojo_react_app inside container.
We are setting some environment variables which we are using in apache.conf (remember ${APACHE_LOG_DIR}).
Next we copied our virtual host config file and copied it inside container(site-available). Also we created a symbolic link for it in /etc/apache2/sites-enabled/.
Now we make the change our working dir to /home/mojo_react_app inside container where all our code base is.
We have exposed the 80 and 443 port to outside world.
We copied the startup script to our current working dir(which is /home/mojo_react_app).
Finally we are running that startup script with mode as param (by default - development).

All the docker keywords and there meaning is already available at - Dockerfile reference. Check it out for more info.

With that we are ready to start our application.

Running the application

Build the dockerfile

docker build --pull=false --no-cache=false -t mojo_react_app:development .

Create the container

docker create --name mojo_react_app_development -t \
       -p 0.0.0.0:80:80 -p 0.0.0.0:443:443 \
       --env mode=development docker create --name

Copy the signed certificates

We will copy the self signed certificate inside the created container. This certificate will be available somewhere on your prod machine(private). For now I am copying it from the project dir.

docker cp mojo_react_app/apache-certificate/apache_certificate_development.crt mojo_react_app_development:/etc/ssl/certs/server.crt

docker cp mojo_react_app/mojo_react_app/apache-certificate/apache_certificate_development.key mojo_react_app_development:/etc/ssl/private/server.key

Remember the destination path is the one which we used in apache2.conf for virtual host configuration.

Start the container

docker start mojo_react_app_development

Show running containers

Just to check whether our container is running or not

docker ps | grep mojo_react_app_development

You can change the development in name and mode to staging or production for different environment.

Lets login to container and see what is going on there.

docker exec -it mojo_react_app_development bash

Lets go to apache log dir to see the logs there

We can see our application access and error are getting generated here. Whatever the request apache received that will be logged here.

Lets go to our project dir and see the uWSGI logs

All the mojolicious app log will be logged here.

Lets tail this log and see whats inside

Hmm, all the 4 works which we have configured in [uwsgi] section inside uwsgi.conf is ready to rock.

Just open the browser and hit the http://localhost

Just look at the Network tab in browser. Even though you hit http it got redirected to https. The status code 301 says so. The url itself is now https://localhost at the top.
Also you can see the server is Apache(2.4.29) running on Ubuntu and not Mojolicious(Perl) since we are using the reverse proxy architecture.

Lets go ahead and click to open the API page.

Voila!!!. So far so good. It is similar to what we have seen previously except on https.

Since we are already tailing the uWSGi log, lets see whats happens there -

We got a GET request on /api and we render the template based on our internal logic. All the debug message will go away when you run it in production mode.

Go ahead and try to do the GET request on the endpoints. You will be able to do it without any issue.

A high level diagram of architecture we followed is -

What next

Well even though we are using docker, we have to run several command manually. We have to take care of different environment while creating and starting the image. Not to mention the copy of Apache certificate. What if we can automate it more.
What if we say - make dev and all the thing got taken cared of.
Similarly for make stag and make prod.
This thing will come handy when you will do the automated deployment using Jenkins or some other ways.
I can see the sparkle in your eyes.

We will look into that in our next section.

Bonus

Our good folks at Mojolicious already thought of containers and clouds. Have a look at Containers for more info.
You can generate Makefile and Dockerfile for your app using just 2 simple command. Inside your project dir -

mojo generate makefile
./script/mojo_react_app generate dockerfile

Honorable Mention -

https://stackoverflow.com/questions/12127566/an-explanation-of-the-nginx-starman-dancer-web-stack/12134555#12134555
https://uwsgi-docs.readthedocs.io/en/latest/index.html
https://codeday.me/es/qa/20190709/1032129.html
https://metacpan.org/pod/PSGI
https://perlmaven.com/deploying-a-mojolicious-application

And a lot of different articles read over the years, the source of which I don't remember. 😭

The above code is also available at github

Perl onion logo taken from here
Mojolicious logo taken from here
OpenAPI logo taken from here
uWSGI logo taken from here
Docker logo taken from here
Apache logo taken from here
Ubuntu logo taken from here

Creating REST APIs with Perl, Mojolicious and OpenAPI

Gaurav Rai — Sat, 01 May 2021 09:02:13 +0000

As promised in previous article, we will be using react.js with jsx. For that we should have our server ready.
Today we will be looking at creating the RESTful API's with OpenAPI.
Again we will using Mojolicious for that. Some other prerequisite which we will be using.

Mojolicious::Plugin::OpenAPI - OpenAPI / Swagger plugin for Mojolicious
Mojolicious::Plugin::SwaggerUI - Swagger UI plugin for Mojolicious

We will be using the same example mentioned previously.
So without further delay lets get started.

Creating the data config

For multi-line chart -

{
    "title": "Number of automobiles sold per day by manufacturer",
    "label": {
        "domainAxis": "Date",
        "rangeAxis": "Numbers of automobiles sold"
    },
    "data": [
        {
            "Date": "2020-04-15",
            "Honda": 10,
            "Toyota": 20,
            "Ford": 6,
            "Renault": 16
        },
        {
            "Date": "2020-04-16",
            "Honda": 3,
            "Toyota": 15,
            "Ford": 19,
            "Renault": 10
        },
        {
            "Date": "2020-04-17",
            "Honda": 5,
            "Toyota": 8,
            "Ford": 12,
            "Renault": 6
        },
        {
            "Date": "2020-04-18",
            "Honda": 9,
            "Toyota": 10,
            "Ford": 4,
            "Renault": 12
        }
    ]
}

We will be creating one more chart in this example - stacked clustered column chart. We will look into it in more details when we will create the client side. But for now, lets see the input data which is similar to previous one.

{
    "title": "Revenues by region",
    "label": {
        "domainAxis": "Date",
        "rangeAxis": "Expenditure"
    },
    "data": [
        {
            "Year": "2003",
            "Europe": 2.5,
            "America": 2.5,
            "Asia": 2.1,
            "Australia": 1.2,
            "Antartica": 0.2,
            "Africa": 0.1
        },
        {
            "Year": "2004",
            "Europe": 2.6,
            "America": 2.7,
            "Asia": 2.2,
            "Australia": 1.3,
            "Antartica": 0.3,
            "Africa": 0.1
        },
        {
            "Year": "2005",
            "Europe": 2.8,
            "America": 2.9,
            "Asia": 2.4,
            "Australia": 1.4,
            "Antartica": 0.3,
            "Africa": 0.1
        }
    ]
}

Creating the mojo app

The version I am using for this article is 9.14.

$  mojo generate app MojoReactApp

Running the app -

$ morbo ./script/mojo_app
Web application available at http://127.0.0.1:3000

Now we will modify this app to suit our need.

Writing the specification

For API creation we will follow the OpenAPI specification. We will be using 3.0 version for this.

{
    "openapi": "3.0.2",
    "info": {
        "version": "1.0",
        "title": "Mojo React App API",
        "description": "This is a sample server for a mojolicious app.",
        "contact": {
            "name": "Gaurav Rai",
            "url": "https://github.com/rai-gaurav"
        }
    },
    "servers": [
        {
            "url": "/api/v1",
            "description": "Version one api"
        }
    ],
    "paths": {
        "/multi-line-chart": {
            "get": {
                "summary": "Get multi line chart data",
                "tags": ["Chart Data"],
                "operationId": "getMultiLineChartData",
                "x-mojo-name": "get_multi_line_chart_data",
                "x-mojo-to": {
                    "controller": "LineCharts",
                    "action": "get_multi_line_chart"
                },
                "responses": {
                    "200": {
                        "description": "Multi Line Chart Response",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "type": "object",
                                    "properties": {
                                        "chart_data": {
                                            "type": "object",
                                            "items": {
                                                "type": "object"
                                            }
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        },
        "/stacked-column-chart": {
            "get": {
                "summary": "Get stacked column chart data",
                "tags": ["Chart Data"],
                "operationId": "getStackedColumnChartData",
                "x-mojo-name": "get_stacked_column_chart_data",
                "x-mojo-to": {
                    "controller": "ColumnCharts",
                    "action": "get_stacked_column_chart"
                },
                "responses": {
                    "200": {
                        "description": "Stacked Column Chart Response",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "type": "object",
                                    "properties": {
                                        "chart_data": {
                                            "type": "object",
                                            "items": {
                                                "type": "object"
                                            }
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

The above specification can also be written in .yaml format also. For now I am using json one. Save it as public/api.json.

The equivalent in .yaml format is -

---
openapi: 3.0.2
info:
  version: '1.0'
  title: Mojo React App API
  description: This is a sample server for a mojolicious app.
  contact:
    name: Gaurav Rai
    url: https://github.com/rai-gaurav
servers:
- url: "/api/v1"
  description: Version one api
paths:
  "/multi-line-chart":
    get:
      summary: Get multi line chart data
      tags:
      - Chart Data
      operationId: getMultiLineChartData
      x-mojo-name: get_multi_line_chart_data
      x-mojo-to:
        controller: LineCharts
        action: get_multi_line_chart
      responses:
        '200':
          description: Multi Line Chart Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  chart_data:
                    type: object
                    items:
                      type: object
  "/stacked-column-chart":
    get:
      summary: Get stacked column chart data
      tags:
      - Chart Data
      operationId: getStackedColumnChartData
      x-mojo-name: get_stacked_column_chart_data
      x-mojo-to:
        controller: ColumnCharts
        action: get_stacked_column_chart
      responses:
        '200':
          description: Stacked Column Chart Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  chart_data:
                    type: object
                    items:
                      type: object

Inside server section we are using /api/v1 as url. You can provide full url or relative one. If relative, it is resolved against the server where the given OpenAPI definition file is hosted. More info here
There are total two API endpoint /multi-line-chart and /stacked-column-chart for our 2 charts. But support only GET request.
I encourage you to checkout the specification for more details on each and every key.
Two keys which are specific to Mojolicious are - x-mojo-name and x-mojo-to. x-mojo-to contains the name of controller and action to perform in case that particular endpoint is requested. Checkout Mojolicious::Plugin::OpenAPI::Guides::OpenAPIv3 for more details.

Using the plugin in Mojolicious App

Lets add the OpenAPI and SwaggerUi plugin in our application.
Inside MojoReactApp.pm

package MojoReactApp;
use Mojo::Base 'Mojolicious', -signatures;
use MojoReactApp::Model::Data;

# This method will run once at server start
sub startup ($self) {

    # Load configuration from config file
    my $config = $self->plugin('NotYAMLConfig');

    # Configure the application
    $self->secrets($config->{secrets});

    # Load the "api.json" specification
    # Can also be written as -
    # "OpenAPI" => {spec => $self->static->file("api.json")->path}

    $self->plugin(
        "OpenAPI" => {
            url => $self->home->rel_file("public/api.json")
        }
    );

    $self->plugin(
        SwaggerUI => {
            route => $self->routes()->any('api'),
            url => "/api/v1",
            title => "My Mojolicious App"
        }
    );

    # Helper to lazy initialize and store our model object
    $self->helper(
        model => sub ($c) {
            state $data = MojoReactApp::Model::Data->new();
            return $data;
        }
    );

    # Router
    my $r = $self->routes;

    # Normal route to controller
    $r->get('/')->to(template => 'home/welcome');
}

1;

In case you want to use .yaml format, you can use that instead of .json

$self->plugin(
        "OpenAPI" => {
            url => $self->home->rel_file("public/api.yaml")
        }
    );

Here I have just used the OpenAPI and SwaggerUI plugin.
We are using the url to load the specification which have created before.
SwaggerUI is just a fancy thing I am using. There is no such limitation to used it for creating the API.

route - The ui will be available on http://{hostname}:{port}/api
url - Url for the specification. In api.json inside server section we have written the url as /api/v1 meaning our specification is available under that path. Hence we have given the same path here.
Please have a look at the options for the meaning of each parameters.
If case you noticed, we have created just one route / which will just show the home page. Since we are crating Rest API's and all the other endpoint are already taken care in api.json we don't have to add it here.
Our directory structure is also simpler.

📦mojo_react_app
┣ 📂etc
┃ ┣ 📜input_multi_line_chart_data.json
┃ ┗ 📜input_stacked_clustered_column_chart.json
┣ 📂lib
┃ ┣ 📂MojoReactApp
┃ ┃ ┣ 📂Controller
┃ ┃ ┃ ┣ 📜ColumnCharts.pm
┃ ┃ ┃ ┗ 📜LineCharts.pm
┃ ┃ ┗ 📂Model
┃ ┃ ┃ ┗ 📜Data.pm
┃ ┗ 📜MojoReactApp.pm
┣ 📂public
┃ ┗ 📜api.json
┣ 📂script
┃ ┗ 📜mojo_react_app
┣ 📂t
┃ ┗ 📜basic.t
┣ 📂templates
┃ ┣ 📂home
┃ ┃ ┗ 📜welcome.html.ep
┃ ┗ 📂layouts
┃ ┃ ┗ 📜default.html.ep
┗ 📜mojo_react_app.yml

The public section only contains api.json. There is no need of css and js as we are creating only API. Even you can remove the whole template section also. I have just added it for / endpoint. The welcome template is just some minor adjustment to default one.

% layout 'default';
% title 'Welcome';
<h2>Hello there</h2>
<p>
  This is the home page.
  For API <%= link_to 'click here' => '/api' %>.
</p>

Since we are decided to created 2 charts, we have 2 json file in etc dir.
Each one has there own controller.
Our Model is almost similar to previous article with minor modification.

package MojoReactApp::Model::Data;

use strict;
use warnings;
use experimental qw(signatures);
use Mojo::JSON qw(decode_json);

sub new ($class) {
    my $self = {};
    bless $self, $class;
    return $self;
}

sub _read_json_file ($self, $json_file) {
    open(my $in, '<', $json_file) or $self->app->log->error("Unable to open file $json_file : $!");
    my $json_text = do { local $/ = undef; <$in>; };
    close($in) or $self->app->log->error("Unable to close file : $!");

    my $config_data = decode_json($json_text);
    return $config_data;
}

sub get_line_data ($self) {
    my $data_in_json = $self->_read_json_file("etc/input_multi_line_chart_data.json");

    return $data_in_json;
}

sub get_column_data ($self) {
    my $data_in_json = $self->_read_json_file("etc/input_stacked_clustered_column_chart.json");

    return $data_in_json;
}

1;

Controller

Lets create our 2 controller.
Inside lib\MojoReactApp\Controller\ColumnCharts.pm

package MojoReactApp::Controller::ColumnCharts;
use Mojo::Base 'Mojolicious::Controller', -signatures;
use Mojo::JSON qw(encode_json);

sub get_stacked_column_chart ($self) {

    # Do not continue on invalid input and render a default 400 error document.
    my $app = $self->openapi->valid_input or return;

    my $data_in_json = $app->model->get_column_data();

    # $output will be validated by the OpenAPI spec before rendered
    my $output = {chart_data => $data_in_json};
    $app->render(openapi => $output);
}

1;

Simple and easy!!. Here we are validating the input in $self->openapi->valid_input. After that we are getting the data from model and just returning that as the response inside the hash.

Similarly for lib\MojoReactApp\Controller\LineCharts.pm

package MojoReactApp::Controller::LineCharts;
use Mojo::Base 'Mojolicious::Controller', -signatures;
use Mojo::JSON qw(encode_json);

sub get_multi_line_chart ($self) {

    # Do not continue on invalid input and render a default 400 error document.
    my $app = $self->openapi->valid_input or return;

    my $data_in_json = $app->model->get_line_data();

    # $output will be validated by the OpenAPI spec before rendered
    my $output = {chart_data => $data_in_json};
    $app->render(openapi => $output);
}

1;

Exact similar to previous one, except we are calling different model function here.

Lets run it and see the output.
Hitting 'http://localhost:3000/'

You can see your home page which get generated by the welcome template. Click on the link or hit 'http://localhost:3000/api' in browser, you can see the shiny Swagger UI.

We can also see the 2 endpoint which you have created.
We can see all the things which we have written in api.json
This swagger ui is ultimately using that specification and generating this page.
Hit 'http://localhost:3000/api/v1' and you can see the specifications also.

Now lets try to do a GET request on one of the endpoint. You can use the SwaggerUI for this or a normal curl request or directly hit the endpoint from browser or use some 3rd party tool like Postman , all will work.
SwaggerUI will just give you some documentation about usage and the expected response.

Let hits the /multi-line-chart endpoint. In SwaggerUi page, click on 'Try it out' and 'execute' it.
You can see the response in JSON format.

We can see there is equivalent curl request already prepared which we can also use to get the same response.
Also in 'Requested URL' we can see ultimately it is calling the '/api/v1/multi-line-chart' which you can hit from browser and see the exact result.

With this our API development is done. Next we will be seeing how to use this API in React.js and create those charts in jsx.
Also, I have used morbo which is good for development. But for production we need to make certain changes. I will talk about those (Docker, Makefile, Apache2/Nginx, uWSGI/Plack and hypnotoad) in a different section.

The above example is available at github.

There is also a good tutorial available on Mojolicious blog - A RESTful API with OpenAPI

Perl onion logo taken from here
Mojolicious logo taken from here
OpenAPI logo taken from here

Data visualization: Using amCharts in React.js with Mojo

Gaurav Rai — Sat, 17 Apr 2021 17:37:18 +0000

In my previous article, I talked about using amCharts library with Perl Mojolicious. Today we will looking at creating the similar chart with React.js instead of plain JavaScript. I will keep it short since we already talked about it previously and will be reusing most of the code.

There are 2 ways we can use the react.js -

Without JSX (using <script> tag)
With JSX

JSX stand for JavaScript XML. It allow you to easily write HTML in react.
For now we will take the baby step and start without JSX.

Creating the data config

We will use the exact same example as in previous article and try to create a multi line chart.

{
    "title": "Number of automobiles sold per day by manufacturer",
    "label": {
        "domainAxis": "Date",
        "rangeAxis": "Numbers of automobiles sold"
    },
    "data": [
        {
            "Date": "2020-04-15",
            "Honda": 10,
            "Toyota": 20,
            "Ford": 6,
            "Renault": 16
        },
        {
            "Date": "2020-04-16",
            "Honda": 3,
            "Toyota": 15,
            "Ford": 19,
            "Renault": 10
        },
        {
            "Date": "2020-04-17",
            "Honda": 5,
            "Toyota": 8,
            "Ford": 12,
            "Renault": 6
        },
        {
            "Date": "2020-04-18",
            "Honda": 9,
            "Toyota": 10,
            "Ford": 4,
            "Renault": 12
        }
    ]
}

Creating the mojo app

The version I am using for this article is 9.14.

$  mojo generate app MojoReactApp

This command will generate a example application with proper directory structure for a MVC application and mentioned previously.

Now go inside the dir and try to run this app.

$ morbo ./script/mojo_app
Web application available at http://127.0.0.1:3000

Open the browser and hit http://localhost:3000/ and you can see the welcome page.

The rest of step is exact similar to mention in 'Creating the mojo app' section in previous article. So I will not board you by repeating it again. We will directly see the react part.

Adding React.js to app

We will update the default.html.ep to include the react.js

<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8" />
        <title><%= title %></title>

        %= content 'head'
    </head>
    <body>
        <div>
            %= content
        </div>
        <script src="https://unpkg.com/react@17/umd/react.production.min.js" crossorigin></script>
        <script src="https://unpkg.com/react-dom@17/umd/react-dom.production.min.js" crossorigin></script>

        %= content 'end'
    </body>
</html>

We are using the production minified version. You can also use the development version also for debugging purpose.
I have added react.js on layout template as we will be using it all our web pages.

In multi_line_chart.html.ep

% layout 'default';
% title 'Charts';

% content_for 'head' => begin
    <link rel="stylesheet" type="text/css" href="css/charts.css">
% end

<div id="root"></div>

% content_for 'end' => begin
    %= javascript "https://cdn.amcharts.com/lib/4/core.js"
    %= javascript "https://cdn.amcharts.com/lib/4/charts.js"
    %= javascript "https://cdn.amcharts.com/lib/4/themes/animated.js"

    %= javascript "js/multi_line_chart.js"

    %= javascript begin 
        var domContainer = document.getElementById("root");
        createMultiLineChart(domContainer, <%== $chart_data %>);
    % end
% end

We are geeting the $chart_data form create_multi_line_chart in lib\MojoReactApp\Controller\Charts.pm when the template get rendered.
Lets update the public/js/multi_line_chart.js to make it a React Component.

"use strict";

// React without JSX

const e = React.createElement;

class MultiLineChart extends React.Component {
    constructor(props) {
        super(props);
        this.state = {
            chartId: this.props.chartId,
            chartData: this.props.data,
        };
    }

    createSeries = (chart, axis, field, name) => {
        // Create series
        var series = chart.series.push(new am4charts.LineSeries());
        series.dataFields.dateX = "Date";
        series.dataFields.valueY = field;
        //series.dataFields.categoryX = "Date";
        series.strokeWidth = 2;
        series.xAxis = axis;
        series.name = name;
        series.tooltipText = "{name}: [bold]{valueY}[/]";
        //series.fillOpacity = 0.8;

        // For curvey lines
        series.tensionX = 0.8;
        series.tensionY = 1;

        // Multiple bullet options - circle, triangle, rectangle etc.
        var bullet = series.bullets.push(new am4charts.CircleBullet());
        bullet.fill = new am4core.InterfaceColorSet().getFor("background");
        bullet.fillOpacity = 1;
        bullet.strokeWidth = 2;
        bullet.circle.radius = 4;

        return series;
    };

    createChart = (chart) => {
        // Increase contrast by taking evey fouth color
        chart.colors.step = 4;
        //chart.hiddenState.properties.opacity = 0;             // this creates initial fade-in

        // Add title to chart
        var title = chart.titles.create();
        title.text = this.state.chartData["title"];
        title.fontSize = 25;
        title.marginBottom = 15;

        chart.data = this.state.chartData["data"];

        // Create axes - for normal Axis
        // var categoryAxis = chart.xAxes.push(new am4charts.CategoryAxis());
        // categoryAxis.dataFields.category = "Date";
        // categoryAxis.renderer.grid.template.location = 0;

        // Create axes - for Date Axis
        var dateAxis = chart.xAxes.push(new am4charts.DateAxis());
        //dateAxis.dataFields.category = "Date";
        dateAxis.renderer.grid.template.location = 0;
        dateAxis.renderer.minGridDistance = 50;
        dateAxis.title.text = this.state.chartData["label"]["domainAxis"];

        var valueAxis = chart.yAxes.push(new am4charts.ValueAxis());
        //valueAxis.renderer.line.strokeOpacity = 1;
        //valueAxis.renderer.line.strokeWidth = 2;
        valueAxis.title.text = this.state.chartData["label"]["rangeAxis"];

        //var single_data_item = this.state.chartData["data"][0];
        var series1 = this.createSeries(chart, dateAxis, "Toyota", "Toyota");
        var series2 = this.createSeries(chart, dateAxis, "Ford", "Ford");
        var series3 = this.createSeries(chart, dateAxis, "Honda", "Honda");
        var series4 = this.createSeries(chart, dateAxis, "Renault", "Renault");

        // Add legend
        chart.legend = new am4charts.Legend();

        // Add cursor
        chart.cursor = new am4charts.XYCursor();
        chart.cursor.xAxis = dateAxis;

        // Add scrollbar
        chart.scrollbarX = new am4core.Scrollbar();

        // Add export menu
        chart.exporting.menu = new am4core.ExportMenu();
    };

    componentDidMount() {
        am4core.useTheme(am4themes_animated);
        const chart = am4core.create(this.state.chartId, am4charts.XYChart);
        this.createChart(chart);
        this.chart = chart;
    }

    componentWillUnmount() {
        if (this.chart) {
            this.chart.dispose();
        }
    }

    render() {
        return e("div", { id: this.state.chartId }, null);
    }
}

function createMultiLineChart(domContainer, chartData) {
    ReactDOM.render(
        e(MultiLineChart, { chartId: "chartdiv", data: chartData }, null),
        domContainer
    );
}

We are calling the createMultiLineChart function from our template with parameters. The main point to know here is the state and lifecycle functions - componentDidMount and componentWillUnmount.
Since there are already plenty of documentation available, I encourage you to look into it. One of the place to learn the concept is react official docs- State and Lifecycle

If you look closely the rest of function definition is not much changes from the previous used javascript. We just wrapped it in react.

The final directory structure is -

📦mojo_react_app
┣ 📂etc
┃ ┗ 📜input_data.json
┣ 📂lib
┃ ┣ 📂MojoReactApp
┃ ┃ ┣ 📂Controller
┃ ┃ ┃ ┗ 📜Charts.pm
┃ ┃ ┗ 📂Model
┃ ┃ ┃ ┗ 📜Data.pm
┃ ┗ 📜MojoReactApp.pm
┣ 📂public
┃ ┣ 📂css
┃ ┃ ┗ 📜charts.css
┃ ┗ 📂js
┃ ┃ ┗ 📜multi_line_chart.js
┣ 📂script
┃ ┗ 📜mojo_react_app
┣ 📂t
┃ ┗ 📜basic.t
┣ 📂templates
┃ ┣ 📂charts
┃ ┃ ┗ 📜multi_line_chart.html.ep
┃ ┗ 📂layouts
┃ ┃ ┗ 📜default.html.ep
┣ 📜mojo_react_app.yml
┗ 📜README.md

Save it and try to hit 'http://localhost:3000' again. From the user perspective side nothing has changed, you will see the same output as before.

As I mentioned before we will starting with baby step.
The usage of the above mentioned way is very limited. You can use the react.js without jsx when your scope is small - where you have to make a website of few pages because here we are not using the full power of react.
To use the react.js with full potential and unleash its power you have to use jsx. We will be looking in to it our next article.

The above example is available at github.

Perl onion logo taken from here
Mojolicious logo taken from here
React logo taken from here
amCharts logo taken form here

Data visualization: Using amCharts with Perl and Mojo

Gaurav Rai — Sun, 11 Apr 2021 15:30:49 +0000

In my previous article, I talked about Chart::Plotly. Today we will looking at creating the similar chart using another javascript library amCharts.

I have the opportunity to work on both v3 and v4 of amCharts. v3 is currently in maintenance mode. v4 is rewritten in typescript. One good thing about the library is there are lot of documentation and examples available on there website. Also you can use it in plain Javascript or integrate them into various application frameworks - React, Angular2+, Ember, Vue.js etc.
Also you don't need to be javascript expert to use it. It is highly configurable. You can use any syntax for configuration - TypeScript/ES6, JavaScript or JSON. For more details have a look at there excellent documentation.

Without further delay lets get started.

Creating the data config

We will use the exact same example as in previous article and try to create a multi line chart. But this time we will tweak the data format a little bit.

{
    "title": "Number of automobiles sold per day by manufacturer",
    "label": {
        "domainAxis": "Date",
        "rangeAxis": "Numbers of automobiles sold"
    },
    "data": [
        {
            "Date": "2020-04-15",
            "Honda": 10,
            "Toyota": 20,
            "Ford": 6,
            "Renault": 16
        },
        {
            "Date": "2020-04-16",
            "Honda": 3,
            "Toyota": 15,
            "Ford": 19,
            "Renault": 10
        },
        {
            "Date": "2020-04-17",
            "Honda": 5,
            "Toyota": 8,
            "Ford": 12,
            "Renault": 6
        },
        {
            "Date": "2020-04-18",
            "Honda": 9,
            "Toyota": 10,
            "Ford": 4,
            "Renault": 12
        }
    ]
}

The reason we are using this format is because amCharts use array of objects to create chart where each object in the array represents a single data point. More info here.
We can use any data format but ultimately we have to convert it as array of object before creating chart which doesn't make sense (especially if you are doing it at time of page loading). So why not to create the data in the format which we can use easily.

Creating the mojo app

We we will be using the Mojolicious framework for server side. You can install it using single command as mentioned on website -

$ curl -L https://cpanmin.us | perl - -M https://cpan.metacpan.org -n Mojolicious

It also have excellent documentation. Have a look at it to learn more.
The version I am using for this article is 9.14.
We will go ahead and create an app from command line.

$  mojo generate app MojoApp

This command will generate a example application with proper directory structure for a MVC application. Easy peasy

📦mojo_app
┣ 📂lib
┃ ┣ 📂MojoApp
┃ ┃ ┗ 📂Controller
┃ ┃ ┃ ┗ 📜Example.pm
┃ ┗ 📜MojoApp.pm
┣ 📂public
┃ ┗ 📜index.html
┣ 📂script
┃ ┗ 📜mojo_app
┣ 📂t
┃ ┗ 📜basic.t
┣ 📂templates
┃ ┣ 📂example
┃ ┃ ┗ 📜welcome.html.ep
┃ ┗ 📂layouts
┃ ┃ ┗ 📜default.html.ep
┗ 📜mojo_app.yml

Now go inside the dir and try to run this app.

$ morbo ./script/mojo_app
Web application available at http://127.0.0.1:3000

Open the browser and hit http://localhost:3000/ and you can see the welcome page.
If you open and look into MojoApp.pm you can see - get request on /(home page) is redirected to example controller (Example.pm) and function welcome is called inside that controller to fulfill the request. You can also see the template example/welcome.html.ep is rendered inside that function which you are seeing when you hit the http://localhost:3000/

We will be adding/modifying some parts of this dir structure to suit our need.

We will be creating a 'mojo_app/etc/' dir to put our 'input_data.json' created previously.
We will be renaming the default controller example to something meaningful
Also we will be modifying the layouts\default.html.ep template.
And we will be adding amCharts javascript library in template.

Update MojoApp.pm with the following changes in startup-

    # Normal route to controller
    $r->get('/')->to('charts#create_multi_line_chart');

Create new or rename Example.pm to Charts.pm in Controller and update it with -

package MojoApp::Controller::Charts;
use Mojo::Base 'Mojolicious::Controller', -signatures;
use Mojo::JSON qw(decode_json encode_json);

sub read_json_file ($self, $json_file) {

    open(my $in, '<', $json_file) or $self->app->log->error("Unable to open file $json_file : $!");
    my $json_text = do { local $/ = undef; <$in>; };
    close($in) or $self->app->log->error("Unable to close file : $!");

    my $config_data = decode_json($json_text);
    return $config_data;
}

sub create_multi_line_chart ($self) {
    my $data_in_json = $self->read_json_file( "etc/input_data.json");

    $self->render(template => 'charts/multi_line_chart', chart_data => encode_json($data_in_json));
}

1;

Here we are just reading the input json file and rendering the template with the chart data. Please note that create_multi_line_chart will be called at every load of page. Here I am reading the file every time. You can optimize it by reading it once at the start or caching it in case your input data doesn't change that often.
The JSON file is just an example. You can get this data from a database also.
Since we are talking about MVC framwork, why not move this data logic to Model.
Create lib\MojoApp\Model\Data.pm and update it with

package MojoApp::Model::Data;

use strict;
use warnings;
use experimental qw(signatures);
use Mojo::JSON qw(decode_json);

sub new ($class) {
    my $self = {};
    bless $self, $class;
    return $self;
}

sub _read_json_file ($self, $json_file) {
    open(my $in, '<', $json_file) or $self->app->log->error("Unable to open file $json_file : $!");
    my $json_text = do { local $/ = undef; <$in>; };
    close($in) or $self->app->log->error("Unable to close file : $!");

    my $config_data = decode_json($json_text);
    return $config_data;
}

sub get_data ($self) {
    my $data_in_json = $self->_read_json_file("etc/input_data.json");

    return $data_in_json;
}

1;

Again, you can connect to DB and generate this data. For simplicity I am just getting the data from JSON file. (This data is actually generated from CouchDB :P).
Lets update our startup in MojoApp.pm

use MojoApp::Model::Data;

sub startup ($self) {

...
    # Helper to lazy initialize and store our model object
    $self->helper(
        model => sub ($c) {
            state $data = MojoApp::Model::Data->new();
            return $data;
        }
    );
...

}

Lets remove the extra thing from controller Charts.pm and use this helper.

package MojoApp::Controller::Charts;
use Mojo::Base 'Mojolicious::Controller', -signatures;
use Mojo::JSON qw(encode_json);

sub create_multi_line_chart ($self) {
    my $data_in_json = $self->model->get_data();

    $self->render(template => 'charts/multi_line_chart', chart_data => encode_json($data_in_json));
}

1;

We updated the controller to use the model for data and render the template.
Now lets go to template section and update/create a folder name charts in which we will be creating template multi_line_chart.html.ep.
Also lets update the default.html.ep template a little bit.

<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8" />
        <title><%= title %></title>

        %= content 'head'
    </head>
    <body>
        <div>
            %= content
        </div>
        %= content 'end'
    </body>
</html>

This is our layout template and we will be using it at our every page throughout the website. There are different placeholders where we will genrating data for different pages. For more details have a look at Mojolicious::Guides::Rendering and Mojo::Template
In multi_line_chart.html.ep

% layout 'default';
% title 'Charts';

% content_for 'head' => begin
    <link rel="stylesheet" type="text/css" href="css/charts.css">
% end

<div id="chartdiv"></div>

% content_for 'end' => begin
    %= javascript "https://cdn.amcharts.com/lib/4/core.js"
    %= javascript "https://cdn.amcharts.com/lib/4/charts.js"
    %= javascript "https://cdn.amcharts.com/lib/4/themes/animated.js"

    %= javascript "js/multi_line_chart.js"

    %= javascript begin
        createMultiLineChart(<%== $chart_data %>);
    % end
% end

In simple language, we are saying here - use the default.html.ep template, update the title of the page to 'Charts', append the head section with the css for this page, in the page body create a 'div' with 'id' chartdiv and in the end of the body add the mentioned javascripts file.
The $chart_data which we are using in javascript, gets passed from server side while rendering the template in create_multi_line_chart method. It is encoded in JSON for which we are decoding on client side.
The top 3 javascript included are amCharts library.
Now lets create charts.css and multi_line_chart.js which we are referencing here. These will be automatically served from 'public' dir.
In public/css/charts.css

#chartdiv {
    width: 850px;
    height: 550px;
}

Its very small css where we just setting the dimensions of the chart.
In public/js/multi_line_chart.js

function createSeries(chart, axis, field, name) {
    // Create series
    var series = chart.series.push(new am4charts.LineSeries());
    series.dataFields.dateX = "Date";
    series.dataFields.valueY = field;
    series.strokeWidth = 2;
    series.xAxis = axis;
    series.name = name;
    series.tooltipText = "{name}: [bold]{valueY}[/]";

    var bullet = series.bullets.push(new am4charts.CircleBullet());

    return series;
}

function createMultiLineChart(chartData) {
    // Themes begin
    am4core.useTheme(am4themes_animated);

    var chart = am4core.create("chartdiv", am4charts.XYChart);

    // Increase contrast by taking every second color
    chart.colors.step = 2;
    // Add title to chart
    var title = chart.titles.create();
    title.text = chartData["title"];

    // Add data to chart
    chart.data = chartData["data"];

    // Create axes
    var dateAxis = chart.xAxes.push(new am4charts.DateAxis());
    dateAxis.title.text = chartData["label"]["domainAxis"];

    var valueAxis = chart.yAxes.push(new am4charts.ValueAxis());
    valueAxis.title.text = chartData["label"]["rangeAxis"];

    //var single_data_item = chartData["data"][0];
    var series1 = createSeries(chart, dateAxis, "Toyota", "Toyota");
    var series2 = createSeries(chart, dateAxis, "Ford", "Ford");
    var series3 = createSeries(chart, dateAxis, "Honda", "Honda");
    var series4 = createSeries(chart, dateAxis, "Renault", "Renault");

    // Add legend
    chart.legend = new am4charts.Legend();

    // Add cursor
    chart.cursor = new am4charts.XYCursor();
    chart.cursor.xAxis = dateAxis;

    // Add scrollbar
    chart.scrollbarX = new am4core.Scrollbar();

    // Add export menu
    chart.exporting.menu = new am4core.ExportMenu();
}

I have added the comments for the description. You can look at reference and xy-chart for more details.
The function createMultiLineChart created here is the one which we are calling in multi_line_chart.html.ep.

Save it and refresh the home page.

I have tried to use mostly the default configuration. The screenshot above is not doing the justice to the actual dynamic chart. For that you have to run and see it for yourself.

Now lets try to modify the public/js/multi_line_chart.js with some more configuration. As I mentioned before it is highly configurable and its difficult to cover each and every thing so I will try to cover whatever I can.

function createSeries(chart, axis, field, name) {
    // Create series
    var series = chart.series.push(new am4charts.LineSeries());
    series.dataFields.dateX = "Date";
    series.dataFields.valueY = field;
    //series.dataFields.categoryX = "Date";
    series.strokeWidth = 2;
    series.xAxis = axis;
    series.name = name;
    series.tooltipText = "{name}: [bold]{valueY}[/]";
    //series.fillOpacity = 0.8;

    // For curvey lines
    series.tensionX = 0.8;
    series.tensionY = 1;

    // Multiple bullet options - circle, triangle, rectangle etc.
    var bullet = series.bullets.push(new am4charts.CircleBullet());
    bullet.fill = new am4core.InterfaceColorSet().getFor("background");
    bullet.fillOpacity = 1;
    bullet.strokeWidth = 2;
    bullet.circle.radius = 4;

    return series;
}

function createMultiLineChart(chartData) {
    // Themes begin
    am4core.useTheme(am4themes_animated);

    var chart = am4core.create("chartdiv", am4charts.XYChart);

    // Increase contrast by taking every second color
    chart.colors.step = 3;
    //chart.hiddenState.properties.opacity = 0; // this creates initial fade-in

    // Add title to chart
    var title = chart.titles.create();
    title.text = chartData["title"];
    title.fontSize = 25;
    title.marginBottom = 15;

    chart.data = chartData["data"];

    // Create axes - for normal Axis
    // var categoryAxis = chart.xAxes.push(new am4charts.CategoryAxis());
    // categoryAxis.dataFields.category = "Date";
    // categoryAxis.renderer.grid.template.location = 0;

    // Create axes - for Date Axis
    var dateAxis = chart.xAxes.push(new am4charts.DateAxis());
    //dateAxis.dataFields.category = "Date";
    dateAxis.renderer.grid.template.location = 0;
    dateAxis.renderer.minGridDistance = 50;
    dateAxis.title.text = chartData["label"]["domainAxis"];

    var valueAxis = chart.yAxes.push(new am4charts.ValueAxis());
    //valueAxis.renderer.line.strokeOpacity = 1;
    //valueAxis.renderer.line.strokeWidth = 2;
    valueAxis.title.text = chartData["label"]["rangeAxis"];

    var series1 = createSeries(chart, dateAxis, "Toyota", "Toyota");
    var series2 = createSeries(chart, dateAxis, "Ford", "Ford");
    var series3 = createSeries(chart, dateAxis, "Honda", "Honda");
    var series4 = createSeries(chart, dateAxis, "Renault", "Renault");

    // Add legend
    chart.legend = new am4charts.Legend();

    // Add cursor
    chart.cursor = new am4charts.XYCursor();
    chart.cursor.xAxis = dateAxis;

    // Add scrollbar
    chart.scrollbarX = new am4core.Scrollbar();

    // Add export menu
    chart.exporting.menu = new am4core.ExportMenu();
}

Now we will try to see the output again -

Somewhat better than the previous one. The three dots on the top right corner gives to more options to interact like - downloading the image as png or svg, getting the data in JSON or CSV format, printing the chart etc.
Also there are certain plugins available which you can use to enhance the experience. More details at Plugins.

As I mentioned there are lot of config options and I haven't
covered all of them. But I will try to cover it in my next installment where I will create the same chart in React.js using Typescript/ES6. Also the above js file can be modified a little bit to make it generalized for any type of multi line chart(especially the 'createSeries' call). I will leave that as an exercise.

The above example is available at github.

Perl onion logo taken from here
Mojolicious logo taken from here
amCharts logo taken form here

Data visualization: Creating charts from perl using plotly.js (Chart::Plotly)

Gaurav Rai — Sun, 28 Mar 2021 17:08:01 +0000

In my previous article, I mentioned I will talking more about Chart::Plotly. Today is that day. So without further ado lets get started.
As mentioned in Chart::Plotly - Chart::Plotly helps you to html/javascript charts from perl data using javascript library plotly.js. The result is a file that you could see in your favorite browser. Also you can save the image locally and use it as per your need.

Plotly.js is very powerful which offers a lot of feature and wide variety of charts. It is built on top of d3.js and stack.gl. The charts are fully customizable. You can provide all the configurations in declarative format (JSON) and it takes care of the rest. You don't need to be a javascript expert to use this as all the complexity is taken care of. There are lot of examples available and plenty of documentation for your help. Also it support statistical and scientific charts.(You can use Chart::Plotly with PDL too.)

So lets get started.

Creating the data config

We will use the exact same example as in previous article and try to create a multi line chart.

{
    "title": "Number of automobiles sold per day by manufacturer",
    "domainAxis": {
        "data": ["2020-04-15", "2020-04-16", "2020-04-17", "2020-04-18"],
        "label": "Date"
    },
    "rangeAxis": {
        "lines": {
            "line1": {
                "data": [10,3,5,9],
                "legendName": "Honda"
            },
            "line2": {
                "data": [20,15,8,10],
                "legendName": "Toyota"
            },
            "line3": {
                "data": [6,19,12,4],
                "legendName": "Ford"
            },
            "line4": {
                "data": [16,10,6,12],
                "legendName": "Renault"
            }
        },
        "label": "Numbers of automobiles sold"
    }
}

This is the config which contains the data for multi line chart.
domainAxis(x-axis) is a date axis while rangeAxis(y-axis) have 4 lines.

Directory structure

Our directory structure will be simple. input for input data, output for generated chart, lib for perl module.

┣ 📂input
┃ ┗ 📜input_data.json
┣ 📂lib
┃ ┗ 📜CreateLineCharts.pm
┣ 📂output
┣ 📜multi_line_chart.pl
┗ 📜README.md

Creating the module

One good thing about this module is- the documentation of plotly.js is rqually applicable here. So even the documentation is little sparse in Chart::Plotly, the documentation in plotly.js comes to rescue and you can use the exact same names and keywords in your perl code without any issue. Also, similar examples are available in python version also on the plotly.js website. So if you understand little bit of any of these 3 languages it is a cakewalk for you in other one.

So, lets create our module.

package CreateLineCharts;
use strict;
use warnings;

# https://plotly.com/javascript/line-charts/
use Chart::Plotly 'show_plot';
use Chart::Plotly::Image 'save_image';
use Chart::Plotly::Plot;
use Chart::Plotly::Trace::Scatter;

sub generate_line_chart {
    my ($self, $chart_out_file, $chart_data) = @_;

    my $x_axis = $chart_data->{domainAxis};
    my $y_axis = $chart_data->{rangeAxis};

    my $plot = Chart::Plotly::Plot->new();
    foreach my $y_line (keys %{$y_axis->{lines}}) {
        my $scatter = Chart::Plotly::Trace::Scatter->new(
            x    => $x_axis->{data},
            y    => $y_axis->{lines}->{$y_line}->{data},
            name => $y_axis->{lines}->{$y_line}->{legendName}
         );
        $plot->add_trace($scatter);
    }

    # Opens the plot or plots in a browser locally
    show_plot($plot);

    # https://metacpan.org/pod/Chart::Plotly::Image#save_image
    save_image(
        file   => $chart_out_file,        # Referring to a local filesystem path
        plot   => $plot,
        width  => 1024,                   # Sets the image width
        height => 768,                    # Sets the image height
        engine => 'auto'
    );
}

Here we are just going through each line and creating a Chart::Plotly::Trace::Scatter object and adding it to plot.
There are 2 functions which I would like to point out -

show_plot function will open the generated chart in you favorite browser locally.
save_image will save the generated file locally in your local filesystem. This will be useful when you want to send this as mail attachment in automated fashion. You can use anyone of them based on your requirement or both.

Using the module

Let create our startup script to access this module for creating our chart.

#!/usr/bin/env perl

use strict;
use warnings;
use Cwd qw( abs_path );
use File::Basename qw( dirname );
use JSON;

BEGIN {
    $ENV{"SCRIPT_DIR"} = dirname(abs_path($0));
}

use lib $ENV{"SCRIPT_DIR"} . "/lib";
use CreateLineCharts;

my $chart_out_file = $ENV{"SCRIPT_DIR"} . "/output/lineChart.png";

sub read_json_file {
    my ($json_file) = @_;
    print "\nReading $json_file\n";

    open(my $in, '<', $json_file) or print "Unable to open file $json_file : $!";
    my $json_text = do { local $/ = undef; <$in>; };
    close($in) or print "\nUnable to close file : $!";

    my $config_data = decode_json($json_text);
    return $config_data;
}


sub main {
    my $data_in_json = read_json_file($ENV{"SCRIPT_DIR"} . "/input/input_data.json");

    my $chart = CreateLineCharts->new();
    $chart->generate_line_chart($chart_out_file, $data_in_json);

}

main;

We are reading the JSON data from input file and calling generate_line_chart of our module with it.

Running the script

Now just run the above script and see the output. Running the script will automatically open your favorite browser with the chart. The output will be generated in 2 places.

On the browser
Locally in output dir Both are similar chart. Of course, the browser one is dynamic and interactive chart and the local filesystem one is static. Also there are several other options available when you hover over the chart in browser. You can see them at top right corner(highlighted in yellow).

That's it. You have your multi line chart with minimal code. There are lot of configuration options available. The default generated chart is already good enough. In case you are interested in more options proceed ahead.

Bonus - If you noticed the url in your browser, you can see the .html file is created in temp location. Go to that location and try to open that file in your favorite editor(or you can inspect element and see there).

<!DOCTYPE html>
<head>
<meta charset="utf-8" />
</head>
<body>
<div id="1ca037ff-8fd4-11eb-bd22-b212bed9e091"></div>
<script src="https://cdn.plot.ly/plotly-1.52.2.min.js"></script>
<script>
Plotly.react(document.getElementById('1ca037ff-8fd4-11eb-bd22-b212bed9e091'),[{"y":[16,10,6,12],"x":["2020-04-15","2020-04-16","2020-04-17","2020-04-18"],"name":"Renault","type":"scatter"},{"x":["2020-04-15","2020-04-16","2020-04-17","2020-04-18"],"y":[20,15,8,10],"type":"scatter","name":"Toyota"},{"name":"Honda","type":"scatter","y":[10,3,5,9],"x":["2020-04-15","2020-04-16","2020-04-17","2020-04-18"]},{"x":["2020-04-15","2020-04-16","2020-04-17","2020-04-18"],"y":[6,19,12,4],"type":"scatter","name":"Ford"}]  );
</script>
</body>
</html>

The above template is generated by Chart::Plotly only(Precisly here). The div id gets randomly generated at each time. After that we are using Plotly.react to create a plot.

Some advanced config options

As I mentioned before plotly.js is powerful and highly configurable. Now lets take a look at those configuration.
In case you are not happy with default config you can change it as per your need. These configuration ranges from styling the plot lines, changing the layout, legend, marker, font, color, text anything you want.
Now lets add these configuration to our module.

package CreateLineCharts;
use strict;
use warnings;

# https://plotly.com/javascript/line-charts/
use Chart::Plotly 'show_plot';
use Chart::Plotly::Image 'save_image';
use Chart::Plotly::Plot;
use Chart::Plotly::Trace::Scatter;

use HTML::Show;

use JSON;
use Data::Dumper;

sub new {
    my ($class, @arguments) = @_;
    my $self = {@arguments};
    bless $self, $class;
    return $self;
}

sub _generate_plot_lines {
    my ($self, $plot, $x_line, $y_line) = @_;

    # https://metacpan.org/pod/Chart::Plotly::Trace::Scatter
    # https://plotly.com/javascript/reference/scatter/
    my $scatter = Chart::Plotly::Trace::Scatter->new(
        x    => $x_line->{data},
        y    => $y_line->{data},
        name => $y_line->{legendName},

        # mode => 'markers',                 # Add markers at data points instead of line
        marker => {
            symbol  => 'diamond',            # Default - circle
            size    => 8,                    # marker size(in px). Default - 6px
            opacity => 0.8,                  # Marker opacity (0-1)
            # color   => 'red'               # Sets the marker color
        },
        opacity => 0.8,
        # text => $x_line->{data}            # Extra text you want to show on mouse hover over all the data points.

        # https://metacpan.org/pod/Chart::Plotly::Trace::Scatter::Line
        line => {
            # width   => 3,                  # Width of line, Default: 2
            # color   => '#45b5c6',          # Color of the line
            shape     => "spline",           # Determines the line shape - one of("linear" | "spline" | "hv" | "vh" | "hvh" | "vhv"), Default: "linear"
            smoothing => 0.5,                # Used only if `shape` is set to "spline", Default: 1
            dash      => "solid",            # Dash style of line - ("solid", "dot", "dash", "longdash", "dashdot", or "longdashdot"). Default: "solid"
            simplify  => JSON::false,        # Simplifies lines by removing nearly-collinear points. Default: true
        }
    );
    return $scatter;
}

sub _add_layout {
    my ($self, $plot, $chart_title, $x_axis_title, $y_axis_title) = @_;
    $plot->layout(
        {
            title => $chart_title,
            # font  => {                          # Sets the global font
            #     family => "Open Sans",          # Default font - ""Open Sans", verdana, arial, sans-serif"
            #     size     => 14                  # default - 12px
            # },
            # https://plotly.com/javascript/legend/
            legend => {
                # orientation => "h",             # Sets the orientation of the legend('v' or 'h'). Default - v(vertical)
                # xanchor     => "left",          # Sets the legend's horizontal position anchor. "left", "center" or "right"
                # yanchor     => "bottom",        # Sets the legend's vertical position anchor. "top", "middle" or "bottom"
                # x           => 0,               # number between or equal to -2 and 3
                #                                 # Sets the x position (in normalized coordinates) of the legend.
                #                                 # Defaults to "1.02" for vertical legends and defaults to "0" for horizontal legends.
                # y           => -0.1,            # number between or equal to -2 and 3
                #                                 # Sets the y position (in normalized coordinates) of the legend.
                #                                 # Defaults to "1" for vertical legends, defaults to "-0.1" for horizontal legends on graphs w/o range sliders and defaults to "1.1" for horizontal legends on graph with one or multiple range sliders.

                bgcolor     => "#ffffff",         # Sets the legend background color . Defaults to `layout.paper_bgcolor`
                bordercolor => "#333333",         # Sets the color of the border enclosing the legend . Default - #444
                borderwidth => 1,                 # Sets the width (in px) of the border enclosing the legend. Default - 0
                font => {                         # Sets the font used to text the legend items.
                    size  => 14,
                    color => "#000000"            # Black
                },
                # title => {                      # Sets the title of the legend. Default - ""
                #     text => "Legend",
                #     font => {size => 14, color => "black"},
                #     side => "top"               # Location of legend's title with respect to the legend items
                # }
            },

            # showlegend => JSON::false,                # Whether you want to display the legend on not. Default - true
            # https://plotly.com/javascript/axes/
            # https://plotly.com/javascript/tick-formatting/
            xaxis => {
                title      => $x_axis_title,            # Text label for x-axis
                type       => "-",                      # x-axis type
                automargin => JSON::true,
                linecolor  => "#333333",                # Sets color of X-axis line
                # titlefont  => {color => '#0066ff'},   # Title font formating
                # tickfont   => {color => '#0066ff'},
                zeroline   => JSON::true,               # Show zero line or not
                zerolinecolor => '#cccccc',             # Assign specific color to zero line
                zerolinewidth => 4,

                # showgrid => JSON::false               # Removes X-axis grid lines
                # rangeslider => { visible => JSON::false },
                # gridcolor   => '#bfbfbf',
                # gridwidth   => 1,
                # tickformat => "YYYY-MM-DD",           # d3-format specifier. If empty or "" plotly will attempt to guess format
                # dtick       => 1                      # step in-between ticks
            },
            yaxis => {
                title      => $y_axis_title,
                tickformat => "",                       # d3-format specifier. If empty or "" plotly will attempt to guess format.
                automargin => JSON::true,
                linecolor  => "#333333",                # Sets color of Y-axis line
                # titlefont  => {color => '#0066ff'},
                # tickfont   => {color => '#0066ff'},
                rangemode  => "tozero",                 # Forced to start from zero. Default - "normal"
                automargin => JSON::true,
                zeroline   => JSON::true,               # Show zero line or not
                # showgrid => JSON::false               # Removes Y-axis grid lines
                # side => "left",                       # Location of y-axis. "left" or "right"
                # gridcolor => '#bfbfbf',               # Assign specific color to grid 
                # gridwidth => 1,
                # dtick => 1                            # step in-between ticks
            },
            paper_bgcolor => "#ffffff",                 # Sets the background color of the paper where the graph is drawn. Default - #FFF
            plot_bgcolor => "#ffffff",                  # Sets the background color of the plotting area in-between x and y axes.
            margin => {                                 # Default(in px): left(80), right(80), top(100), bottom(80)
                'l' => 50,
                'r' => 50,
                't' => 50,
                'b' => 50
            },
            width  => 1000,                             # Sets the plot's width. Default - 700px
            height => 750,                              # Sets the plot's height. Default - 450px
        }
    );
}

sub _add_config {
    my ($self, $plot) = @_;
    # https://plotly.com/javascript/configuration-options/
    my %config = (
        scrollZoom => JSON::true,                   # mousewheel or two-finger scroll zooms the plot
        editable   => JSON::true,                   # In editable mode, users can edit the chart title, axis labels and trace names in the legend
        # staticPlot => JSON::true,                 # Create a static chart
        toImageButtonOptions => {                   # Customize Download Plot Options
            format   => 'svg',                      # one of png, svg, jpeg, webp. Default - png
            filename => 'multi_line_chart',         # Default name - newplot
            height   => 550,
            width    => 800,
            scale    => 1                           # Multiply title/legend/axis/canvas sizes by this factor
        },
        # displayModeBar => JSON::true,             # Force The Modebar at top to Always Be Visible.
                                                    # By default, the modebar is only visible while the user is hovering over the chart.
                                                    # Making it 'false' will never Display The Modebar
        modeBarButtonsToRemove => ['sendDataToCloud'],          # Delete some buttons from the modebar

        showLink        => JSON::true,                          # Display the `Edit Chart` Link
        plotlyServerURL => "https://chart-studio.plotly.com",   # Here you can directly edit your chart in browser
        linkText        => 'Edit chart in chart studio',

        # locale        => 'fr',                    # Change the Default Locale.
                                                    # More info - https://github.com/plotly/plotly.js/blob/master/dist/README.md#to-include-localization
        displaylogo      => JSON::false,            # Hide the Plotly Logo on the Modebar
        # responsive     => JSON::true,             # Responsive to window size
        # doubleClickDelay => 1000,                 # maximum delay between two consecutive clicks to be interpreted as a double-click in ms (default 300 ms)
    );
    $plot->config(\%config);
}

sub generate_line_chart {
    my ($self, $chart_out_file, $chart_data) = @_;

    my $x_axis = $chart_data->{domainAxis};
    my $y_axis = $chart_data->{rangeAxis};

    my $plot = Chart::Plotly::Plot->new();
    foreach my $y_line (keys %{$y_axis->{lines}}) {
        my $scatter = $self->_generate_plot_lines($plot, $x_axis, $y_axis->{lines}->{$y_line});
        $plot->add_trace($scatter);
    }

    $self->_add_layout($plot, $chart_data->{title}, $x_axis->{label}, $y_axis->{label});

    $self->_add_config($plot);

    my $html = $plot->html(
        div_id => 'my_div_id',                          # Id of the div, in which you want your chart to be embedded
        load_plotly_using_script_tag => 'embed'         # Can be : 1 or cdn, embed, module_dist.
                                                        # * By default, it is 1(cdn) meaning it will load plotly.js using cdn link.
                                                        # * 'embed' will take it from the plotly.js that is shipped wth Chart::Plotly and paste it within <script> tag.
                                                        # * 'module_dist' is similar to 'embed' except it will provide the source in <script src="file://">.
                                                        # Please note that using 'module_dist' will show the actual location of the file on your location machine(e.g. /usr/lib/perl/5.30/site/lib/Chart-Plotly/plotly.js/plotly.min.js).
                                                        # So, beware of this as you will be showing this location in your browser
    );

    # Returns the structure suitable to serialize to JSON corresponding to the plot
    # print Dumper($plot->TO_JSON);

    # Returns the plot serialized in JSON . Not suitable to use in nested structures
    # print Dumper($plot->to_json_text);

    # Opens the plot or plots in a browser locally.
    # Both are equal. In second statement we are just updating the div id with user defined one and determining how to load plotly.js
    show_plot($plot);
    HTML::Show::show($html);

    # https://metacpan.org/pod/Chart::Plotly::Image#save_image
    save_image(
        file   => $chart_out_file,          # Referring to a local filesystem path
        format => "png",                    # Supported formats are png, jpeg, webp, svg, pdf, eps.
                                            # By default it's inferred from the specified file name extension
        scale => 1,                         # Multiply title/legend/axis/canvas sizes by this factor
        plot  => $plot,
        # width  => 1024,                   # Sets the image width
        # height => 768,                    # Sets the image height
        engine => 'auto'
    );
}

1;

I know its overwhelming. But, I spend quite a few time just to know what each configuration does. I have added those as comments with do and don't in front or top of those config options.
I have gone through ploty.js doc and tried those in perl version. To my surprise each and every one of them worked in perl world. As I mentioned before the documentation of Chart::Plotly can be improved and this example can help someone working on it for first time.
Also you can look at examples section of Chart::Plotly for more examples of different charts.

Now, lets run and see the output again. There will be 3 output generated. Two in browser and one in local dir.
Some thing I would like to point.

show_plot($plot) - The first output in browser is generated form here.
HTML::Show::show($html) - The second output in browser is generated from here. This statement is similar to first one with only 2 difference. One being the div_id which we are using as user defined. Second being, how to get plotly.js for generating chart(through cdn or the one shipped with Chart::Plotly etc). Inspect element and check the difference between this and first point, you will understand. Below you can see the 'div' id is user defined and <script> tag contains plotly.js as embeded instead of cdn link.
save_image - The third chart generated locally is from this function.
$plot->TO_JSON - Returns the structure suitable to serialize to JSON corresponding to the plot. You can dump it to see what is exactly happening inside chart. You will see all the config and set options.
$plot->config(\%config) - Extra config for the chart. These are mostly specific to chart which you are seeing in browser.
JSON::false and JSON::true - As perl doesn't have boolean, I am using this for setting the config option as false or true depending on the javascript false or true. You can use any other JSON module for similar purpose if you want.

As mentioned, I have only created multi line chart but there are multiple options available each one having there one configuration. You can choose from them.
I hope you got some clarity on how to use this module. So, next time when you got any opportunity to create chart you know what to use.

The above example is also available at github.

Perl Onion logo taken from here
Plotly logo taken from here
Chart image in cover taken from here

Data visualization: Creating charts using Perl (Chart::Clicker)

Gaurav Rai — Fri, 19 Mar 2021 17:56:14 +0000

As usual there are multiple ways to create beautiful charts in Perl.

We will be using Chart::Clicker in this post.

But let me get few thing clear before that. Both Chart::Clicker and GD::Graph hasn't been updated in last 4+ years. Especially for GD::Graph there are multiple issue opened and not sure about current status.

Chart::Plotly is something which I am more interested in right now. It is based on plotly.js.
I also encourage you to checkout Dash which is built on top of Plotly.js, React and Flask for building ML & data science web apps.
There is something equivalent in Perl world also from same author(Chart::Plotly) - Dash which will be using Mojolicious or Dancer2. But right now it is in experimental stage and still under active development. I hope it will be ready for production soon because I can't wait to try my hands on it.

I have used Chart::Clicker 2-3 years back and it was good for my need at that time. It is based on Cairo. The reason I used it instead of Chart::Plotly is because it was very new. But its 2021 and there are lots of beautiful charting library already available (specially on front-end side) which is not just easy to use but also look good to eyes. Few notable ones are -

There are many more (a lot) but that is not the focus of today. Out of these

Chart.js is lightweight and elegant when you have some basic and simple requirement. When your requirement gets more complex or too specific it will be little difficult to achieve and you have to write some extra codes to achieve it.
D3.js is the beast which is difficult to tame but can win you any battle.(I spent some time on taming it but still not confident - maybe I am not a good Javascript developer till now.)
I can personally vouch for Amcharts. After spending time on above two library, Amcharts is the sweet spot. It has everything you need and out of the box. Plenty of good examples on there website and highly configurable to your need - especially there v4 which is written in Typescript.

That's been said lets start with our example. There is a talk given by author sometime back. You can also take a look at it for more info.

Basic chart elements

Lets look at basic chart terminologies. From canvasJS docs

Creating the data config

We will be creating multi-line chart for today. To make it easy to use and highly configurable we will be creating a input data json file

{
    "title": "Number of automobiles sold per day by manufacturer",
    "domainAxis": {
        "data": ["2020-04-15", "2020-04-16", "2020-04-17", "2020-04-18"],
        "label": "Date"
    },
    "rangeAxis": {
        "lines": {
            "line1": {
                "data": [10,3,5,9],
                "legendName": "Honda"
            },
            "line2": {
                "data": [20,15,8,10],
                "legendName": "Toyota"
            },
            "line3": {
                "data": [6,19,12,4],
                "legendName": "Ford"
            },
            "line4": {
                "data": [16,10,6,12],
                "legendName": "Renault"
            }
        },
        "label": "Numbers of automobiles sold"
    }
}

The names are self explanatory. domainAxis is the x-axis which is a Date. rangeAxis is y-axis which has 3 lines for now.

Directory structure

Our directory structure will be simple. input for input data, output for generated chart, lib for perl module.

┣ 📂input
┃ ┗ 📜input_data.json
┣ 📂lib
┃ ┗ 📜CreateCharts.pm
┣ 📂output
┣ 📜multi_line_chart.pl
┗ 📜README.md

Creating the module

While working on creating the multi-line chart, I spend a quite amount of time understanding it. The documentation is good but still I think it can be improved. Why like this and how this is happening is unanswered at some places. One good thing is there are lot of example present. So, you can look at it and found your way out.

So, lets create our module.

package CreateCharts;
use strict;
use warnings;

use Chart::Clicker;
use Chart::Clicker::Context;
use Chart::Clicker::Data::DataSet;
use Chart::Clicker::Data::Marker;
use Chart::Clicker::Data::Series;
use Chart::Clicker::Axis::DateTime;

use Geometry::Primitive::Rectangle;
use Geometry::Primitive::Circle;

use Graphics::Color::RGB;

use DateTime;

sub new {
    my ($class, @arguments) = @_;
    my $self = {@arguments};
    bless $self, $class;
    return $self;
}

sub _generate_specific_colors {
    my ($self) = @_;

    # build the color allocator
    # Add more colors in case your line is more than 6
    my $ca = Chart::Clicker::Drawing::ColorAllocator->new;

    my $red    = Graphics::Color::RGB->new({red => .75, green => 0,   blue => 0,   alpha => .8});
    my $green  = Graphics::Color::RGB->new({red => 0,   green => .75, blue => 0,   alpha => .8});
    my $blue   = Graphics::Color::RGB->new({red => 0,   green => 0,   blue => .75, alpha => .8});
    my $orange = Graphics::Color::RGB->new(red => .88, green => .48, blue => .09, alpha => .8);
    my $aqua   = Graphics::Color::RGB->new(red => 0, green => 1, blue => 1, alpha => .8);
    my $fuchsia = Graphics::Color::RGB->new(red => 1, green => 0, blue => 1, alpha => .8);

    $ca->add_to_colors($green);
    $ca->add_to_colors($red);
    $ca->add_to_colors($blue);
    $ca->add_to_colors($orange);
    $ca->add_to_colors($aqua);
    $ca->add_to_colors($fuchsia);

    return $ca;
}

sub _genrate_random_colors {

    # let Chart::Clicker autmatically pick complementing colors for you
    # https://metacpan.org/pod/Chart::Clicker::Drawing::ColorAllocator#AUTOMATIC-COLOR-ALLOCATION
    my $ca = Chart::Clicker::Drawing::ColorAllocator->new({
            seed_hue => 0,    #red
    });
    return $ca;
}

sub _add_series {
    my ($self, $x_axis, $y_axis) = @_;
    my $ds = Chart::Clicker::Data::DataSet->new;
    foreach my $axis (keys %{$y_axis}) {
        $ds->add_to_series(
            Chart::Clicker::Data::Series->new(
                keys   => $x_axis,
                values => $y_axis->{$axis}->{data},
                name   => $y_axis->{$axis}->{legendName},
            )
        );
    }
    return $ds;
}

sub _add_title {
    my ($self, $cc, $title) = @_;
    $cc->title->font->family('Helvetica');

    $cc->title->text($title);
    $cc->title->font->size(20);
    $cc->title->padding->bottom(10);
}

sub _style_legend {
    my ($self, $cc) = @_;
    $cc->legend->font->size(20);
    $cc->legend->font->family('Helvetica');
}

sub _add_background {
    my ($self, $cc) = @_;

    # https://metacpan.org/pod/Graphics::Color::RGB
    my $titan_white = Graphics::Color::RGB->new(red => .98, green => .98, blue => 1, alpha => 1);
    my $white       = Graphics::Color::RGB->new(red => 1,   green => 1,   blue => 1, alpha => 1);

    $cc->plot->grid->visible(1);
    $cc->background_color($white);
    $cc->plot->grid->background_color($titan_white);
    $cc->border->width(0);
}

sub _add_label {
    my ($self, $def, $x_label, $y_label) = @_;
    $def->domain_axis->label($x_label);
    $def->range_axis->label($y_label);
    $def->domain_axis->label_font->size(20);
    $def->range_axis->label_font->size(20);
}

sub _add_shapes_to_lines {
    my ($self, $defctx) = @_;

    # https://metacpan.org/pod/Chart::Clicker::Renderer::Line#shape
    $defctx->renderer->shape(Geometry::Primitive::Circle->new({radius => 6,}));

    # https://metacpan.org/pod/Chart::Clicker::Renderer::Line#shape_brush
    $defctx->renderer->shape_brush(
        Graphics::Primitive::Brush->new(
            width => 2,
            color => Graphics::Color::RGB->new(red => 1, green => 1, blue => 1)
        )
    );

    $defctx->renderer->brush->width(2);
}

sub generate_chart {
    my ($self, $chart_loc, $summary_info) = @_;

    my $cc = Chart::Clicker->new(width => 800, height => 600, format => 'png');

    my $x_axis = $summary_info->{domainAxis};
    my $y_axis = $summary_info->{rangeAxis};

    my (@epoch_datetime);

    for my $datetime (@{$x_axis->{data}}) {

        # https://github.com/gphat/chart-clicker/blob/master/example/date-axis.pl
        # Need to convert date time string to epoch time
        my ($y, $m, $d) = split(/-/, $datetime);
        my $epoch = DateTime->new(year => $y, month => $m, day => $d)->epoch;
        push @epoch_datetime, $epoch;
    }

    my $ds = $self->_add_series(\@epoch_datetime, $y_axis->{lines});
    $cc->add_to_datasets($ds);

    # To generate random colors and let Chart::Clicker autmatically pick color
    # my $ca = $self->_genrate_random_colors();

    # To generate some specific colors for lines ise this function
    my $ca = $self->_generate_specific_colors();
    $cc->color_allocator($ca);

    $self->_add_title($cc, $summary_info->{title});
    $self->_style_legend($cc);
    $self->_add_background($cc);

    my $defctx = $cc->get_context('default');

    # For range axis
    $defctx->range_axis->range(Chart::Clicker::Data::Range->new(lower => 0));
    $defctx->range_axis->format('%d');

    # https://metacpan.org/pod/Chart::Clicker::Axis#fudge_amount
    # $defctx->range_axis->fudge_amount(0.02);

    # For domain axis
    $defctx->domain_axis(
        Chart::Clicker::Axis::DateTime->new(
            format => "%Y-%m-%d",
            ticks  => scalar @{$x_axis->{data}},
            position         => 'bottom',
            tick_label_angle => 0.78539816,                       # 45 deg in radians
            orientation      => 'vertical',
            tick_font        => Graphics::Primitive::Font->new({family => 'Helvetica', slant => 'normal'})
        )
    );

    $self->_add_label($defctx, $x_axis->{label}, $y_axis->{label});
    $self->_add_shapes_to_lines($defctx);

    $cc->write_output($chart_loc);
}

1;

I tried to use proper names for subroutines, so it is easy to understand. Comments are also added.

_generate_specific_colors - generates specific set of color for your lines. You can update it to suit your need.
_generate_random_colors - allow the colors to be chosen randomly by the module itself.
_add_series - add the series/ lines to your chart. The loop inside will run for each line(total 3) and add it to dataset.
_add_title - add title to your chart which you have provided in input json file. There is different format options available which you can update to suit your requirements.
_style_legend - is to format the legends.
_add_background - add the background colors and format to the chart.
_add_label - to the charts and format them.
_add_shapes_to_lines - add the circular shape and brush to the lines.
generate_chart - this is the subroutine we will be calling from outside with the input json data.

You may have notices the underscore in front of subroutines name. This is because they are private to this module.

Using the module

Let create our startup script to access this module for creating our chart.

#!/usr/bin/env perl

use strict;
use warnings;
use Cwd qw( abs_path );
use File::Basename qw( dirname );
use JSON;

BEGIN {
    $ENV{"SCRIPT_DIR"} = dirname(abs_path($0));
}

use lib $ENV{"SCRIPT_DIR"} . "/lib";
use CreateCharts;

my $chart_out_file = $ENV{"SCRIPT_DIR"} . "/output/lineChart.png";

sub read_json_file {
    my ($json_file) = @_;
    print "\nReading $json_file";

    open(my $in, '<', $json_file) or print "Unable to open file $json_file : $!";
    my $json_text = do { local $/ = undef; <$in>; };
    close($in) or print "\nUnable to close file : $!";

    my $config_data = decode_json($json_text);
    return $config_data;
}

sub main {
    my $data_in_json = read_json_file($ENV{"SCRIPT_DIR"} . "/input/input_data.json");

    my $chart = CreateCharts->new();
    $chart->generate_chart($chart_out_file, $data_in_json);
}

main;

We are reading the JSON data from input file and calling generate_chart with it.

Running the script

Not lets run this script and check the output.

The output which you are seeing above is from a linux machine.
One of the limitation I found about this module is it is difficult to make it run on windows. Maybe because of cairo or something else - not sure. Installing it on windows is not straight forward. I tried it but failed to make it working. If you know a way, let me know.

Usage

As I mentioned before because of new modern libraries available, there is little chance you will be tempted to use this module.
The one place I can see its usability today is -

You want to quickly create a chart and you have no intention of using it on the web.
Creating a chart over a period of time and sending it as mail attachment. Your input can be from a SQL or NoSQL database. That's the reason I have created the input as JSON so it can be created easily.
I have taken the example of only line chart, but it also support many other types also. You can look at the examples for more understanding.
Remember that colors play an important role in chart. Choosing a proper color with proper opacity (hue/saturation) play a vital role. The same chart can look good or bad based on color choices. There are various online tools which can help you. My personal favorite is W3 color picker. You can look at the color tutorial for more understanding.

The above example is also available at github

What next

I will be writing more about Chart::Plotly and its usage as I found it more modern.
Also, I will be writing about Amcharts and how you can use it with Perl(Mojolicious) and React.js

Cover image taken from here

Create Excel file using Perl from JSON data

Gaurav Rai — Tue, 02 Feb 2021 17:10:39 +0000

Excel::Writer::XLSX is one of the de facto standard for creating excel file in Perl.
The documentation of module contain the ample example of the usage.
I am just adding one more case where you have JSON data and you have to write it in excel.
Before starting, I have to confess something. I hate writing in excel. I never liked creating excel. But it is very popular in business world. I have got several request where people ask data in excel format. One time the request was to create charts in excel. As a technical person who knows little bit of JavaScript and charting library, creating chart in excel is never interest to me.
So, I have created a script for this purpose only.
So lets get started.

1. Input data-

The input to this script is a JSON file. It is in a particular format. As it is JSON, it is easy to generate either programmatically or manually. You can generate it from database. If you are using NoSQL it will be piece of cake. Even for SQL generating JSON is very easy.
Let look at the the input JSON file.
For demo purpose I am taking the sample financial data available at Microsoft docs.

{
    "Financial Info": [
        { "Date": ["01-22-2021", "04-05-2017", "07-12-2019", "19-05-2020"] },
        { "Segment": ["Government", "Enterprise", "Channel Partners", "Midmarket"] },
        { "Country": ["Canada", "Germany", "France", "India"] },
        { "Product": ["Montana", "Paseo", "VTT", "Hero"] },
        { "Unit Sold": [1618, 888, 1545, 5693] },
        { "Sale Price": [20, 350, 300, 693] },
        { "Profit": [2457, 0, -350, 3763] }
    ],
    "User Info": [
        { "Name": ["Gaurav", "Saurabh"] },
        { "Age": [10, 5] },
        { "Salary": [5673, 3355] }
    ]
}

I will walk through the JSON file but before that if you look at the generated excel file things will become clear.

These two images are from a single excel file having 2 worksheet.
Now if you compare the JSON file with the images you will have a pretty good idea about. Lets visit those.

The first keys Financial Info and User Info are name of the 2 worksheet. You can have any name you want.
The value of these contains the data which you wan to show in tabular format in sequential order(hence inside array).
Inside those data the keys are table headers or top heading which you wan to highlight.
The values of these key contain the data which you want to put in the column having that key as header.

You can add more worksheet or add more headers or do any thing. The baseline is you just have to stick to this format.

2. Parsing the JSON file

Lets start with the parsing this JSON file.

#!/usr/bin/env perl
use strict;
use warnings;

use JSON;

sub read_json_file {
    my ($json_file) = @_;
    print "\nReading $json_file";

    open(my $in, '<', $json_file) or die "Unable to open file $json_file : $!";
    my $json_text = do { local $/ = undef; <$in>; };
    close($in) or die "Unable to close file : $!";

    my $config_data = decode_json($json_text);
    return $config_data;
}

sub main() {
    my $output_file = "output.xlsx";
    my $data        = read_json_file("input.json");
}

main();

Here we are slurping the whole file and decoding it.

3. Writing to excel

Lets try to creating excel file out of parsed JSON file to get the expected output as mentioned in image above.
We will create a separate function for that. Adding this code to above one

use Excel::Writer::XLSX;

sub write_to_excel {
    my ($output_file, $data_to_write) = @_;
    print "\nWriting output to $output_file ";
    my $workbook = Excel::Writer::XLSX->new($output_file);

    # https://metacpan.org/pod/Excel::Writer::XLSX#add_format(-%25properties-)
    # Format for 'heading' (background - light blue)
    my $header_format = $workbook->add_format(
        border    => 1,
        bg_color  => '#99c2ff',
        bold      => 1,
        text_wrap => 1,
        valign    => 'vcenter',
        align     => 'center',
    );
    my %font = (color => 'black', valign => 'vcenter', align => 'left', font => 'Calibri', border => 1);

    # Format for a normal text (background - white)
    my $normal_format = $workbook->add_format(%font);

    }
}

We have created 2 different format for now -

Header format where we want to highlight the header
A normal standard format for all other text We are using add_format for that Now just take a look at excel file and understand how to access a particular cell. The first cell is located at A1 or (0, 0), second B1 or (0, 1) and so on. Now to automatically access these cell we have to write code in such way that will get increment to B,C,D... while going horizontally. Similar for going vertically the number should increment i.e. 1,2,3... etc. We can create a separate module for that where we can generate the next string.

package StringIterator;
use strict;
use warnings;

sub new {
    my ($class, @arguments) = @_;
    my %self = @arguments;
    if (!exists $self{set}) {
        $self{set} = ["A" .. "Z"];
    }
    if (!exists $self{value}) {
        $self{value} = -1;
    }
    $self{size} = @{$self{set}};

    return bless \%self, $class;
}

# Increment the string to the next one
sub increment {
    my $self = shift;
    return $self->{value}++;
}

# get the current string (e.g. A or B ...)
sub get_current {
    my $self = shift;
    my $n    = $self->{value};
    my $size = $self->{size};
    my $s    = "";

    while ($n >= $size) {
        my $offset = $n % $size;
        $s = $self->{set}[$offset] . $s;
        $n /= $size;
    }
    $s = $self->{set}[$n] . $s;

    return $s;
}

# Get next string
sub get_next {
    my $self = shift;
    $self->increment;
    return $self->get_current;
}

1;

I have to give credit to someone for this code. While doing this work, I have found the help on the internet sometime back which I don't remember now from where. I tried to search now but unable to found it. I have modified it little bit for my need. But if someone found the actual source please let me know.
In the above code we have an array contains character form "A" to "Z". We also have a variable having initial value '-1'. With every increment call we are increasing the value of variable and returning the character present at that index in get_current.

Now lets use this module in our code and try to fill the excel.

#!/usr/bin/env perl
use strict;
use warnings;

# https://metacpan.org/pod/Excel::Writer::XLSX
use Excel::Writer::XLSX;
use JSON;
use Cwd qw( abs_path );
use File::Basename qw( dirname basename );
use lib dirname(abs_path($0));
use StringIterator;

sub write_to_excel {
    my ($output_file, $data_to_write) = @_;
    print "\nWriting output to $output_file ";
    my $workbook = Excel::Writer::XLSX->new($output_file);

    # https://metacpan.org/pod/Excel::Writer::XLSX#add_format(-%25properties-)
    # Format for 'heading' (background - light blue)
    my $header_format = $workbook->add_format(
        border    => 1,
        bg_color  => '#99c2ff',
        bold      => 1,
        text_wrap => 1,
        valign    => 'vcenter',
        align     => 'center',
    );
    my %font = (color => 'black', valign => 'vcenter', align => 'left', font => 'Calibri', border => 1);

    # Format for a normal text (background - white)
    my $normal_format = $workbook->add_format(%font);

    # Excel column start from 1
    my $row = 1;
    foreach my $key (keys %{$data_to_write}) {
        my $iter      = StringIterator->new();
        my $worksheet = $workbook->add_worksheet($key);

        # Column A width set to 20
        $worksheet->set_column('A:A', 20);

        # Add a table to the worksheet.
        if (@{$data_to_write->{$key}}) {
            foreach my $element (@{$data_to_write->{$key}}) {
                foreach my $header (keys %{$element}) {
                    my $column_initial = $iter->get_next;

                    $worksheet->write($column_initial . $row, $header, $header_format);
                    $worksheet->write_col($column_initial . ($row + 1),
                        $element->{$header}, $normal_format);
                }
            }
        }
    }
    $workbook->close();
    print "\nFinished writing to file : $output_file";
    return 1;
}

sub main() {
    my $output_file = "out.xlsx";
    my $data        = read_json_file("data.json");

    write_to_excel($output_file, $data);
}

main();

Here we looping through our JSON data ($data_to_write) and adding the initial keys as worksheet name in workbook(add_worksheet).
Next we are setting the width of first column(A) as 20 by using set_column
After that, we are looping through the values which contain the actual data.
In each iteration we are getting the column initial (i.e A or B or C etc), writing the header using write with the header format and adding the whole array of data for that particular header using write_col.

Save the file and try to run it. We will get the exact output as mentioned in Step 1.
The above code is generic enough to generate a excel file if you provide the JSON in that particular format without any issue.

4. Some more addition - making specific

Sometime there is also a requirement to highlight a particular field or set of fields.
E.g. For finance data, we want to highlight field where we are in loss as red, profit as green, no profit no loss as yellow. Also, we wan to highlight field where we have sold more than 1600 unit.
Lets update our code to accumulate this requirement.

We will use add_format to create more format for profit, loss and others.
We will be using conditional_formatting to format specific cell satisfying certain criteria. Adding more to the subroutine above.

    # Format for error text (background - red)
    my $error_format = $workbook->add_format(%font, bg_color => '#ff0000');

    # Format for success text (background - green)
    my $success_format = $workbook->add_format(%font, bg_color => '#00ff00');

    # Format for neutral text (background - yellow)
    my $neutral_format = $workbook->add_format(%font, bg_color => '#ffff00');

    # Excel column start from 1
    my $row = 1;

    foreach my $key (keys %{$data_to_write}) {
        my $iter      = StringIterator->new();
        my $worksheet = $workbook->add_worksheet($key);
        if ($key eq "Financial Info") {
            my $row_size = scalar @{$data_to_write->{$key}->[6]->{'Profit'}};
            # 'Profit' is in 'G' column or '6'.
            # The data start from (1, 6) to (no of elements, 6)
            # If the value is greater than 0, apply the success format
            $worksheet->conditional_formatting(1, 6, $row_size, 6,
                {
                    type     => 'cell',
                    criteria => 'greater than',
                    value    => 0,
                    format   => $success_format
                }
            );
            # If the value is equal to 0, apply the neutral format
            $worksheet->conditional_formatting(1, 6, $row_size, 6,
                {
                    type     => 'cell',
                    criteria => 'equal to',
                    value    => 0,
                    format   => $neutral_format
                }
            );
            # If the value is less than 0, apply the error format
            $worksheet->conditional_formatting(1, 6, $row_size, 6,
                {
                    type => 'cell',
                    criteria => 'less than',
                    value => 0,
                    format => $error_format
                }
            );
            # 'Unit Sold' is in 'E' column or '4'.
            # The data start from (1, 4) to (no of elements, 4)
            # If the value is greater than 1600, apply the success format
            $worksheet->conditional_formatting(1, 4, $row_size, 4,
                {
                    type => 'cell',
                    criteria => '>',
                    value => 1600,
                    format => $success_format
                }
            );
        }

        # Column A width set to 20
        $worksheet->set_column('A:A', 20);

I have added the comments for clarity.
Save the file and run it. You can see the colored output.

5. Adding charts

We can also add different charts. There are lot of option available at Excel::Writer::XLSX::Chart.
For now lets create a column chart for number of units sold.
We will be using add_chart for this.

        # Add a column chart
        # https://metacpan.org/pod/Excel::Writer::XLSX#add_chart(-%properties-)
        my $chart = $workbook->add_chart(type => 'column', name => 'chart', embedded => 1);

        # https://metacpan.org/pod/Excel::Writer::XLSX::Chart#add_series()
        # ranges: [ $sheetname, $row_start, $row_end, $col_start, $col_end ]
        $chart->add_series(
            name       => "Unit Sold",
            categories => ["Financial Info", 1, $row_size, 2, 4],
            values     => ["Financial Info", 1, 4, $row_size, 4],
            line       => {color => 'blue'},
        );

        # https://metacpan.org/pod/Excel::Writer::XLSX#insert_chart(-$row,-$col,-$chart,-%7B-%25options-%7D-)
        $worksheet->insert_chart('J2', $chart);

I have again added the comments for explanation.

6. Final look

Lets look at our final full script after the addition.

#!/usr/bin/env perl
use strict;
use warnings;

# https://metacpan.org/pod/Excel::Writer::XLSX
use Excel::Writer::XLSX;
use JSON;
use Cwd qw( abs_path );
use File::Basename qw( dirname basename );
use lib dirname(abs_path($0));
use StringIterator;

sub write_to_excel {
    my ($output_file, $data_to_write) = @_;
    print "\nWriting output to $output_file ";
    my $workbook = Excel::Writer::XLSX->new($output_file);

    # https://metacpan.org/pod/Excel::Writer::XLSX#add_format(-%25properties-)
    # Format for 'heading' (background - light blue)
    my $header_format = $workbook->add_format(
        border    => 1,
        bg_color  => '#99c2ff',
        bold      => 1,
        text_wrap => 1,
        valign    => 'vcenter',
        align     => 'center',
    );
    my %font = (
        color  => 'black',
        valign => 'vcenter',
        align  => 'left',
        font   => 'Calibri',
        border => 1
    );

    # Format for a normal text (background - white)
    my $normal_format = $workbook->add_format(%font);

    # Format for error text (background - red)
    my $error_format = $workbook->add_format(%font, bg_color => '#ff0000');

    # Format for success text (background - green)
    my $success_format = $workbook->add_format(%font, bg_color => '#00ff00');

    # Format for neutral text (background - yellow)
    my $neutral_format = $workbook->add_format(%font, bg_color => '#ffff00');

    # Excel column start from 1
    my $row = 1;

    foreach my $key (keys %{$data_to_write}) {
        my $iter      = StringIterator->new();
        my $worksheet = $workbook->add_worksheet($key);
        if ($key eq "Financial Info") {
            my $row_size = scalar @{$data_to_write->{$key}->[6]->{'Profit'}};
            # 'Profit' is in 'G' column or '6'.
            # The data start from (1, 6) to (no of elements, 6)
            # If the value is greater than 0, apply the success format
            $worksheet->conditional_formatting(1, 6, $row_size, 6,
                {
                    type     => 'cell',
                    criteria => 'greater than',
                    value    => 0,
                    format   => $success_format
                }
            );
            # If the value is equal to 0, apply the neutral format
            $worksheet->conditional_formatting(1, 6, $row_size, 6,
                {
                    type     => 'cell',
                    criteria => 'equal to',
                    value    => 0,
                    format   => $neutral_format
                }
            );
            # If the value is less than 0, apply the error format
            $worksheet->conditional_formatting(1, 6, $row_size, 6,
                {
                    type => 'cell',
                    criteria => 'less than',
                    value => 0,
                    format => $error_format
                }
            );
            # 'Unit Sold' is in 'E' column or '4'.
            # The data start from (1, 4) to (no of elements, 4)
            # If the value is greater than 1600, apply the success format
            $worksheet->conditional_formatting(1, 4, $row_size, 4,
                {
                    type => 'cell',
                    criteria => '>',
                    value => 1600,
                    format => $success_format
                }
            );

            # Add a column chart
            # https://metacpan.org/pod/Excel::Writer::XLSX#add_chart(-%properties-)
            my $chart = $workbook->add_chart(type => 'column', name => 'chart', embedded => 1);

            # https://metacpan.org/pod/Excel::Writer::XLSX::Chart#add_series()
            # ranges: [ $sheetname, $row_start, $row_end, $col_start, $col_end ]
            $chart->add_series(
                name       => "Unit Sold",
                categories => ["Financial Info", 1, $row_size, 2, 4],
                values     => ["Financial Info", 1, 4, $row_size, 4],
                line       => {color => 'blue'},
            );

            # https://metacpan.org/pod/Excel::Writer::XLSX#insert_chart(-$row,-$col,-$chart,-%7B-%25options-%7D-)
            $worksheet->insert_chart('J2', $chart);
        }

        # Column A width set to 20
        $worksheet->set_column('A:A', 20);

        # Add a table to the worksheet.
        if (@{$data_to_write->{$key}}) {
            foreach my $element (@{$data_to_write->{$key}}) {
                foreach my $header (keys %{$element}) {
                    my $column_initial = $iter->get_next;

                    $worksheet->write($column_initial . $row, $header, $header_format);
                    $worksheet->write_col($column_initial . ($row + 1),
                        $element->{$header}, $normal_format);
                }
            }
        }
    }

    $workbook->close();
    print "\nFinished writing to file : $output_file";
    return 1;
}

sub read_json_file {
    my ($json_file) = @_;
    print "\nReading $json_file";

    open(my $in, '<', $json_file) or die "Unable to open file $json_file : $!";
    my $json_text = do { local $/ = undef; <$in>; };
    close($in) or die "Unable to close file : $!";

    my $config_data = decode_json($json_text);
    return ($config_data);
}

sub main() {
    my $output_file = "out.xlsx";
    my $data        = read_json_file("data.json");

    write_to_excel($output_file, $data);
}

main();

After running the script we will get following output -

You can update the specific things to suite your requirements.
It is quite handy where you can generate JSON data in particular format(which is quite easy across any language) and convert it into beautiful excel.
Code is also available at Github

Perl Onion logo taken from here
Excel logo taken from here
JSON logo taken from here

Generate checksum using Perl

Gaurav Rai — Thu, 21 Jan 2021 17:20:51 +0000

In this article we will be looking at how to generate different type of checksum across Linux and Windows.

In Perl, there are various module available to generate checksum -

Digest::SHA - Perl core module
CryptX::Digest

There are few other but either they are specific to particular digest or abandoned.

Digest::SHA is in Perl core since v5.9.3. So, if you have Perl installed having higher version than 5.9.3 you will have Digest::SHA and installation is not needed.
But what if you are in some closed system, stuck with some older Perl version. What if you don't have permission and you can't install/use any external modules. I have faced similar issue and I have created this script just for that purpose.

So lets get started.
Both linux and windows comes with inbuilt utilities to calculate different hash -

Linux - md5sum, sha1sum, sha256sum, sha512sum etc.
Windows - certutil.exe

We can execute these commands using backticks operator and get there output.
For linux the commands are present in /usr/bin. For windows it is in C:/Windows/System32/

#!/usr/bin/env perl

use strict;
use warnings;

# HashAlgorithm choices: MD2 MD4 MD5 SHA1 SHA256 SHA384 SHA512
sub get_checksum {
    my ($path_to_file, $hash_algorithm) = @_;
    my $checksum;
    my $os = $^O;
    eval {
        if (-s $path_to_file) {
            if ($os =~ /MSWin32/i) {
                $checksum = (
                    split(
                        '\n',
                        `C:/Windows/System32/certutil.exe -hashfile $path_to_file $hash_algorithm`
                    )
                )[1];
            }
            else {
                my $cmd = "/usr/bin/" . lc($hash_algorithm) . "sum $path_to_file";
                $checksum = (split(' ', `$cmd`))[0];
            }
        }
        else {
            $checksum = "File is empty";
        }
    };
    if ($@) {
        print("Unable to calculate the $hash_algorithm for $path_to_file : $@");
    }
    return $checksum;
}

# Calculate checksum of the given file
my $checksum = get_checksum("test.txt", "SHA256");
print $checksum;

The output from external command contains some extra elements also. We are removing those and getting only the checksum.
Running this script generated the following output -

946576fd7b973e3a8c78e5695637727399f34ca8d6fd6b89f1f5328c1e0b0bdc

We can pass different type of checksum parameter to get_checksum function and and get different output.
You can also modify the script to use it as command line tool.
This can come in handy if you want to check integrity of file across different OS without changing your code.

Perl onion logo taken from here
Hash function image taken from here

Sending HTML Emails Using Perl

Gaurav Rai — Wed, 20 Jan 2021 15:31:36 +0000

We will be looking at sending HTML emails with attachments using Perl.
As per Perl TIMTOWTDI philosophy there are multiple ways to send email. Few of them are -

Using /usr/sbin/sendmail utility - for linux
MIME::Lite
Email::Send
Email::Sender

I remember when I started using Perl and when a day arrived where I have to send email, I found MIME::Lite. It was good for that time being and its gets the job done.
But as the time progress multiple alternative appears and also in current day MIME::Lite is not the recommended way to send email. From there documentation -

MIME::Lite is not recommended by its current maintainer. There are a number of alternatives, like Email::MIME or MIME::Entity and Email::Sender, which you should probably use instead. MIME::Lite continues to accrue weird bug reports, and it is not receiving a large amount of refactoring due to the availability of better alternatives. Please consider using something else.

Also, for Email::Send -

Email::Send is going away... well, not really going away, but it's being officially marked "out of favor." It has API design problems that make it hard to usefully extend and rather than try to deprecate features and slowly ease in a new interface, we've released Email::Sender which fixes these problems and others. As of today, 2008-12-19, Email::Sender is young, but it's fairly well-tested. Please consider using it instead for any new work.

I also got opportunity to work on some older codes and there were extensive use of MIME::Lite and sendmail (which is the Linux utility for sending emails) and I thought these are de-facto standards but as the time progress I realize there are more and better ways to send email.
Today we will be seeing one of those ways to end email which is recommended as of now - Email::Sender.

Lets revisit few of the basic process of sending email-

A mail template containing mail body in HTML format. In case you want to send plain text email you can ignore this.
Creating mail with the subject, attachment and mail body created in Step 1.
Send Email using SMTP server.

1. Creating the Email package -

We will be creating a package containing all our email utilities(SendMail.pm).

package SendMail;
use strict;
use warnings;

sub new {
    my ( $class, @arguments ) = @_;
    my $self = {@arguments};
    bless $self, $class;
    return $self;
}

1;

2. Generating Mail Template -

We will be creating a method for generating our mail template. You can use any library of your choice to generate this template. Few notable are -

I am using HTML::Template for this job as our scope is limited only to creating mail body and it is right tool for the job.
If you have some complex requirement you can check out other two (especially Text::Template).
Adding HTML::Template to the above code.

use HTML::Template;

sub generate_mail_template {
    my ( $self, $filename, $parameters ) = @_;

    # create mail body template. We don't want to die/exit if any of the parameters is missing
    my $template = HTML::Template->new( filename => $filename, die_on_bad_params => 0 );
    $template->param($parameters);
    return $template;
}

This takes the absolute filename of the template file and the parameters we will be substituting in that template file.
Right now I am using a simple template which can be extended based on requirement (template.html)

<html>
    <head>
        <title>Test Email Template</title>
    </head>
    <body>
        My Name is <TMPL_VAR NAME=NAME>
        My Location is <TMPL_VAR NAME=LOCATION>
    </body>
</html>

3. Creating Email to send -

Now comes the part where we will be creating our email with subject, attachment and body before sending.
But before that, we want to make the users to which we will be sending email configurable and for that we will be creating a config file.
Right now I am creating a json based config (config.json). But, you can create it in any format of you choice.

{
    "mail" : {
                "mail_from" : "test@xyz.com",
                "mail_to" : ["grai@xyz.com"],
                "mail_cc_to" : ["grai@xyz.com", "grai2@xyz.com"],
                "smtp_server" : "abc.xyz"
             }
}

The key names are self explanatory.

Now back to the part of creating email. We will create another method for this by adding to above code.


use Email::MIME;
use File::Basename qw( basename );

sub create_mail {
    my ( $self, $file_attachments, $mail_subject, $mail_body ) = @_;

    my @mail_attachments;
    if (@$file_attachments) {
        foreach my $attachment (@$file_attachments) {
            my $single_attachment = Email::MIME->create(
                attributes => {
                    filename     => basename($attachment),
                    content_type => "application/json",
                    disposition  => 'attachment',
                    encoding     => 'base64',
                    name         => basename($attachment)
                },
                body => io->file($attachment)->all
            );
            push( @mail_attachments, $single_attachment );
        }
    }
    # Multipart message : It contains attachment as well as html body
    my @parts = (
        @mail_attachments,
        Email::MIME->create(
            attributes => {
                content_type => 'text/html',
                encoding     => 'quoted-printable',
                charset      => 'US-ASCII'
            },
            body_str => $mail_body,
        ),
    );

    my $mail_to_users    = join ', ', @{ $self->{config}->{mail_to} };
    my $cc_mail_to_users = join ', ', @{ $self->{config}->{mail_cc_to} };

    my $email = Email::MIME->create(
        header => [
            From    => $self->{config}->{mail_from},
            To      => $mail_to_users,
            Cc      => $cc_mail_to_users,
            Subject => $mail_subject,
        ],
        parts => [@parts],
    );
    return $email;
}

This method will take list of attachments, subject and body and create a MIME part using Email::MIME.
Also there is Email::Stuffer which you can also use for similar part. But for now I am using Email::MIME.

4. Sending Email -

Now our mail to ready for send. We will create another method for it in addition to above methods.


use Email::Sender::Simple qw( sendmail );
use Email::Sender::Transport::SMTP;

sub send_mail {
    my ( $self, $email ) = @_;
    my $transport = Email::Sender::Transport::SMTP->new(
        {
            host => $self->{config}->{smtp_server}
        }
    );
    eval { sendmail( $email, { transport => $transport } ); };
    if ($@) {
        return 0, $@;
    }
    else {
        return 1;
    }
}

1;

This method will just take the email which we have created before and sent it to requested recipient.
Since our work is done here we will end the module with a true value.

5. Using the module -

Now we have the module ready. Lets create a perl script and try to utilize the methods which we have created(mail.pl).

#!/usr/bin/env perl

use strict;
use warnings;
use JSON;
use Cwd qw( abs_path );
use File::Basename qw( dirname );
use lib dirname(abs_path($0));
use SendMail;

sub read_json_file {
    my ($json_file) = @_;
    print "Reading $json_file";

    open (my $in, '<', $json_file) or print "Unable to open file $json_file : $!";
    my $json_text = do { local $/ = undef; <$in>; };
    close ($in) or print "Unable to close file : $!";

    my $config_data = decode_json($json_text);
    return ($config_data);
}

# Read the config file
my $config = read_json_file("config.json");

my $mail = SendMail->new("config" => $config->{'mail'});

# param which you want to substitute in mail template
my $mail_parameters = "NAME => 'Gaurav', LOCATION => 'INDIA'";

# path to mail attachments
my $attachments = ["abc.txt"];

# path to mail template
my $mail_template = "mail_template/template.html";

print "Generating HTML template for mail body";
my $template = $mail->generate_mail_template($mail_template, $mail_parameters);

print "Creating mail with body and attachments to send";
my $mail_subject = "Test Mail";
my $email = $mail->create_mail($attachments, $mail_subject, $template->output);

print "Sending email...";
my ($mail_return_code, $mail_exception) = $mail->send_mail($email);

if (defined $mail_exception) {
    print "Exception while sending mail: $mail_exception";
    return 0;
}
else {
    print "Mail Sent successfully";
    return 1;
}

And we are done. You can run this script from your terminal and test this. The full code for reference is also available at Github for your reference

Perl onion logo taken from here
Mail logo taken from here and here

Automate Win32 GUI using Perl and Win32::GuiTest

Gaurav Rai — Tue, 19 Jan 2021 16:44:30 +0000

We will be looking at how to automate windows application using Perl.
Again there are multiple ways to achieve this. Few of them are -

There are other autohotkey and autoitscript which are quite good in windows automation. I encourage you to check them out.

For this blog we will be using Win32::GuiTest for automating windows application.
We will start with the a small example which can later be extended.
Lets visit what we will be doing -

We will open the given directory in windows explorer (Right now C:/Perl64/lib/)
Search for a file or folder (e.g. strict.pm)
Right-click on that particular file or folder(this is achieved using keyboard shortcuts)
Select an option (right now 'Open with Notepad++')
Wait for the action to be completed because of the selected option
After the action is successful close it (close Notepad++)
Close the windows explorer

See, that quite simple thing we will be doing. We will take baby steps by starting with our first step.

1. Creating the package -

We will be creating a package containing all our automation utilities(Win32Operations.pm).

package Win32Operations;
use strict;
use warnings;

sub new {
    my $class = shift;
    my ($self) = {@_};
    bless $self, $class;
    return $self;
}

2. Starting Windows Explorer operations -

Now we have to open a dir in windows explorer. We will create a method for this.

sub start_explorer_operations {
    my ($self, $root_directory, $filename_to_open, $max_retry_attempts) = @_;

    # Path to windows explorer. The Windows directory or SYSROOT(%windir%) is typically C:\Windows.
    my $explorer = "%windir%\\explorer.exe";

    # Since we are on windows removed forward slash with backslashes
    $root_directory =~ s/\//\\/g;

    # seconds to spend between keyboard key presses
    my $key_press_delay = 1;

    # this is the interval the tester sleeps before checking/closing
    # any window; this is just for an eye effect so we can
    # watch what happens
    my $wait_time_for_windows = 3;

    my $count = 0;
    while ($count < $max_retry_attempts) {
        $count++;
        $self->{'logger'}->info("Opening dir: $root_directory in windows explorer");
        my $status_code = system($explorer , "$root_directory");
        if ($status_code == -1) {
            $self->{'logger'}->error(
                "Unable to open 'explorer.exe' with mentioned dir : $root_directory. Exited with return code :". $status_code);
            next;
        }
        elsif ($status_code & 127) {
            $self->{'logger'}->error("Child died with signal : " . ($status_code & 127));
            next;
        }
        else {
            # Windows explorer is opened hear and we are in the given dir
        }
    }
}

The comments are self explanatory. We will be opening windows explorer. It will retry for some time in case it fails.
Also we have to cross check whether the open window is the location to our expected folder or not.
We will also give some time to system for this operation.
For this we will create a generalized method -

sub _wait_for {
    my ($self, $title, $wait_time) = @_;
    my $win;
    while (1) {
        # This will look for the window with the given title
        ($win) = FindWindowLike(0, $title);

        if (defined $win) {
            $self->{'logger'}->info("Found window with title : $title.");
            last;
        }
        else {
            $self->{'logger'}->info("Unable to find window with title : $title. Retrying...");
            sleep $wait_time;
        }
    }
    return $win;
}

3. Bringing Window to front -

Now we are able to open the explorer windows and confirm that it is that window.
We will be trying to bring back that particular window to front. Reason being what if in between opening of explorer and starting our operation, our focus moved to somewhere else (someone clicked outside the explorer or open some other application). For this we will be creating another method -

sub _bring_window_to_front {
    my ($self, $window, $retry_attempts, $pause_between_operations) = @_;
    my $success = 1;
    my $count   = 0;
    while ($count < $retry_attempts) {
        $count++;
        if (SetActiveWindow($window)) {
            $self->{'logger'}->info("* Successfully set the window id: $window active");
        }
        else {
            $self->{'logger'}->warn("* Could not set the window id: $window active: $!");
            $success = 0;
        }
        if (SetForegroundWindow($window)) {
            $self->{'logger'}->info("* Window id: $window brought to foreground");
        }
        else {
            $self->{'logger'}->warn("* Window id: $window could not be brought to foreground: $!");
            $success = 0;
        }
        sleep $pause_between_operations;
        my $foreground_window_id = GetForegroundWindow();
        if ($foreground_window_id =~ /$window/i) {
            last;
        }
        else {
            $self->{'logger'}->info(
                "Found - $foreground_window_id instead of expected - $window. Will try again...");
            next;
        }
    }
    return $success;
}

The system restricts which processes can set the foreground window. A process can set the foreground window only if one of the following conditions is true:

The process is the foreground process.
The process was started by the foreground process.
The process received the last input event.
There is no foreground process.
The foreground process is being debugged.
The foreground is not locked.
The foreground lock time-out has expired.
No menus are active.

Please keep these in mind. Otherwise this method will fail.

4. Opening and closing in Notepad++ -

Now back to our 'else' part in 'start_explorer_operations' where we will be using these methods.
Now we have to search for our file, 'right-click' on it and select 'edit with Notepad++'.
After opening we will close it and close the explorer also.

Searching file - We may have noticed that if we open explore and start typing our file name on our keyboard, window will automatically select that particular file. We will be using similar thing here.
Right Click - we will be using context menu key on the keyboard (Another alternative - Shift + F10) for simulating right click.
Open with Application - If you press context menu key or Shift + F10, you can see different options and each options have a underline on one of the characters. These are the shortcuts to select that particular option/ In our case the underline is under N meaning if we press N we will be selecting that option. After that we will press Enter to open it.
Closing the application - We will be using the close command from the System Menu to close it. On the top of Notepad++ if you right click you can see the System Menu

sub start_explorer_operations {
    ....
        }
        else {
            my $window = $self->_wait_for(basename($root_directory), $wait_time_for_windows);
            $self->{'logger'}->info("Opened 'explorer.exe'. Window id : $window");

            $self->_bring_window_to_front($window, $max_retry_attempts, $wait_time_for_windows);
            $self->{'logger'}->info("Opening the file in Notepad++...");

            # This will use your keyboard to
            # Write the 'filename' to search
            # Right click on it(context menu) - {APP}
            # Select 'N' for Notepad++ and press 'Enter' to open
            # Replace 'N' with your own application shortcut if you are using something other application
            my @keys = ("$filename_to_open", "{APP}", "N", "{ENTER}");
            $self->_keys_press(\@keys, $key_press_delay);

            $self->{'logger'}->info("Opened the file in Notepad++. Closing it...");
            # Checking wheteher we are actually able to open Notepad++ or not
            my $opened_explore_window = $self->_wait_for($filename_to_open, $wait_time_for_windows);
            if (!$opened_explore_window) {
                $self->{'logger'}->warn("Cannot find window with title/caption $filename_to_open");
                next;
            }
            else {
               # We will come here when we successfully opened the file in Notepad++
               $self->{'logger'}
                    ->info("Window handle of $filename_to_open is " . $opened_explore_window);
                print $opened_explore_window;
                $self->_bring_window_to_front($opened_explore_window, $max_retry_attempts,
                    $wait_time_for_windows);
                $self->{'logger'}->info("Closing Notepad++ window...");
                MenuSelect("&Close", 0, GetSystemMenu($opened_explore_window, 0));
                $self->{'logger'}->info("Bringing Explorer window to front...");
                $self->_bring_window_to_front($window, $max_retry_attempts, $wait_time_for_windows);
                $self->{'logger'}->info("Closing Explorer window...");

                # There are different way to close windows File explorer -
                # 1. Alt+F4 (Will close anything on the top - little risky)
                # 2. Alt+F, then C
                # 3. CTRL+w (only closes the current files you're working on but leaves the program open)
                SendKeys("^w");
                return 1;
            }
        }
}

1;

Again, the comments are self explanatory. Another thing to note here is that, there are various ways to close the application. One of the common one is Alt + F4. But is it also risky. Reason being it can close anything on the top, anything.
What if on top there is some other application which you don't want to close.
Better to use context menu or application defined close.

5. Using the module -

Now we have the module ready. Lets create a perl script and try to utilize the methods which we have created(Win32Operations.pm). We will be using Log4perl for our logging. You can use anyone of your choice.

use strict;
use warnings;
use Log::Log4perl;
use Cwd qw( abs_path );
use File::Basename qw( dirname );
use lib dirname(abs_path($0));

use Win32Operations;

sub initialize_logger {

    # initialize logger, you can put this in config file also
    my $conf = qq(
            log4perl.category                   = INFO, Logfile, Screen
            log4perl.appender.Logfile           = Log::Log4perl::Appender::File
            log4perl.appender.Logfile.filename  = win32op.log
            log4perl.appender.Logfile.mode      = write
            log4perl.appender.Logfile.autoflush = 1
            log4perl.appender.Logfile.buffered  = 0
            log4perl.appender.Logfile.layout    = Log::Log4perl::Layout::PatternLayout
            log4perl.appender.Logfile.layout.ConversionPattern = [%d{ISO8601} %p] [%r] (%F{3} line %L)> %m%n
            log4perl.appender.Screen            = Log::Log4perl::Appender::Screen
            log4perl.appender.Screen.stderr     = 0
            log4perl.appender.Screen.layout     = Log::Log4perl::Layout::SimpleLayout
        );
    Log::Log4perl->init(\$conf);
    my $logger = Log::Log4perl->get_logger;
    $Log::Log4perl::DateFormat::GMTIME = 1;
    return $logger;
}

my $logger    = initialize_logger();
my $win32_obj = Win32Operations->new("logger" => $logger);

# This will open the given dir in windows explorer
# Right click on the given filename and open it in Notepad++ and then close it.
my $input_dir          = "C:/Program Files/Notepad++";
my $filename_to_open   = "readme.txt";
my $max_retry_attempts = 5;

$win32_obj->start_explorer_operations($input_dir, $filename_to_open, $max_retry_attempts);

And we are done. You can run this script from your terminal and test this. The full code for reference is also available at Github for your reference.

Perl onion logo taken from here
Windows logo taken from here