DEV Community

Anthony Leignel
Anthony Leignel

Posted on Edited on

πŸ“‘ Building Public Data Monitoring Dashboards: Automated Collection, SQLite Storage, and Time-Series Analysis

Fuel Radar interface displaying fuel availability indicators and historical time-series data.

Public data is available across a growing number of platforms, APIs, and institutional sources.

Electricity production, gas storage, fuel availability, cybersecurity vulnerabilities, maritime traffic, and Internet infrastructure can all be monitored using publicly accessible information.

However, accessing the data is only one part of the problem.

Different sources expose different formats, update at different frequencies, and describe different types of events. Historical information may be difficult to retrieve, and raw measurements do not necessarily provide a useful representation of the current situation.

The public data monitoring ecosystem developed by Palks Studio addresses this problem through five independent dashboards built around automated collection, local data storage, historical analysis, and native web visualization.

The technical objective is to transform heterogeneous public datasets into consistent, readable monitoring interfaces without requiring a large application framework or a centralized analytics platform.


1. Five Monitoring Domains, One Architectural Principle

The ecosystem currently covers five areas:

Monitoring domain Primary data categories
Energy Electricity, gas storage, oil stocks
Fuel availability Fuel availability, prices, geographical data
Cybersecurity Exploited vulnerabilities, network observations
Maritime traffic Vessel movements, strategic maritime flows
Digital infrastructure Internet signals, cloud incidents, DNS, routing

Each domain has its own data sources, collection requirements, and interpretation rules.

An electricity measurement cannot be processed in exactly the same way as a cybersecurity vulnerability or a cloud service incident.

However, the underlying architecture follows a common principle:

Public data sources
         β”‚
         β–Ό
     Collection
         β”‚
         β–Ό
     Validation
         β”‚
         β–Ό
    Normalization
         β”‚
         β–Ό
       Storage
         β”‚
         β–Ό
     Aggregation
         β”‚
         β–Ό
     Visualization
Enter fullscreen mode Exit fullscreen mode

The visible dashboard is the final layer of the system.

Most of the technical complexity exists before the data reaches the interface.


2. Separating Collection From Visualization

A monitoring dashboard should not necessarily retrieve every external dataset whenever a user opens the page.

Doing so would make the interface dependent on the availability, latency, and response time of multiple external services.

A more controlled architecture separates the collection process from the public interface.

Scheduled collectors
         β”‚
         β–Ό
External data sources
         β”‚
         β–Ό
Local data storage
         β”‚
         β–Ό
PHP application layer
         β”‚
         β–Ό
HTML / CSS / JavaScript / SVG
Enter fullscreen mode Exit fullscreen mode

The collectors retrieve information independently of page requests.

The public interface reads the available stored data and prepares the indicators, tables, and historical series required for visualization.

This separation provides several architectural advantages:

  • External API latency does not directly determine page rendering time
  • Historical information can remain available when a source becomes temporarily inaccessible
  • Data can be normalized before reaching the presentation layer
  • Collection and visualization can evolve independently
  • Different interfaces can reuse the same underlying datasets

The ecosystem uses scheduled collection processes, SQLite storage, PHP processing, and native browser technologies.

The public dashboards are not responsible for performing the complete collection pipeline on every request.


3. Scheduled Collection With Cron

Different data sources do not update at the same frequency.

An electricity dataset may provide daily or more frequent measurements. A vulnerability catalog may change only when a new entry is published. An Internet monitoring source may expose observations across short time windows.

The collection schedule must therefore reflect the characteristics of each source.

A conceptual collection workflow is:

Cron
  β”‚
  β–Ό
Collector execution
  β”‚
  β”œβ”€β”€ Request source data
  β”œβ”€β”€ Validate response
  β”œβ”€β”€ Normalize records
  β”œβ”€β”€ Store new information
  └── Record execution status
Enter fullscreen mode Exit fullscreen mode

A scheduled collector can run independently of user activity.

For example, a generic PHP collector could be scheduled every hour using:

0 * * * * /usr/bin/php /path/to/collector.php
Enter fullscreen mode Exit fullscreen mode

This command is an educational example of cron scheduling. It does not reproduce a production command or server path.

Collection Frequency Is Not Data Frequency

Running a collector every hour does not mean that its source publishes new information every hour.

Consider a source that publishes one measurement per day.

Collection time Source measurement date New observation
08:00 November 14 Yes
09:00 November 14 No
10:00 November 14 No
11:00 November 15 Yes

The collection process may execute several times before a new measurement becomes available.

This is why collection timestamps and source timestamps must remain distinct.

Otherwise, repeated retrievals of the same information could be incorrectly interpreted as new observations.


4. Normalizing Heterogeneous Data

Public datasets do not necessarily use the same structure.

One source may return JSON, another may expose CSV files, and another may provide an event-oriented API.

Even when two sources return JSON, their field names, timestamps, units, and identifiers may differ.

A conceptual example illustrates the problem.

Source A:

{
    "date": "2026-11-14",
    "value": 72.4,
    "unit": "percent"
}
Enter fullscreen mode Exit fullscreen mode

Source B:

{
    "timestamp": "2026-11-14T00:00:00Z",
    "measurement": 0.724
}
Enter fullscreen mode Exit fullscreen mode

Both records could represent the same percentage, but their representations are different.

A normalized record could be:

{
    "source": "example-source",
    "observed_at": "2026-11-14T00:00:00Z",
    "metric": "storage_level",
    "value": 72.4,
    "unit": "percent"
}
Enter fullscreen mode Exit fullscreen mode

These records are illustrative and do not reproduce an actual external API response or database schema.

Normalization can involve several operations:

Operation Purpose
Timestamp conversion Establish a consistent time basis
Unit conversion Preserve comparable measurements
Field mapping Standardize source-specific names
Type conversion Convert textual values into numbers
Identifier handling Recognize repeated observations
Source attribution Preserve the origin of information

Normalization does not mean forcing every dataset into a single universal structure.

A numerical time series and an incident record may require different storage models.

The important principle is consistency within each analytical domain.


5. SQLite as a Storage Layer

The monitoring ecosystem uses SQLite to retain collected information and historical datasets.

SQLite is particularly useful for applications where the storage requirements are primarily local and the workload consists of periodic writes followed by analytical reads.

A monitoring database can contain several types of information:

SQLite database
    β”œβ”€β”€ Latest observations
    β”œβ”€β”€ Historical measurements
    β”œβ”€β”€ Vulnerability records
    β”œβ”€β”€ Incident records
    β”œβ”€β”€ Source metadata
    └── Collection information
Enter fullscreen mode Exit fullscreen mode

The exact organization depends on the monitoring domain.

For example, gas storage measurements are naturally represented as dated numerical observations.

A cloud incident requires a different structure because it may contain an identifier, title, status, start time, and subsequent updates.

A Time-Series Storage Example

A simplified SQLite table for numerical observations could be defined as:

CREATE TABLE measurements (
    id INTEGER PRIMARY KEY,
    source TEXT NOT NULL,
    metric TEXT NOT NULL,
    observed_at TEXT NOT NULL,
    value REAL NOT NULL,
    unit TEXT NOT NULL,

    UNIQUE (source, metric, observed_at)
);
Enter fullscreen mode Exit fullscreen mode

This is an educational schema, not the production schema of the radars.

The unique constraint prevents the same source, metric, and observation timestamp from being inserted multiple times.

An index can support historical queries:

CREATE INDEX idx_measurements_history
ON measurements (
    source,
    metric,
    observed_at
);
Enter fullscreen mode Exit fullscreen mode

A query can then retrieve a specific historical series without scanning unrelated measurements.

SQLite provides a relatively small storage layer while still supporting indexes, constraints, transactions, and SQL aggregation.


6. Deduplication and Repeated Collection

Repeated collection introduces an important problem: the same observation may be retrieved multiple times.

A collector must distinguish between:

  • A genuinely new measurement
  • An observation already stored
  • An existing observation that has been corrected
  • An existing incident whose status has changed

These cases require different handling.

For numerical observations, a combination of source, metric, and timestamp can often provide a stable identity.

For incidents or vulnerability records, an identifier supplied by the source may be more appropriate.

A conceptual SQLite insertion could use:

INSERT INTO measurements (
    source,
    metric,
    observed_at,
    value,
    unit
)
VALUES (
    :source,
    :metric,
    :observed_at,
    :value,
    :unit
)
ON CONFLICT (
    source,
    metric,
    observed_at
)
DO NOTHING;
Enter fullscreen mode Exit fullscreen mode

This approach ignores repeated observations with the same identity.

However, ignoring duplicates is not always sufficient.

If a source corrects a previously published measurement, an update strategy may be required:

ON CONFLICT (
    source,
    metric,
    observed_at
)
DO UPDATE SET
    value = excluded.value,
    unit = excluded.unit;
Enter fullscreen mode Exit fullscreen mode

The appropriate strategy depends on the meaning of the record.

An immutable historical event should not necessarily be overwritten in the same way as a corrected numerical measurement.

The radars include anti-duplicate mechanisms, but the SQL examples above illustrate general storage strategies rather than reproducing the implementation of every collector.


7. Managing Historical Data

A current measurement provides limited context without its historical evolution.

For example:

Gas storage: 72%
Enter fullscreen mode Exit fullscreen mode

The value becomes more useful when compared with earlier measurements.

30 days ago : 81%
7 days ago  : 76%
Yesterday   : 73%
Today       : 72%
Enter fullscreen mode Exit fullscreen mode

The historical series reveals a direction of change that is not visible in the latest value alone.

However, retaining historical information creates another challenge: the amount of data that must be processed and displayed increases over time.

A dashboard covering several years does not necessarily need to render every individual measurement.

The monitoring interfaces therefore use different temporal resolutions depending on the selected period and dataset.

Time range Possible display resolution
Short term Individual or hourly records
30 days Daily observations
90 days Daily observations
One year Daily or aggregated values
All time Monthly or yearly aggregates

The exact resolution varies across the radars.

The purpose of aggregation is to reduce the number of displayed points without removing the historical information from storage.


8. Temporal Aggregation With SQL

SQL can transform detailed measurements into historical summaries before the data reaches the browser.

For example, a monthly average can be calculated from daily observations:

SELECT
    substr(observed_at, 1, 7) AS month,
    AVG(value) AS average_value,
    COUNT(*) AS sample_count
FROM measurements
WHERE metric = :metric
GROUP BY month
ORDER BY month;
Enter fullscreen mode Exit fullscreen mode

The resulting series contains one record per month instead of one record per observation.

Daily measurements
    β”‚
    β”œβ”€β”€ November 01
    β”œβ”€β”€ November 02
    β”œβ”€β”€ November 03
    β”œβ”€β”€ ...
    └── November 30
             β”‚
             β–Ό
       Monthly average
Enter fullscreen mode Exit fullscreen mode

The sample_count field is useful because a monthly average calculated from three observations is not equivalent in coverage to one calculated from thirty observations.

Aggregation Depends on the Metric

Not every historical series should use the same aggregation function.

Metric type Relevant aggregation
Storage percentage Average over available observations
New vulnerabilities Count of additions during a period
Event records Count of distinct events or signals
Peak measurement Maximum within a period
Latest observation Most recent valid value

A monthly count of newly published vulnerabilities has a different meaning from a monthly average gas storage level.

The aggregation method must preserve the meaning of the underlying data.

For example, a power measurement expressed in megawatts should not simply be summed to calculate energy consumption in megawatt-hours. That calculation requires accounting for the duration represented by each measurement.


9. Energy Radar: Combining Different Time Series

The Energy Radar brings together information relating to electricity, gas, and oil stocks.

Its datasets include:

  • Electricity consumption and production
  • Electricity generation mix
  • Gas consumption
  • Gas storage levels in France and the European Union
  • Historical oil stocks

These datasets represent different physical quantities.

Electricity production may be expressed as power, gas storage as a percentage of capacity, and oil stocks as a physical quantity.

Combining them into a single numerical score would remove important distinctions.

Instead, the interface presents separate indicators and historical series.

Gas Storage History

Gas storage provides a useful example of temporal aggregation.

Daily measurements can be retained for recent periods, while a longer historical view can use monthly averages.

A simplified query illustrating the approach is:

SELECT
    substr(date, 1, 7) AS month,
    zone,
    AVG(remplissage_pct) AS average_fill
FROM stocks_gaz
GROUP BY month, zone
ORDER BY month, zone;
Enter fullscreen mode Exit fullscreen mode

The stocks_gaz dataset distinguishes geographical zones, including France and the European Union.

Keeping the zone in the aggregation prevents measurements from different geographical scopes from being incorrectly combined.

The resulting historical view can compare the evolution of both storage series.

Electricity Generation Mix

Electricity production is also presented by generation source.

Electricity generation
    β”œβ”€β”€ Nuclear
    β”œβ”€β”€ Hydro
    β”œβ”€β”€ Wind
    β”œβ”€β”€ Solar
    β”œβ”€β”€ Gas
    β”œβ”€β”€ Bioenergy
    β”œβ”€β”€ Fuel oil
    └── Coal
Enter fullscreen mode Exit fullscreen mode

This makes it possible to inspect the composition of production rather than only its total value.

The distinction between total production, consumption, and generation mix is important because these indicators answer different questions about the energy system.

The dashboard is designed to present measured information and historical context, not to predict future energy availability.


10. Fuel Radar: Geographical Coverage and Availability

Fuel availability introduces a different analytical problem.

A national indicator can remain relatively stable while difficulties develop in specific geographical areas.

A monitoring interface therefore needs to distinguish between overall availability and the distribution of available observations.

A conceptual model could represent fuel information as:

Fuel observations
    β”œβ”€β”€ Station identifier
    β”œβ”€β”€ Geographical location
    β”œβ”€β”€ Fuel type
    β”œβ”€β”€ Availability information
    β”œβ”€β”€ Price information
    └── Observation timestamp
Enter fullscreen mode Exit fullscreen mode

This model illustrates the information categories involved rather than the exact production database structure.

Availability Is Not the Same as Reporting Coverage

An important limitation of public fuel data is that missing information does not automatically establish that a fuel is unavailable.

Several situations may exist:

Observation state Possible interpretation
Available Fuel reported as available
Unavailable Fuel reported as unavailable
Missing No usable observation was provided
Outdated Last observation may no longer apply

Treating every missing record as a confirmed shortage would produce misleading results.

Similarly, an old price or availability report should not automatically be treated as a current observation.

The Fuel Radar combines availability indicators, price information, geographical context, and historical evolution to make changes easier to examine.

The quality and freshness of the underlying source data remain essential when interpreting the results.


11. Cyber Radar: Vulnerabilities and Network Observations

Cybersecurity monitoring requires a distinction between several categories of information.

The existence of a vulnerability, the publication of an alert, and the observation of malicious network activity do not represent the same event.

The Cyber Radar includes data from the CISA Known Exploited Vulnerabilities catalog and DShield observations.

These sources provide different perspectives.

Source Information category
CISA KEV Vulnerabilities with known exploitation
DShield Reported network activity observations

The CISA KEV catalog contains vulnerabilities for which exploitation in the wild has been established according to its inclusion criteria.

The historical catalog begins in November 2021.

A monthly history of additions can therefore be represented as:

November 2021 β†’ Number of additions
December 2021 β†’ Number of additions
January 2022  β†’ Number of additions
...
Enter fullscreen mode Exit fullscreen mode

Catalog Additions Are Not Attack Counts

A vulnerability added to the KEV catalog is not equivalent to a newly observed cyberattack.

Several dates may be relevant:

  • The original vulnerability disclosure date
  • The date exploitation was observed
  • The date the vulnerability was added to the catalog
  • The date the collector retrieved the updated record

These dates must not be treated as interchangeable.

A monthly KEV history describes additions to the catalog, not the total number of attacks that occurred during that month.

DShield observations introduce another consideration: historical averages must be calculated from available measurements rather than assuming that missing observations represent zero activity.

The Cyber Radar therefore provides selected public indicators rather than attempting to replace a professional security monitoring or threat intelligence platform.


12. Maritime Traffic Radar: Monitoring Strategic Flows

Maritime traffic introduces a geographical and temporal dimension to monitoring.

A vessel movement is associated with a location, a timestamp, and potentially additional information about the vessel or route.

The Maritime Traffic Radar focuses on strategic maritime flows and their evolution.

A conceptual processing pipeline is:

Maritime observations
         β”‚
         β–Ό
Geographical selection
         β”‚
         β–Ό
Time-based grouping
         β”‚
         β–Ό
Flow indicators
         β”‚
         β–Ό
Historical comparison
Enter fullscreen mode Exit fullscreen mode

Unlike a static geographical dataset, maritime observations describe movement over time.

A useful monitoring interface must therefore distinguish between different types of measurements.

For example:

Measurement Meaning
Vessel presence Vessel observed within an area
Vessel movement Change in reported position
Passage count Recorded passages during a period
Historical flow Evolution of a selected indicator

These categories illustrate general maritime data analysis considerations.

The interpretation of maritime information also depends on the coverage of the underlying source.

A reduction in recorded activity may reflect an actual change in traffic, but it can also be influenced by reporting gaps or incomplete observations.

Monitoring strategic maritime flows requires keeping that distinction visible rather than treating every variation as a confirmed disruption.


13. Digital Infrastructure Radar: Combining Multiple Signal Types

The Digital Infrastructure Radar introduces a broader monitoring architecture.

It brings together several categories of Internet and cloud infrastructure information:

  • Internet disruption signals from IODA
  • Cloud provider incidents and service status information
  • Cloudflare component status and maintenance information
  • RIPE Atlas network measurements
  • RIPE RIS routing observations
  • DNS measurements

These sources do not measure the same phenomenon.

Digital infrastructure monitoring
    β”œβ”€β”€ Internet disruption signals
    β”œβ”€β”€ Cloud service incidents
    β”œβ”€β”€ Component status
    β”œβ”€β”€ Network measurements
    β”œβ”€β”€ Routing observations
    └── DNS measurements
Enter fullscreen mode Exit fullscreen mode

An Internet disruption signal may indicate a change in network visibility or reachability.

A cloud provider incident may describe a confirmed service degradation.

A DNS measurement may report response success or latency.

A routing observation may reflect changes in the network control plane.

These observations can complement one another, but they should not be collapsed into a single count of infrastructure failures.

Signals Are Not Confirmed Incidents

This distinction is particularly important for IODA data.

Multiple observations may relate to the same network, geographical area, or underlying disruption.

For example:

One underlying disruption
         β”‚
         β”œβ”€β”€ BGP-related signal
         β”œβ”€β”€ Active probing signal
         └── Additional network observation
Enter fullscreen mode Exit fullscreen mode

Three signals do not necessarily establish three independent incidents.

The radar therefore presents source-specific observations and historical evolution while preserving the distinction between detected signals and confirmed provider incidents.


14. Event-Oriented Storage for Infrastructure Monitoring

Infrastructure incidents require a different storage approach from numerical time series.

An incident can evolve after its initial publication.

For example:

10:00 β†’ Investigating
10:25 β†’ Identified
11:10 β†’ Monitoring
12:00 β†’ Resolved
Enter fullscreen mode Exit fullscreen mode

These updates may all refer to the same incident.

A collector must therefore recognize the incident identity while retaining the relevant information about its evolution.

A simplified event record could contain:

{
    "source": "example-provider",
    "kind": "incident",
    "external_id": "incident-001",
    "title": "Service degradation",
    "status": "monitoring",
    "started_at": "2026-11-14T10:00:00Z",
    "first_seen": "2026-11-14T10:05:00Z",
    "last_seen": "2026-11-14T11:15:00Z"
}
Enter fullscreen mode Exit fullscreen mode

This example is illustrative.

The infrastructure monitoring database uses source information, record types, external identifiers, status fields, event timestamps, payloads, hashes, and first-seen and last-seen timestamps.

These fields serve different purposes.

Field Purpose
Source Identify the originating provider
Kind Distinguish record categories
External ID Identify the source record
Status Represent the current documented state
Started at Record the reported event start
First seen Record initial collection time
Last seen Record the latest observation
Hash Help identify changes in stored data

An external identifier helps recognize a record that has already been collected.

A payload hash can help detect whether its contents have changed.

First-seen and last-seen timestamps provide collection context without replacing the original event timestamps.

This is particularly useful when historical records are retrieved retrospectively.


15. Historical Coverage and Retrospective Collection

A monitoring system can retrieve historical information after the original events occurred.

This introduces an important distinction between when an event happened and when it was collected.

Consider the following example:

Event start: November 09
First seen:  November 13
Last seen:   November 14
Enter fullscreen mode Exit fullscreen mode

The event belongs to November 9 in a historical event timeline.

However, it was not collected until November 13.

If historical collection is incomplete, the number of stored events may increase simply because the collector begins retrieving more records.

That increase does not automatically establish an increase in real-world disruptions.

Pagination and Collection Completeness

External APIs may limit the number of records returned by a single request.

A collector that retrieves only the first page may miss additional observations.

A conceptual paginated workflow is:

Request page 1
      β”‚
      β–Ό
Store returned records
      β”‚
      β–Ό
More records available?
      β”‚
      β”œβ”€β”€ Yes β†’ Request next page
      β”‚
      └── No  β†’ Finish collection
Enter fullscreen mode Exit fullscreen mode

Pagination is especially important when collecting historical event datasets or when the number of observations increases rapidly.

The infrastructure monitoring system uses paginated collection for relevant IODA requests.

Nevertheless, historical completeness depends on the available source data, requested time windows, pagination behavior, and successful collection runs.

A historical graph should therefore be interpreted according to the coverage of its underlying dataset.


16. DNS Measurements and Weighted Aggregation

The Digital Infrastructure Radar also includes DNS measurements collected through RIPE Atlas.

The measurements provide information such as response success rates and response times.

The collection uses measurement windows rather than continuously observing every DNS request.

Historical views can aggregate the available measurements into hourly, daily, weekly, or longer periods.

Why Weighted Aggregation Matters

Consider two measurement windows:

Window Measurements Successful responses Success rate
A 100 100 100%
B 900 810 90%

A simple average of the two success rates would produce:

(100% + 90%) / 2 = 95%
Enter fullscreen mode Exit fullscreen mode

But the combined success rate is:

Successful responses = 100 + 810
Total measurements   = 100 + 900

Combined success rate = 910 / 1000 Γ— 100

Combined success rate = 91%
Enter fullscreen mode Exit fullscreen mode

The difference comes from the unequal number of measurements in each window.

The DNS history uses weighted aggregation based on available sample counts where appropriate.

A conceptual calculation can be represented as:

SELECT
    period,
    SUM(success_rate * sample_count)
        / NULLIF(SUM(sample_count), 0)
        AS weighted_success_rate
FROM dns_measurements
GROUP BY period
ORDER BY period;
Enter fullscreen mode Exit fullscreen mode

This example illustrates the weighting principle rather than reproducing the production query.

Missing measurement windows should remain distinguishable from measured zero values.

A gap in DNS observations is not automatically evidence that DNS resolution failed during that period.


17. Native SVG Visualization

The dashboards use native SVG for chart rendering rather than relying on a large visualization framework.

SVG provides a coordinate system that can represent lines, points, labels, axes, and interactive overlays.

A simplified chart could be represented as:

<svg viewBox="0 0 400 180">
    <line
        x1="40"
        y1="150"
        x2="380"
        y2="150"
        stroke="currentColor"
    />

    <line
        x1="40"
        y1="20"
        x2="40"
        y2="150"
        stroke="currentColor"
    />

    <polyline
        points="40,120 120,90 200,105 280,60 380,45"
        fill="none"
        stroke="#2563eb"
        stroke-width="2"
    />
</svg>
Enter fullscreen mode Exit fullscreen mode

This example illustrates the native SVG mechanism, not the exact chart markup used by the radars.

The browser renders the graphical elements directly.

Converting Data Into Coordinates

A time-series chart requires mapping numerical values into SVG coordinates.

For a chart with a defined minimum, maximum, and drawing height, the vertical coordinate can be calculated as:

function scaleY(value, min, max, top, height) {
    if (max === min) {
        return top + height / 2;
    }

    return top + height
        - ((value - min) / (max - min)) * height;
}
Enter fullscreen mode Exit fullscreen mode

The subtraction is necessary because SVG coordinates increase downward.

A corresponding horizontal calculation can map observation indexes or timestamps into the available drawing width.

The exact horizontal scale depends on whether the chart represents evenly spaced periods or irregular observation times.

Hover Zones and Tooltips

The dashboards also use interactive hover behavior to expose additional information without permanently displaying every value.

A conceptual interaction is:

Pointer movement
       β”‚
       β–Ό
Determine nearest observation
       β”‚
       β–Ό
Update crosshair
       β”‚
       β–Ό
Display date and metric values
Enter fullscreen mode Exit fullscreen mode

The interaction can be implemented using JavaScript event listeners and SVG elements.

This allows the chart to remain visually compact while providing detailed information when the user inspects a particular point.


18. Period Selection and Missing Data

Historical interfaces must balance readability with data completeness.

Displaying every observation across several years can produce unnecessary rendering complexity.

Conversely, reducing every dataset to yearly averages may hide meaningful short-term changes.

The radars use period controls and dataset-specific aggregation to manage this trade-off.

A conceptual request could be:

Selected period: 90 days
         β”‚
         β–Ό
Retrieve relevant historical data
         β”‚
         β–Ό
Prepare appropriate time resolution
         β”‚
         β–Ό
Render updated chart
Enter fullscreen mode Exit fullscreen mode

The underlying storage does not need to change when the display period changes.

Missing Values

Missing data requires particular care in time-series visualization.

Consider:

const values = [
    72.4,
    72.1,
    null,
    71.8,
    71.5
];
Enter fullscreen mode Exit fullscreen mode

The null value represents an unavailable observation.

Replacing it with zero would introduce an artificial collapse in the series.

Connecting the surrounding points without indicating the gap could also suggest that measurements existed during the missing interval.

Depending on the chart, the interface can leave a visible gap or otherwise distinguish unavailable observations from measured values.

The appropriate behavior depends on the meaning of the dataset.


19. Bilingual Interfaces With Shared Data

The monitoring ecosystem provides French and English interfaces.

The underlying data does not need to be collected separately for each language.

Public sources
      β”‚
      β–Ό
Shared collection and storage
      β”‚
      β”œβ”€β”€ French interface
      β”‚
      └── English interface
Enter fullscreen mode Exit fullscreen mode

The localized interfaces can use the same numerical observations, timestamps, incident records, and historical series.

Only the presentation needs to change.

For example:

French label English label
Remplissage Storage level
Consommation Consumption
VulnΓ©rabilitΓ©s Vulnerabilities
Dernière observation Latest observation
DonnΓ©es disponibles Available data

JavaScript can also format numerical values according to the interface language.

const value = 72.4;

const french = new Intl.NumberFormat("fr-FR", {
    maximumFractionDigits: 1
}).format(value);

const english = new Intl.NumberFormat("en-US", {
    maximumFractionDigits: 1
}).format(value);
Enter fullscreen mode Exit fullscreen mode

The same stored value produces different textual representations:

French  β†’ 72,4
English β†’ 72.4
Enter fullscreen mode Exit fullscreen mode

Localization should not change the underlying measurement.

Keeping the data independent of the presentation language avoids duplicated collection processes and helps maintain consistency between the two interfaces.


20. Internal Usage Analytics

The monitoring ecosystem also includes a small internal analytics system.

Its purpose is to understand how the public resources are being used without introducing a third-party analytics platform.

The architecture follows a separate processing pipeline:

Public radar visits
         β”‚
         β–Ό
Minimal visit logs
         β”‚
         β–Ό
Analytics collector
         β”‚
         β”œβ”€β”€ Known bot filtering
         β”œβ”€β”€ Page view calculation
         └── Unique visitor estimation
         β”‚
         β–Ό
Daily SQLite statistics
         β”‚
         β–Ό
Private analytics dashboard
Enter fullscreen mode Exit fullscreen mode

The usage analytics system is separate from the public monitoring datasets.

It records information about resource usage rather than electricity, fuel, vulnerabilities, maritime traffic, or infrastructure conditions.

Interpreting Visitor Metrics

Page views and unique visitors describe different measurements.

A single visitor may open several radars or return to the same resource multiple times.

Bot filtering also has limitations. A system can exclude known automated traffic without guaranteeing that every remaining request represents a human visitor.

Similarly, unique visitor counts depend on the identification and aggregation method used by the analytics system.

The objective is therefore to provide useful internal usage indicators rather than claim perfectly accurate identification of every visitor.


21. Reliability and Failure Isolation

A public monitoring system depends on external sources that may occasionally become unavailable.

Possible failures include:

  • Network timeouts
  • API rate limits
  • Unexpected response formats
  • Missing observations
  • Incomplete historical retrieval
  • Changes to source schemas

A collection failure should not necessarily make the public interface unavailable.

Separating collection from rendering makes it possible to retain previously collected information while investigating the failure.

A conceptual failure-handling workflow is:

Collection attempt
        β”‚
        β”œβ”€β”€ Success
        β”‚      └── Store valid observations
        β”‚
        └── Failure
               β”œβ”€β”€ Record collection error
               β”œβ”€β”€ Preserve existing data
               └── Retry on a later execution
Enter fullscreen mode Exit fullscreen mode

The interface must still communicate the freshness of the information.

Displaying an old measurement without its observation date can create the impression that it represents the current situation.

This is particularly important for infrastructure monitoring, where the status of an incident can change rapidly.

The latest available observation and the latest successful collection are related but distinct indicators.


22. Storage Growth and Database Maintenance

A long-running monitoring system accumulates information continuously.

Even with effective deduplication, historical datasets can grow as new observations and events are collected.

The storage architecture must therefore account for:

  • The frequency of new measurements
  • The number of monitored sources
  • The size of retained event payloads
  • Historical retention requirements
  • Index maintenance
  • Database file growth

SQLite provides tools for inspecting database size and reclaiming unused storage.

For example:

PRAGMA page_count;
PRAGMA page_size;
PRAGMA freelist_count;
Enter fullscreen mode Exit fullscreen mode

These values provide information about the database's allocated pages and unused space.

SQLite also provides:

VACUUM;
Enter fullscreen mode Exit fullscreen mode

This operation rebuilds the database and can reclaim unused space.

However, VACUUM does not replace deduplication, remove unnecessary historical records, or establish a retention policy.

It also requires appropriate operational planning because it rebuilds the database and may need additional temporary disk space.

The distinction between storage maintenance and data deletion is important for monitoring systems.

Historical records may be essential to understanding long-term changes. Removing them simply to reduce database size could eliminate the analytical value of the system.


23. Security Boundaries in a Public Monitoring Architecture

A public dashboard does not need to expose its collectors, configuration files, or database files directly.

A conceptual separation is:

Private processing environment
    β”œβ”€β”€ Collector scripts
    β”œβ”€β”€ Source configuration
    β”œβ”€β”€ API credentials
    β”œβ”€β”€ SQLite databases
    └── Technical logs

Public web interface
    β”œβ”€β”€ Prepared indicators
    β”œβ”€β”€ Historical series
    β”œβ”€β”€ Charts
    └── Public data exports
Enter fullscreen mode Exit fullscreen mode

This diagram illustrates the intended architectural separation rather than exposing the production server structure.

The public interface should receive only the information required for its presentation.

External API credentials, internal logs, and private configuration remain outside that responsibility.

Imported data must also be treated as untrusted input.

A provider-supplied incident title, for example, is text that should be displayed safely rather than interpreted as HTML.

In PHP, a general output-escaping example is:

echo htmlspecialchars(
    $incidentTitle,
    ENT_QUOTES | ENT_SUBSTITUTE,
    'UTF-8'
);
Enter fullscreen mode Exit fullscreen mode

This example illustrates a standard output-escaping practice.

Similarly, SQL queries should use bound parameters when incorporating external or user-controlled values.

These are general security considerations for public monitoring applications, not a reproduction of every security mechanism used by the ecosystem.


24. What the Architecture Does Not Establish

A monitoring interface provides a representation of available data.

It does not automatically establish the cause of every observed change.

Several distinctions remain essential:

Observation Does not automatically establish
Increasing fuel unavailability A nationwide fuel shortage
Falling gas storage An imminent energy supply failure
More KEV additions A corresponding increase in attacks
More IODA signals The same number of distinct outages
Lower recorded maritime flow A confirmed physical traffic blockage
Missing DNS measurements A confirmed DNS service failure

The same principle applies to historical comparisons.

A rise in collected observations may reflect a real change, improved collection coverage, a source update, or a combination of these factors.

The purpose of the monitoring architecture is to make the information available, structured, and comparable while preserving these limitations.

A useful dashboard identifies what has changed. Establishing why it changed may require additional evidence.


Conclusion

Building a public data monitoring ecosystem involves substantially more than displaying charts.

The architecture must retrieve heterogeneous information, validate source responses, normalize records, prevent duplicate observations, preserve historical context, and select appropriate aggregation methods.

SQLite provides a compact storage layer for both numerical measurements and event-oriented datasets.

PHP, JavaScript, and native SVG provide the processing and visualization layers without requiring a large application framework.

The five monitoring domains demonstrate why a common architectural principle does not require identical data models or analytical rules.

Energy measurements, fuel availability, cybersecurity indicators, maritime observations, and digital infrastructure signals each have their own meaning and limitations.

The most important design decision is to preserve those distinctions throughout the pipeline.

A monitoring dashboard becomes useful when it can transform changing public information into readable historical context without presenting incomplete observations as established conclusions.


https://palks-studio.com

Top comments (0)