Public data is available across a growing number of platforms, APIs, and institutional sources.
Electricity production, gas storage, fuel availability, cybersecurity vulnerabilities, maritime traffic, and Internet infrastructure can all be monitored using publicly accessible information.
However, accessing the data is only one part of the problem.
Different sources expose different formats, update at different frequencies, and describe different types of events. Historical information may be difficult to retrieve, and raw measurements do not necessarily provide a useful representation of the current situation.
The public data monitoring ecosystem developed by Palks Studio addresses this problem through five independent dashboards built around automated collection, local data storage, historical analysis, and native web visualization.
The technical objective is to transform heterogeneous public datasets into consistent, readable monitoring interfaces without requiring a large application framework or a centralized analytics platform.
1. Five Monitoring Domains, One Architectural Principle
The ecosystem currently covers five areas:
| Monitoring domain | Primary data categories |
|---|---|
| Energy | Electricity, gas storage, oil stocks |
| Fuel availability | Fuel availability, prices, geographical data |
| Cybersecurity | Exploited vulnerabilities, network observations |
| Maritime traffic | Vessel movements, strategic maritime flows |
| Digital infrastructure | Internet signals, cloud incidents, DNS, routing |
Each domain has its own data sources, collection requirements, and interpretation rules.
An electricity measurement cannot be processed in exactly the same way as a cybersecurity vulnerability or a cloud service incident.
However, the underlying architecture follows a common principle:
Public data sources
β
βΌ
Collection
β
βΌ
Validation
β
βΌ
Normalization
β
βΌ
Storage
β
βΌ
Aggregation
β
βΌ
Visualization
The visible dashboard is the final layer of the system.
Most of the technical complexity exists before the data reaches the interface.
2. Separating Collection From Visualization
A monitoring dashboard should not necessarily retrieve every external dataset whenever a user opens the page.
Doing so would make the interface dependent on the availability, latency, and response time of multiple external services.
A more controlled architecture separates the collection process from the public interface.
Scheduled collectors
β
βΌ
External data sources
β
βΌ
Local data storage
β
βΌ
PHP application layer
β
βΌ
HTML / CSS / JavaScript / SVG
The collectors retrieve information independently of page requests.
The public interface reads the available stored data and prepares the indicators, tables, and historical series required for visualization.
This separation provides several architectural advantages:
- External API latency does not directly determine page rendering time
- Historical information can remain available when a source becomes temporarily inaccessible
- Data can be normalized before reaching the presentation layer
- Collection and visualization can evolve independently
- Different interfaces can reuse the same underlying datasets
The ecosystem uses scheduled collection processes, SQLite storage, PHP processing, and native browser technologies.
The public dashboards are not responsible for performing the complete collection pipeline on every request.
3. Scheduled Collection With Cron
Different data sources do not update at the same frequency.
An electricity dataset may provide daily or more frequent measurements. A vulnerability catalog may change only when a new entry is published. An Internet monitoring source may expose observations across short time windows.
The collection schedule must therefore reflect the characteristics of each source.
A conceptual collection workflow is:
Cron
β
βΌ
Collector execution
β
βββ Request source data
βββ Validate response
βββ Normalize records
βββ Store new information
βββ Record execution status
A scheduled collector can run independently of user activity.
For example, a generic PHP collector could be scheduled every hour using:
0 * * * * /usr/bin/php /path/to/collector.php
This command is an educational example of cron scheduling. It does not reproduce a production command or server path.
Collection Frequency Is Not Data Frequency
Running a collector every hour does not mean that its source publishes new information every hour.
Consider a source that publishes one measurement per day.
| Collection time | Source measurement date | New observation |
|---|---|---|
| 08:00 | November 14 | Yes |
| 09:00 | November 14 | No |
| 10:00 | November 14 | No |
| 11:00 | November 15 | Yes |
The collection process may execute several times before a new measurement becomes available.
This is why collection timestamps and source timestamps must remain distinct.
Otherwise, repeated retrievals of the same information could be incorrectly interpreted as new observations.
4. Normalizing Heterogeneous Data
Public datasets do not necessarily use the same structure.
One source may return JSON, another may expose CSV files, and another may provide an event-oriented API.
Even when two sources return JSON, their field names, timestamps, units, and identifiers may differ.
A conceptual example illustrates the problem.
Source A:
{
"date": "2026-11-14",
"value": 72.4,
"unit": "percent"
}
Source B:
{
"timestamp": "2026-11-14T00:00:00Z",
"measurement": 0.724
}
Both records could represent the same percentage, but their representations are different.
A normalized record could be:
{
"source": "example-source",
"observed_at": "2026-11-14T00:00:00Z",
"metric": "storage_level",
"value": 72.4,
"unit": "percent"
}
These records are illustrative and do not reproduce an actual external API response or database schema.
Normalization can involve several operations:
| Operation | Purpose |
|---|---|
| Timestamp conversion | Establish a consistent time basis |
| Unit conversion | Preserve comparable measurements |
| Field mapping | Standardize source-specific names |
| Type conversion | Convert textual values into numbers |
| Identifier handling | Recognize repeated observations |
| Source attribution | Preserve the origin of information |
Normalization does not mean forcing every dataset into a single universal structure.
A numerical time series and an incident record may require different storage models.
The important principle is consistency within each analytical domain.
5. SQLite as a Storage Layer
The monitoring ecosystem uses SQLite to retain collected information and historical datasets.
SQLite is particularly useful for applications where the storage requirements are primarily local and the workload consists of periodic writes followed by analytical reads.
A monitoring database can contain several types of information:
SQLite database
βββ Latest observations
βββ Historical measurements
βββ Vulnerability records
βββ Incident records
βββ Source metadata
βββ Collection information
The exact organization depends on the monitoring domain.
For example, gas storage measurements are naturally represented as dated numerical observations.
A cloud incident requires a different structure because it may contain an identifier, title, status, start time, and subsequent updates.
A Time-Series Storage Example
A simplified SQLite table for numerical observations could be defined as:
CREATE TABLE measurements (
id INTEGER PRIMARY KEY,
source TEXT NOT NULL,
metric TEXT NOT NULL,
observed_at TEXT NOT NULL,
value REAL NOT NULL,
unit TEXT NOT NULL,
UNIQUE (source, metric, observed_at)
);
This is an educational schema, not the production schema of the radars.
The unique constraint prevents the same source, metric, and observation timestamp from being inserted multiple times.
An index can support historical queries:
CREATE INDEX idx_measurements_history
ON measurements (
source,
metric,
observed_at
);
A query can then retrieve a specific historical series without scanning unrelated measurements.
SQLite provides a relatively small storage layer while still supporting indexes, constraints, transactions, and SQL aggregation.
6. Deduplication and Repeated Collection
Repeated collection introduces an important problem: the same observation may be retrieved multiple times.
A collector must distinguish between:
- A genuinely new measurement
- An observation already stored
- An existing observation that has been corrected
- An existing incident whose status has changed
These cases require different handling.
For numerical observations, a combination of source, metric, and timestamp can often provide a stable identity.
For incidents or vulnerability records, an identifier supplied by the source may be more appropriate.
A conceptual SQLite insertion could use:
INSERT INTO measurements (
source,
metric,
observed_at,
value,
unit
)
VALUES (
:source,
:metric,
:observed_at,
:value,
:unit
)
ON CONFLICT (
source,
metric,
observed_at
)
DO NOTHING;
This approach ignores repeated observations with the same identity.
However, ignoring duplicates is not always sufficient.
If a source corrects a previously published measurement, an update strategy may be required:
ON CONFLICT (
source,
metric,
observed_at
)
DO UPDATE SET
value = excluded.value,
unit = excluded.unit;
The appropriate strategy depends on the meaning of the record.
An immutable historical event should not necessarily be overwritten in the same way as a corrected numerical measurement.
The radars include anti-duplicate mechanisms, but the SQL examples above illustrate general storage strategies rather than reproducing the implementation of every collector.
7. Managing Historical Data
A current measurement provides limited context without its historical evolution.
For example:
Gas storage: 72%
The value becomes more useful when compared with earlier measurements.
30 days ago : 81%
7 days ago : 76%
Yesterday : 73%
Today : 72%
The historical series reveals a direction of change that is not visible in the latest value alone.
However, retaining historical information creates another challenge: the amount of data that must be processed and displayed increases over time.
A dashboard covering several years does not necessarily need to render every individual measurement.
The monitoring interfaces therefore use different temporal resolutions depending on the selected period and dataset.
| Time range | Possible display resolution |
|---|---|
| Short term | Individual or hourly records |
| 30 days | Daily observations |
| 90 days | Daily observations |
| One year | Daily or aggregated values |
| All time | Monthly or yearly aggregates |
The exact resolution varies across the radars.
The purpose of aggregation is to reduce the number of displayed points without removing the historical information from storage.
8. Temporal Aggregation With SQL
SQL can transform detailed measurements into historical summaries before the data reaches the browser.
For example, a monthly average can be calculated from daily observations:
SELECT
substr(observed_at, 1, 7) AS month,
AVG(value) AS average_value,
COUNT(*) AS sample_count
FROM measurements
WHERE metric = :metric
GROUP BY month
ORDER BY month;
The resulting series contains one record per month instead of one record per observation.
Daily measurements
β
βββ November 01
βββ November 02
βββ November 03
βββ ...
βββ November 30
β
βΌ
Monthly average
The sample_count field is useful because a monthly average calculated from three observations is not equivalent in coverage to one calculated from thirty observations.
Aggregation Depends on the Metric
Not every historical series should use the same aggregation function.
| Metric type | Relevant aggregation |
|---|---|
| Storage percentage | Average over available observations |
| New vulnerabilities | Count of additions during a period |
| Event records | Count of distinct events or signals |
| Peak measurement | Maximum within a period |
| Latest observation | Most recent valid value |
A monthly count of newly published vulnerabilities has a different meaning from a monthly average gas storage level.
The aggregation method must preserve the meaning of the underlying data.
For example, a power measurement expressed in megawatts should not simply be summed to calculate energy consumption in megawatt-hours. That calculation requires accounting for the duration represented by each measurement.
9. Energy Radar: Combining Different Time Series
The Energy Radar brings together information relating to electricity, gas, and oil stocks.
Its datasets include:
- Electricity consumption and production
- Electricity generation mix
- Gas consumption
- Gas storage levels in France and the European Union
- Historical oil stocks
These datasets represent different physical quantities.
Electricity production may be expressed as power, gas storage as a percentage of capacity, and oil stocks as a physical quantity.
Combining them into a single numerical score would remove important distinctions.
Instead, the interface presents separate indicators and historical series.
Gas Storage History
Gas storage provides a useful example of temporal aggregation.
Daily measurements can be retained for recent periods, while a longer historical view can use monthly averages.
A simplified query illustrating the approach is:
SELECT
substr(date, 1, 7) AS month,
zone,
AVG(remplissage_pct) AS average_fill
FROM stocks_gaz
GROUP BY month, zone
ORDER BY month, zone;
The stocks_gaz dataset distinguishes geographical zones, including France and the European Union.
Keeping the zone in the aggregation prevents measurements from different geographical scopes from being incorrectly combined.
The resulting historical view can compare the evolution of both storage series.
Electricity Generation Mix
Electricity production is also presented by generation source.
Electricity generation
βββ Nuclear
βββ Hydro
βββ Wind
βββ Solar
βββ Gas
βββ Bioenergy
βββ Fuel oil
βββ Coal
This makes it possible to inspect the composition of production rather than only its total value.
The distinction between total production, consumption, and generation mix is important because these indicators answer different questions about the energy system.
The dashboard is designed to present measured information and historical context, not to predict future energy availability.
10. Fuel Radar: Geographical Coverage and Availability
Fuel availability introduces a different analytical problem.
A national indicator can remain relatively stable while difficulties develop in specific geographical areas.
A monitoring interface therefore needs to distinguish between overall availability and the distribution of available observations.
A conceptual model could represent fuel information as:
Fuel observations
βββ Station identifier
βββ Geographical location
βββ Fuel type
βββ Availability information
βββ Price information
βββ Observation timestamp
This model illustrates the information categories involved rather than the exact production database structure.
Availability Is Not the Same as Reporting Coverage
An important limitation of public fuel data is that missing information does not automatically establish that a fuel is unavailable.
Several situations may exist:
| Observation state | Possible interpretation |
|---|---|
| Available | Fuel reported as available |
| Unavailable | Fuel reported as unavailable |
| Missing | No usable observation was provided |
| Outdated | Last observation may no longer apply |
Treating every missing record as a confirmed shortage would produce misleading results.
Similarly, an old price or availability report should not automatically be treated as a current observation.
The Fuel Radar combines availability indicators, price information, geographical context, and historical evolution to make changes easier to examine.
The quality and freshness of the underlying source data remain essential when interpreting the results.
11. Cyber Radar: Vulnerabilities and Network Observations
Cybersecurity monitoring requires a distinction between several categories of information.
The existence of a vulnerability, the publication of an alert, and the observation of malicious network activity do not represent the same event.
The Cyber Radar includes data from the CISA Known Exploited Vulnerabilities catalog and DShield observations.
These sources provide different perspectives.
| Source | Information category |
|---|---|
| CISA KEV | Vulnerabilities with known exploitation |
| DShield | Reported network activity observations |
The CISA KEV catalog contains vulnerabilities for which exploitation in the wild has been established according to its inclusion criteria.
The historical catalog begins in November 2021.
A monthly history of additions can therefore be represented as:
November 2021 β Number of additions
December 2021 β Number of additions
January 2022 β Number of additions
...
Catalog Additions Are Not Attack Counts
A vulnerability added to the KEV catalog is not equivalent to a newly observed cyberattack.
Several dates may be relevant:
- The original vulnerability disclosure date
- The date exploitation was observed
- The date the vulnerability was added to the catalog
- The date the collector retrieved the updated record
These dates must not be treated as interchangeable.
A monthly KEV history describes additions to the catalog, not the total number of attacks that occurred during that month.
DShield observations introduce another consideration: historical averages must be calculated from available measurements rather than assuming that missing observations represent zero activity.
The Cyber Radar therefore provides selected public indicators rather than attempting to replace a professional security monitoring or threat intelligence platform.
12. Maritime Traffic Radar: Monitoring Strategic Flows
Maritime traffic introduces a geographical and temporal dimension to monitoring.
A vessel movement is associated with a location, a timestamp, and potentially additional information about the vessel or route.
The Maritime Traffic Radar focuses on strategic maritime flows and their evolution.
A conceptual processing pipeline is:
Maritime observations
β
βΌ
Geographical selection
β
βΌ
Time-based grouping
β
βΌ
Flow indicators
β
βΌ
Historical comparison
Unlike a static geographical dataset, maritime observations describe movement over time.
A useful monitoring interface must therefore distinguish between different types of measurements.
For example:
| Measurement | Meaning |
|---|---|
| Vessel presence | Vessel observed within an area |
| Vessel movement | Change in reported position |
| Passage count | Recorded passages during a period |
| Historical flow | Evolution of a selected indicator |
These categories illustrate general maritime data analysis considerations.
The interpretation of maritime information also depends on the coverage of the underlying source.
A reduction in recorded activity may reflect an actual change in traffic, but it can also be influenced by reporting gaps or incomplete observations.
Monitoring strategic maritime flows requires keeping that distinction visible rather than treating every variation as a confirmed disruption.
13. Digital Infrastructure Radar: Combining Multiple Signal Types
The Digital Infrastructure Radar introduces a broader monitoring architecture.
It brings together several categories of Internet and cloud infrastructure information:
- Internet disruption signals from IODA
- Cloud provider incidents and service status information
- Cloudflare component status and maintenance information
- RIPE Atlas network measurements
- RIPE RIS routing observations
- DNS measurements
These sources do not measure the same phenomenon.
Digital infrastructure monitoring
βββ Internet disruption signals
βββ Cloud service incidents
βββ Component status
βββ Network measurements
βββ Routing observations
βββ DNS measurements
An Internet disruption signal may indicate a change in network visibility or reachability.
A cloud provider incident may describe a confirmed service degradation.
A DNS measurement may report response success or latency.
A routing observation may reflect changes in the network control plane.
These observations can complement one another, but they should not be collapsed into a single count of infrastructure failures.
Signals Are Not Confirmed Incidents
This distinction is particularly important for IODA data.
Multiple observations may relate to the same network, geographical area, or underlying disruption.
For example:
One underlying disruption
β
βββ BGP-related signal
βββ Active probing signal
βββ Additional network observation
Three signals do not necessarily establish three independent incidents.
The radar therefore presents source-specific observations and historical evolution while preserving the distinction between detected signals and confirmed provider incidents.
14. Event-Oriented Storage for Infrastructure Monitoring
Infrastructure incidents require a different storage approach from numerical time series.
An incident can evolve after its initial publication.
For example:
10:00 β Investigating
10:25 β Identified
11:10 β Monitoring
12:00 β Resolved
These updates may all refer to the same incident.
A collector must therefore recognize the incident identity while retaining the relevant information about its evolution.
A simplified event record could contain:
{
"source": "example-provider",
"kind": "incident",
"external_id": "incident-001",
"title": "Service degradation",
"status": "monitoring",
"started_at": "2026-11-14T10:00:00Z",
"first_seen": "2026-11-14T10:05:00Z",
"last_seen": "2026-11-14T11:15:00Z"
}
This example is illustrative.
The infrastructure monitoring database uses source information, record types, external identifiers, status fields, event timestamps, payloads, hashes, and first-seen and last-seen timestamps.
These fields serve different purposes.
| Field | Purpose |
|---|---|
| Source | Identify the originating provider |
| Kind | Distinguish record categories |
| External ID | Identify the source record |
| Status | Represent the current documented state |
| Started at | Record the reported event start |
| First seen | Record initial collection time |
| Last seen | Record the latest observation |
| Hash | Help identify changes in stored data |
An external identifier helps recognize a record that has already been collected.
A payload hash can help detect whether its contents have changed.
First-seen and last-seen timestamps provide collection context without replacing the original event timestamps.
This is particularly useful when historical records are retrieved retrospectively.
15. Historical Coverage and Retrospective Collection
A monitoring system can retrieve historical information after the original events occurred.
This introduces an important distinction between when an event happened and when it was collected.
Consider the following example:
Event start: November 09
First seen: November 13
Last seen: November 14
The event belongs to November 9 in a historical event timeline.
However, it was not collected until November 13.
If historical collection is incomplete, the number of stored events may increase simply because the collector begins retrieving more records.
That increase does not automatically establish an increase in real-world disruptions.
Pagination and Collection Completeness
External APIs may limit the number of records returned by a single request.
A collector that retrieves only the first page may miss additional observations.
A conceptual paginated workflow is:
Request page 1
β
βΌ
Store returned records
β
βΌ
More records available?
β
βββ Yes β Request next page
β
βββ No β Finish collection
Pagination is especially important when collecting historical event datasets or when the number of observations increases rapidly.
The infrastructure monitoring system uses paginated collection for relevant IODA requests.
Nevertheless, historical completeness depends on the available source data, requested time windows, pagination behavior, and successful collection runs.
A historical graph should therefore be interpreted according to the coverage of its underlying dataset.
16. DNS Measurements and Weighted Aggregation
The Digital Infrastructure Radar also includes DNS measurements collected through RIPE Atlas.
The measurements provide information such as response success rates and response times.
The collection uses measurement windows rather than continuously observing every DNS request.
Historical views can aggregate the available measurements into hourly, daily, weekly, or longer periods.
Why Weighted Aggregation Matters
Consider two measurement windows:
| Window | Measurements | Successful responses | Success rate |
|---|---|---|---|
| A | 100 | 100 | 100% |
| B | 900 | 810 | 90% |
A simple average of the two success rates would produce:
(100% + 90%) / 2 = 95%
But the combined success rate is:
Successful responses = 100 + 810
Total measurements = 100 + 900
Combined success rate = 910 / 1000 Γ 100
Combined success rate = 91%
The difference comes from the unequal number of measurements in each window.
The DNS history uses weighted aggregation based on available sample counts where appropriate.
A conceptual calculation can be represented as:
SELECT
period,
SUM(success_rate * sample_count)
/ NULLIF(SUM(sample_count), 0)
AS weighted_success_rate
FROM dns_measurements
GROUP BY period
ORDER BY period;
This example illustrates the weighting principle rather than reproducing the production query.
Missing measurement windows should remain distinguishable from measured zero values.
A gap in DNS observations is not automatically evidence that DNS resolution failed during that period.
17. Native SVG Visualization
The dashboards use native SVG for chart rendering rather than relying on a large visualization framework.
SVG provides a coordinate system that can represent lines, points, labels, axes, and interactive overlays.
A simplified chart could be represented as:
<svg viewBox="0 0 400 180">
<line
x1="40"
y1="150"
x2="380"
y2="150"
stroke="currentColor"
/>
<line
x1="40"
y1="20"
x2="40"
y2="150"
stroke="currentColor"
/>
<polyline
points="40,120 120,90 200,105 280,60 380,45"
fill="none"
stroke="#2563eb"
stroke-width="2"
/>
</svg>
This example illustrates the native SVG mechanism, not the exact chart markup used by the radars.
The browser renders the graphical elements directly.
Converting Data Into Coordinates
A time-series chart requires mapping numerical values into SVG coordinates.
For a chart with a defined minimum, maximum, and drawing height, the vertical coordinate can be calculated as:
function scaleY(value, min, max, top, height) {
if (max === min) {
return top + height / 2;
}
return top + height
- ((value - min) / (max - min)) * height;
}
The subtraction is necessary because SVG coordinates increase downward.
A corresponding horizontal calculation can map observation indexes or timestamps into the available drawing width.
The exact horizontal scale depends on whether the chart represents evenly spaced periods or irregular observation times.
Hover Zones and Tooltips
The dashboards also use interactive hover behavior to expose additional information without permanently displaying every value.
A conceptual interaction is:
Pointer movement
β
βΌ
Determine nearest observation
β
βΌ
Update crosshair
β
βΌ
Display date and metric values
The interaction can be implemented using JavaScript event listeners and SVG elements.
This allows the chart to remain visually compact while providing detailed information when the user inspects a particular point.
18. Period Selection and Missing Data
Historical interfaces must balance readability with data completeness.
Displaying every observation across several years can produce unnecessary rendering complexity.
Conversely, reducing every dataset to yearly averages may hide meaningful short-term changes.
The radars use period controls and dataset-specific aggregation to manage this trade-off.
A conceptual request could be:
Selected period: 90 days
β
βΌ
Retrieve relevant historical data
β
βΌ
Prepare appropriate time resolution
β
βΌ
Render updated chart
The underlying storage does not need to change when the display period changes.
Missing Values
Missing data requires particular care in time-series visualization.
Consider:
const values = [
72.4,
72.1,
null,
71.8,
71.5
];
The null value represents an unavailable observation.
Replacing it with zero would introduce an artificial collapse in the series.
Connecting the surrounding points without indicating the gap could also suggest that measurements existed during the missing interval.
Depending on the chart, the interface can leave a visible gap or otherwise distinguish unavailable observations from measured values.
The appropriate behavior depends on the meaning of the dataset.
19. Bilingual Interfaces With Shared Data
The monitoring ecosystem provides French and English interfaces.
The underlying data does not need to be collected separately for each language.
Public sources
β
βΌ
Shared collection and storage
β
βββ French interface
β
βββ English interface
The localized interfaces can use the same numerical observations, timestamps, incident records, and historical series.
Only the presentation needs to change.
For example:
| French label | English label |
|---|---|
| Remplissage | Storage level |
| Consommation | Consumption |
| VulnΓ©rabilitΓ©s | Vulnerabilities |
| Dernière observation | Latest observation |
| DonnΓ©es disponibles | Available data |
JavaScript can also format numerical values according to the interface language.
const value = 72.4;
const french = new Intl.NumberFormat("fr-FR", {
maximumFractionDigits: 1
}).format(value);
const english = new Intl.NumberFormat("en-US", {
maximumFractionDigits: 1
}).format(value);
The same stored value produces different textual representations:
French β 72,4
English β 72.4
Localization should not change the underlying measurement.
Keeping the data independent of the presentation language avoids duplicated collection processes and helps maintain consistency between the two interfaces.
20. Internal Usage Analytics
The monitoring ecosystem also includes a small internal analytics system.
Its purpose is to understand how the public resources are being used without introducing a third-party analytics platform.
The architecture follows a separate processing pipeline:
Public radar visits
β
βΌ
Minimal visit logs
β
βΌ
Analytics collector
β
βββ Known bot filtering
βββ Page view calculation
βββ Unique visitor estimation
β
βΌ
Daily SQLite statistics
β
βΌ
Private analytics dashboard
The usage analytics system is separate from the public monitoring datasets.
It records information about resource usage rather than electricity, fuel, vulnerabilities, maritime traffic, or infrastructure conditions.
Interpreting Visitor Metrics
Page views and unique visitors describe different measurements.
A single visitor may open several radars or return to the same resource multiple times.
Bot filtering also has limitations. A system can exclude known automated traffic without guaranteeing that every remaining request represents a human visitor.
Similarly, unique visitor counts depend on the identification and aggregation method used by the analytics system.
The objective is therefore to provide useful internal usage indicators rather than claim perfectly accurate identification of every visitor.
21. Reliability and Failure Isolation
A public monitoring system depends on external sources that may occasionally become unavailable.
Possible failures include:
- Network timeouts
- API rate limits
- Unexpected response formats
- Missing observations
- Incomplete historical retrieval
- Changes to source schemas
A collection failure should not necessarily make the public interface unavailable.
Separating collection from rendering makes it possible to retain previously collected information while investigating the failure.
A conceptual failure-handling workflow is:
Collection attempt
β
βββ Success
β βββ Store valid observations
β
βββ Failure
βββ Record collection error
βββ Preserve existing data
βββ Retry on a later execution
The interface must still communicate the freshness of the information.
Displaying an old measurement without its observation date can create the impression that it represents the current situation.
This is particularly important for infrastructure monitoring, where the status of an incident can change rapidly.
The latest available observation and the latest successful collection are related but distinct indicators.
22. Storage Growth and Database Maintenance
A long-running monitoring system accumulates information continuously.
Even with effective deduplication, historical datasets can grow as new observations and events are collected.
The storage architecture must therefore account for:
- The frequency of new measurements
- The number of monitored sources
- The size of retained event payloads
- Historical retention requirements
- Index maintenance
- Database file growth
SQLite provides tools for inspecting database size and reclaiming unused storage.
For example:
PRAGMA page_count;
PRAGMA page_size;
PRAGMA freelist_count;
These values provide information about the database's allocated pages and unused space.
SQLite also provides:
VACUUM;
This operation rebuilds the database and can reclaim unused space.
However, VACUUM does not replace deduplication, remove unnecessary historical records, or establish a retention policy.
It also requires appropriate operational planning because it rebuilds the database and may need additional temporary disk space.
The distinction between storage maintenance and data deletion is important for monitoring systems.
Historical records may be essential to understanding long-term changes. Removing them simply to reduce database size could eliminate the analytical value of the system.
23. Security Boundaries in a Public Monitoring Architecture
A public dashboard does not need to expose its collectors, configuration files, or database files directly.
A conceptual separation is:
Private processing environment
βββ Collector scripts
βββ Source configuration
βββ API credentials
βββ SQLite databases
βββ Technical logs
Public web interface
βββ Prepared indicators
βββ Historical series
βββ Charts
βββ Public data exports
This diagram illustrates the intended architectural separation rather than exposing the production server structure.
The public interface should receive only the information required for its presentation.
External API credentials, internal logs, and private configuration remain outside that responsibility.
Imported data must also be treated as untrusted input.
A provider-supplied incident title, for example, is text that should be displayed safely rather than interpreted as HTML.
In PHP, a general output-escaping example is:
echo htmlspecialchars(
$incidentTitle,
ENT_QUOTES | ENT_SUBSTITUTE,
'UTF-8'
);
This example illustrates a standard output-escaping practice.
Similarly, SQL queries should use bound parameters when incorporating external or user-controlled values.
These are general security considerations for public monitoring applications, not a reproduction of every security mechanism used by the ecosystem.
24. What the Architecture Does Not Establish
A monitoring interface provides a representation of available data.
It does not automatically establish the cause of every observed change.
Several distinctions remain essential:
| Observation | Does not automatically establish |
|---|---|
| Increasing fuel unavailability | A nationwide fuel shortage |
| Falling gas storage | An imminent energy supply failure |
| More KEV additions | A corresponding increase in attacks |
| More IODA signals | The same number of distinct outages |
| Lower recorded maritime flow | A confirmed physical traffic blockage |
| Missing DNS measurements | A confirmed DNS service failure |
The same principle applies to historical comparisons.
A rise in collected observations may reflect a real change, improved collection coverage, a source update, or a combination of these factors.
The purpose of the monitoring architecture is to make the information available, structured, and comparable while preserving these limitations.
A useful dashboard identifies what has changed. Establishing why it changed may require additional evidence.
Conclusion
Building a public data monitoring ecosystem involves substantially more than displaying charts.
The architecture must retrieve heterogeneous information, validate source responses, normalize records, prevent duplicate observations, preserve historical context, and select appropriate aggregation methods.
SQLite provides a compact storage layer for both numerical measurements and event-oriented datasets.
PHP, JavaScript, and native SVG provide the processing and visualization layers without requiring a large application framework.
The five monitoring domains demonstrate why a common architectural principle does not require identical data models or analytical rules.
Energy measurements, fuel availability, cybersecurity indicators, maritime observations, and digital infrastructure signals each have their own meaning and limitations.
The most important design decision is to preserve those distinctions throughout the pipeline.
A monitoring dashboard becomes useful when it can transform changing public information into readable historical context without presenting incomplete observations as established conclusions.

Top comments (0)