DEV Community: XRPSCAN Engineering

XRPL Deep search platform - Developer preview

XRPSCAN Engineering — Tue, 30 Jan 2024 20:19:00 +0000

The XRPScan team is pleased to announce the developer preview of Deep search platform back-end service (Platform), a data processing pipeline to store XRPL transactions in a distributed search and analytics platform.

XRP Ledger exploration and reporting tools available today (rippled, clio and various explorer APIs) provide access to ledger data based on object's primary key such as ledger index, transaction hash, account address, NFT id, object ids etc.

This project aims to provide deeper search capability such as filtering transactions by source/destination tags, range query over payment amounts, aggregate volumes and much more. This is enabled by indexing all properties of the transactions in an analytics engine.

Platform requires access to a rippled server. Optionally, it can also back-fill older ledgers by connecting to a full-history server.

Architecture

Installing platform

Platform is known to run on Linux and macOS. Please follow the installation guide to set it up on CentOS.

Running the service

Run bin/platform-server to start the service. It will subscribe to your rippled server and start indexing transactions, ledgers and validations.

Back-filling data

./bin/platform-cli backfill -verbose -from 82000000 -to 82999999

Optionally, a custom full-history server URL may be provided via the command line

./bin/platform-cli backfill -verbose -from 82000000 -to 82999999 -server wss://xrplcluster.com

Searching transactions

A front-end for searching transactions would be available in the future. In this developer preview, transactions can be searched by connecting to Elastisearch directly.

curl -k -u elastic:$ELASTICSEARCH_PASSWORD \
-H 'Content-type: application/json' \
-XPOST 'https://localhost:9200/platform.transactions/_search' \
-d '{"query":{"term":{"ctid":"C511CC0400850000" }}}' \
| jq .hits.hits

Response:

[
  {
    "_index": "platform.transactions-85m",
    "_id": "E7B699BA359C4E9E48D64DE17D2914BF97C922B32BF03493A1B95C48BC51CB8D",
    "_score": 15.314049,
    "_source": {
      "Account": "rMdG3ju8pgyVh29ELPWaDuA74CpWW6Fxns",
      "Amount": {
        "currency": "XRP",
        "value": 17805999000
      },
      "Destination": "rsyncTM4ojq33ZVJeBmDXvvJibVftJ2sPy",
      "Fee": 10,
      "Sequence": 3365263,
      "SigningPubKey": "0255EECA852E7C26C0219F0792D1229F1147366D4C936FF3ED83AC32354F6F8EF3",
      "TransactionType": "Payment",
      "TxnSignature": "304402203C4CF26B45FFB8DBB357421AD5E75C38C3CD3D5E290D2E3B0EA9F06548970904022063841E568F02BAE096775017EC7FFD9A12BEE2CD5858650D0870ECF491DD6B94",
      "ctid": "C511CC0400850000",
      "date": 757688782,
      "hash": "E7B699BA359C4E9E48D64DE17D2914BF97C922B32BF03493A1B95C48BC51CB8D",
      "ledger_index": 85052420,
      "meta": {
        "TransactionIndex": 133,
        "TransactionResult": "tesSUCCESS",
        "delivered_amount": {
          "currency": "XRP",
          "value": 17805999000
        }
      },
      "validated": true
    }
  }
]

Share your feedback

We invite you to run Deep search platform and report your observations, bugs and feature requests. The project is available at https://github.com/xrpscan/platform

Online Delete support is now available in Hermes

XRPSCAN Engineering — Wed, 08 Feb 2023 15:12:37 +0000

TLDR: Online Delete support is now available in Hermes v0.4.1

Hermes is a layer 2 messaging network for storing and relaying XRPL validation messages. Since its release in Aug 2022, several operators have volunteered to run Hermes nodes and participate in the P2P network.

XRPL validators generate millions of validation messages collectively every day. The online deletion feature lets the service delete validation messages from older ledgers to keep disk usage from rapidly growing over time. The default config lets the service store all validation messages indefinitely. In order to enable online deletion, set ONLINE_DELETE setting in .env file to the number of latest validated ledgers you wish to keep.

To store validations from the last 10 days approx, use:

ONLINE_DELETE = 250000

or, to store validations from the last 1 month approx, use:

ONLINE_DELETE = 660000

If needed, MongoDB collection may be compacted to free up disk space and return it back to the OS.

$ npm run online-delete compact
[online-delete] Compact validations collection: 344 MB freed

The XRPScan team is committed to running a long term Hermes node and will add disk capacity if possible. As of today, we store more than 676 Million validation messages and are happy to serve anyone who wishes to peer.

However with this addition we hope the node operators would be able to have better control over their resource usage and associated costs.

Hermes: XRPL Validation message service

XRPSCAN Engineering — Sat, 20 Aug 2022 15:39:00 +0000

XRPScan is one of the most popular XRPL explorer and network analysis platform. Our Amendments and Validators registry is widely used by the community to keep track of the state of the XRP Ledger.

Validator agreement data is generated on the basis of validations emitted by the validators. However, these validations are not on-ledger. There are a few sources that track validator agreement, but none that we're aware of are 100% resilient or supported.

Introducing Hermes (Beta)

Hermes is a layer 2 messaging network for storing and relaying XRPL validation messages. Hermes servers can be linked together to create a P2P network to spool validationReceived messages emitted by rippled.

Validation messages are ephemeral, therefore they must be saved as soon as they're emitted. If a service processing XRPL Validation messages need to be momentarily offline (due to an outage, service restart or upgrade), it can fetch missing Validation messages from the Hermes network. Hermes server offers data access via REST and gRPC endpoints.

Access to historical validation messages opens the possibility of evaluating validators for their robustness or for inclusion/exclusion from UNLs.

Architecture

Installation

Hermes server is tested on Linux. It needs OpenSSL binaries, MongoDB, access to rippled and Node.js runtime.

Please follow installation guide to run Hermes locally and peer with other nodes.

Recommendations

Assuming 3.85 seconds ledger interval time, polling every 1 hour and fetching 2000 ledger validations may provide a good balance between data coverage and network traffic.

PEERSYNC_ENABLED = true
PEERSYNC_POLL_INTERVAL = 3600
PEERSYNC_FETCH_DEPTH = 2000

Performance test

XRPL closes roughly 21,500 ledgers a day. To do this, it generates more than 3.4 Million (mainnet and other nets) validation messages every day. We performed a simple test on our test server to measure how much time it takes to sync mainnet validation messages from the p2p network.

Range	Ledgers	Validations	Time
1 hour	898	117,021	0m3.051s
1 day	21,501	2,791,281	1m1.338s
all	391,058	34,322,489	13m11.065s

Beta participants

Following Hermes servers are available for beta testing. If you're willing to run an instance and would like to peer with us, use this endpoint:

vms.test.xrpscan.com

What can be improved

Hermes servers collect validation messages directly from the rippled servers they're connected to, therefore a lot of peersync response contains duplicate validation messages. To optimize data transferred over the p2p network, Hermes would need to add a de-duplication strategy at the request time.

Conclusion

We hope this project provides a way to persist validation messages and allow granular access to validator performance data. Our goal is to provide high-quality validation data, so the network operators can make objective decisions about their UNLs.

If you're running a Hermes server, we'd love to peer with your node. Please contact us and we'd get peering.

Thank you

Thank you to XPRL Grants for sponsoring this project. A special thank you to XRPL Foundation for providing testing infrastructure, and to nixerFFM for independently running various Hermes prototypes.