SLPDB v1.0 Beta Released

Feb 24, 2020 | Featured

We’re excited to announce SLPDB v1.0 beta is released!  The SLP developer community has been very supportive during this journey of building SLPDB, so we want to reflect on the accomplishments made, present our vision for the future of SLPDB, and recognize the key contributors.  First, let’s take a moment to review why SLPDB is a unique and critical piece of infrastructure within the SLP ecosystem.

SLPDB Features

    • Fast Validation: SLPDB v1.0 can validate any SLP token transaction almost instantly using a simple DB query or by subscribing to realtime server-side events (SSE) notifications.
    • Realtime Notifications: SLPDB publishes ZMQ notifications for all valid SLP transactions and blocks, these ZMQ notifications can be accessed via HTTP SSE using the SlpSockServe repository.
    • Token Holder Lists for Airdrops: SLPDB provides the ability to get a list of token holders at any point in time so users can airdrop rewards to token holders pro-rata.
    • Open Telemetry Reporting:  Every SLPDB installation provides telemetry updates to status.slpdb.io every 60 seconds.  These status reports provide a useful way for users to know which SLPDB nodes are online at any moment.
    • Flexible API: SLPDB can be easily accessed from the web using SlpServe and SlpSocketServe HTTP gateways.  These provide direct and almost full access to the MongoDB and allow subscription real-time notifications for SLP transactions and block events.
    • Lightweight: SLPDB can only monitor and store data associated with only the tokens you care about.  This can drastically reduce the data storage and cost requirements for the host machine.  Token filtering also provides a performance boost since CPU processing is not being wasted on tokens that you don’t care about.
    • Extensive Testing: SLPDB leverages three separate test harnesses which include (1) SLP message parsing unit tests and differential fuzzing, (2)  SLP input unit tests, (3) end-to-end tests using bitcoin regtest network.  Without such test harnesses, there is no way to be confident in the performance of such a critical piece of software, not to mention a way to be able to detect regressions that might be caused by faulty programming.
    • Peer-Reviewed: SLPDB has been developed as an open-source project and every design decision has been driven by the SLP developer community with full disclosure, awareness, and ample discussion.  This process has resulted in SLPDB becoming an extremely versatile tool that can be used in a very wide range of applications.  Anyone can join in discussing SLPDB development here: t.me/slpdb

Performance Improvements

The initial version of SLPDB was released within weeks from when its need was initially identified in early 2019.  The need for such a service was enormous, as it was required for SLP block explorers and SLP based web apps in early 2019.  At that time there was no indexer for SLP available, so there was a bit of a rush to build one as soon as possible which resulted in poor performance which many of you are familiar with.  Please enjoy these statistics describing the v1.0 SLPDB performance compared to the performance of the previous version.

    • Initial Blockchain Sync Time: 8 hours (previous version: 1 week)
    • Startup Time: 1 minute (previous version: 30 minutes to 6 hours)
    • Validation Speed: ~10 tx/sec (previous version: 1 tx/sec)
    • RAM: 2.5GB (previous version: 6.5GB)
    • node.js CPU: <10% of a single core (previous version: 50%+)

Backward Compatibility

SLPDB v1.0 maintains all functionality that previous versions of SLPDB have had.  However, the DB schema has changed and the queries for getting at information have also changed, resulting in several breaking changes that are documented in the project change log.  However, the most significant breaking changes are:

    • Two of the DB collections have been removed (Addresses and Utxos).  This means any query that previously dependent on one of these collections needs to be updated to pull the information from the “graphs” collection.
    • Several calculated token properties have been removed.  This includes: block_last_active_send, block_last_active_mint, qty_valid_token_utxos, qty_valid_token_addresses, qty_token_minted, qty_token_burned, qty_token_circulating_supply
    • Renamed qty_valid_txns_since_genesis to approx_txns_since_genesis, since this can be corrupted if the block checkpoint is reset.
    • Removed UTXO status enums SPENT_INVALID_SLP, BATON_SPENT_INVALID_SLP

What’s left in development for v1.0?

SLPDB v1.0 will be released after the v1.0 beta has been tested by the community for 4 weeks.  Also, the previous version’s performance is so poor many users may find v1.0 beta to be a significant improvement over the previous v0.15.6 release.  During this period we also need to finish up some critical end-to-end tests using the regtest network.  Currently, all unit tests and end-to-end tests are passing, however, we want to add more tests around edge cases.

Future for SLPDB

    • Continued Performance Optimizations: Our next goal is to achieve 100 tx/sec token validation processing capacity which is approximately a 10x improvement to the v1.0 release.
    • Improved Installation Experience: One of the pain points for SLPDB is the initial installation experience. Some of the npm packages require compilation which is not always taken care of after running: npm install.  We want to greatly improve the installation experience since it can be a source of frustration and deterrent for new users interested in SLP.
    • Lazy Loading: This improvement would result in only loading token graphs that are recently active.  If a transaction was detected for an inactive and unloaded token, SLPDB would load the token’s unpruned graph at that time.  Implementing lazy loading would improve start-up time, graceful shutdown time, and reduce the SLPDB RAM requirements.
    • Utilize MongoDB Sharding and Replica Sets: Read and write DB performance will slow down as the number of transaction records increases.  Implementing sharding and replica sets with MongoDB would scale SLPDB horizontally and optimize long-term DB performance.
    • Future SLP Protocol Upgrades:  As SLP continues to gain momentum the protocol is likely to gain new features and SLPDB should be updated to accommodate those improvements.
    • SMS Notification Service:  SLPDB has built-in telemetry updates as described above, and we are working on extending the functionality of the status.slpdb.io by adding the ability for users to receive SMS notifications about critical events for the user’s SLPDB node.

SLPDB Contributors

A number of individuals were involved in building SLPDB, making design decisions and bolstering the test harnesses behind the scenes since its inception in early 2019.

    • Jt Freeman (Blockparty and Fountainhead.cash)
    • James Cramer (Simple Ledger, Inc.)
    • Chris Troutner (Bitcoin.com)
    • SpendBCH (Bitcoin.com)
    • imaginary_username (independent contributor)
    • Blackhorse (independent contributor)

We want to thank each of the above-listed individual team members for their technical expertise and vision.  SLPDB would not exist without your keen assistance, persistence, and teamwork along the way.