Indexing for Next-Gen dApps | Unleashing the FireSquid

It’s official! After months of development, we are proud to finally announce the start of the FireSquid rollout. In case you haven’t heard about it, this is Subsquid’s most significant product update since the launch of the project.

We expect this release to bring our tech stack even further ahead of the competition. While our documentation is currently getting brought up-to-date to include everything developers need to know about FireSquid, we’d like now to take a little time to provide our community with a brief overview of the new features.

FireSquid Release Highlights

• Rapid-fire syncing → 10–50k blocks per second
• Low-maintenance Archives → 10 times cheaper to maintain
• Updated GraphQL interfaces → Faster and more efficient APIs
• Human-readable data → SQL-like experience comes to Web3

What is FireSquid?

FireSquid is an umbrella release of the Subsquid stack which includes major improvements to Archives, the Squid Processor SDK, and the Squid CLI. The most exciting part of the FireSquid release is the introduction of BatchProcessor to the Squid SDK. This enables faster syncing by many orders of magnitude, with a whopping 10–50k blocks per second as the typical speed.

This breakthrough in synchronization speed is thanks to the new way that on-chain data is handled by Squid Processors: Instead of processing events and calls one by one, the Squid essentially transforms a vector of event and call data as a whole. The change is not dissimilar from how vector-optimized GPU algorithms brought about a paradigm shift in deep learning and crypto mining.

Further improvements come from a complete set of optimizations of how the data is packed and fetched from the Archive, further saving bandwidth and increasing processing speed.

New Archives

Like their predecessors, FireSquid Archives ingest and archive raw on-chain data. Compared to v5, however, these new Archives are at least 5x faster and 10x cheaper to maintain in production. Indeed, this is the most performant archiving solution for Substrate available today.

We have achieved maximum performance for these archives through the following innovations:

• Ingesting on-chain data from multiple gRPC endpoints in parallel;
• Support for a clustered, highly available, Cockroach DB with built-in data compaction and replication.

In fact, FireSquid Archives are considerably faster than the Substrate Archive by Parity which is written in Rust, even when run against public endpoints. For example, it takes less than a day to synchronize a full archive of 12M+ Kusama blocks. To index all the historical blocks, events, calls, and extrinsics requires less than 350 Gb of storage.

FireSquid Archives come with two GraphQL interfaces, one for programmatic data extraction optimized for Squids, and another which serves as an explorer interface for humans. The interface for Squids enables a long-awaited feature of direct lookups of calls even if a call is wrapped into a sudo or batch transaction, together with numerous optimizations for filtering and batching.

On the human side, the explorer interface is a powerful tool for an SQL-like experience of querying transactions, calls, events, and block data. Go ahead and try it out at https://kusama.explorer.subsquid.io/graphql

About Subsquid

Subsquid is an on-chain indexing and querying solution that enables Web3 builders to gain access to on-chain data on their own terms. Featuring modular architecture and decentralised governance, this is the most developer-friendly and resource-efficient way to build, test, and deploy customised APIs for blockchain-facing applications.

Twitter | Discord | LinkedIn | Telegram | Github | Newsletter | Subsocial