Dmitry Zhelezov: The Challenges of On-Chain Data

This article was written by Subsquid CEO Dmitry Zhelezov. To receive regular updates from Dmitry and from the entire Subsquid core team, subscribe to our email list.

In 2011, Marc Andreessen famously wrote that “software is eating the world.” His prediction was that software companies would disrupt traditional industries, and since then we’ve seen companies like Amazon, Netflix, and Airbnb do just that. Now, ten years later, we are all betting on Web3 — and in particular, the Polkadot ecosystem — eating the software world.

Application data, which is a key asset and a competitive moat for Web2 companies, is now freely available on-chain rather than siloed in a walled garden. Imagine having open access to a full history of anonymized Visa transactions, readily available together with (privacy-enabled) records of Uber rides and Airbnb bookings.

The caveat is that blockchain, while open to everyone, is a really badly suited database for on-chain data search and retrieval. Indeed, a blockchain can be seen as an append-only log of transactions with no support for querying whatsoever.

The Substrate framework has unlocked the possibility for building app-specific chains without the limits placed on the by their host platform (ie. Ethereum). Our key belief at Subsquid is that within the next few years we will see:

• A huge leap in the complexity of dApps
• Many terabytes of loosely interconnected app data stored across hundreds of (para)chains

Notable examples of highly complex dApps include Subsocial (a decentralized social network) and Joystream (a decentralized video publishing platform), which are both already generating high volumes of valuable use-generated data. RMRK leverages the rich cross-chain messaging capabilities of DotSama to build what is arguably the most advanced NFT metaverse project on the market.

The mission of Subsquid is to build a Web3 future where highly sophisticated products can still enjoy the benefits of decentralized backends. As the term “decentralization” has been overused to the point of being almost meaningless, we will elaborate our understanding of what this could actually entail in future writings. For now, let’s focus on what makes Subsquid stand apart from other data-oriented projects in the space and how our high-complexity thesis has affected the product that we’ve been building:

First, we have gone for a highly modular and flexible, multi-layered, architecture consisting of Squid Archives, Squid Processor, and Gateways. This is in surprising contrast to the mainstream, monolithic, approach to data access, with most products offering a Swiss-army knife like experience.

Navigating the Web3 infrastructure landscape, a dApp developer can go for a wide range of APIs and toolings, covering most of the standard use-cases. However, doing anything extra leads to ugly hacks at best, and a protocol-initiated bond slashing at worst. The only hope for the dev is that some day the core protocol team will introduce the missing feature (assuming there’s enough demand for it).

The point is, finer control over APIs enables Web3 developers to build better products. Tasks such as building a recommendation system for Joystream, or an open API for the Subsocial social graph, requires a bespoke backend. Subsquid (in many ways similar to the Substrate framework) conveniently offers the necessary API building blocks with limitless options for customization of the storage, data transformation and enrichment.

Second, we double down on dedicated technical support for the projects that are building on Subsquid. It is indeed contrarian to the prevailing anon-culture of crypto, where there’s no support line to call if a private key is lost. In fact, we are in direct contact with most of the builders leveraging Subsquid on Telegram.

This way of doing things enables us to get first-hand feedback, which is extremely valuable for us. In fact, we often go on to make Pull Requests to help our partners. Constant “dogfooding” allows us to quickly react and build the best possible experience for developers that are working with substrate data.

One example of a feature that has grown out of our own frustration is native support for Substrate runtime upgrades. The painful experience of chasing the data format changes led us to creating an exploration tool that generates the boilerplate code that deals with upgrades.

If you are building an innovative product on Substrate, feel free to reach out to me directly. It would be my pleasure to discuss your data needs with you and your team!

Dmitry Zhelezov is the CEO and technical co-founder of Subsquid. Dmitry holds a Ph.D. in Mathematics and has extensive experience leading development teams in the blockchain space. He can be contacted directly via Telegram (@dzlzv) or Twitter (@dizhel).

Our Channels

Email list: http://eepurl.com/hBOqLT

Website: http://subsquid.io/

Twitter: https://www.twitter.com/subsquid

Discord: https://discord.gg/5cQBWHWJvW

Telegram: http://t.me/subsquid

Subsocial: https://app.subsocial.network/@subsquid