Developer Advocate Journal #2
Docs, docs, docs…
Updating our documentation has been a central task over the past few days, so, at the risk of sounding repetitive, I feel it’s worth saying a few words about the reasoning behind these changes.
I tend to take a methodical approach to projects, so prior to starting this task I took some time to do some research. I came across this documentation framework and found it really interesting — the division into four quadrants is very close to my natural approach towards docs as a consumer and something that was missing in our materials up until now.
One can think of these quadrants as the lens through which our new documentation should be viewed. 🧐 Each part of the updated materials will fall into one of these categories and will be easy to find, depending on the needs of the reader.
My first priority was to improve the pages that explained the most basic concepts of Subsquid (the ‘explanation’ quadrant). For developers who are new to blockchain technology there are just so many new notions to absorb!
Of course, it wouldn’t be possible to explain every little detail about blockchain in this documentation, but we can at least try to provide the minimum necessary for those who are just starting to build on Subsquid.
I have also tried to retrace my own steps, scattering external links like breadcrumbs throughout our pages. Developers can follow these links to become better acquainted with any new or unfamiliar topics.
Getting up to speed, one line of code at a time
I ended last week by telling myself:
“If I am to be advocating for Subsquid developers, my first priority should be to know the ins and outs of the product itself!”
So, after settling in and getting to know my colleagues and establishing the first tasks I’d be getting into (spoiler: it’s documentation! 😬), it was finally time to sink my teeth into the tech.
The task of restructuring, updating, and improving our documentation really helps in this sense, first because I need to read it, and second because while writing, all sorts of questions start to pop up.
Let’s take the example of our multi-layered approach. OK, so our architecture is divided into Archives and Squids. The former collects raw blockchain data and makes it exploitable, while the latter leverages this data and performs custom transformation on it. That’s easy enough to understand.
But while trying to explain it in more detail, I found that my initial level of understanding simply was not enough.
That is why, at some point, I took to old habits: I went to explore our codebase. And that’s where I finally started connecting the dots (Polkadot pun intended😉).
The inner workings of our Archive became clearer when I looked into how the IndexBuilder , BlockProducer, IndexerStatusService, and PooledExecutor classes interacted with each other:
- IndexBuilder orchestrates blockchain data ingestion by fetching information from the BlockProducer
- It then instantiates PooledExecutor so that multiple workers extract raw data, wrap it in TypeScript classes representing ORM entities (namely the Block itself, Extrinsics, Events), and saves it in the database.
- It emits events, so that the IndexerStatusService is updated on the synchronisation status.
In short, it provides continuous ingestion of blockchain data, conveniently storing it in a more exploitable way.
Sounds simple, doesn’t it? That’s because it’s a very elegant implementation, in my opinion. Kudos to Eldar, our Senior Core Dev! 🙌
As for the ‘Squid’ part of the ETL pipeline, it was quite clear from the start that the Processor is the key, so I went to take a look at the SubstrateProcessor class. In short, this class sets up ingestion metrics, groups blocks in batches, then starts a loop that ingests those batches.
The nice thing about it however is that it sets up methods for the developer to customise data processing by adding functions as Event Handlers, Extrinsic Handlers, and pre- or post-Block hooks.
I have used Django in the past and I instantly connected Block hooks to the concept of middleware in web servers (NestJS has a great explanation for its middleware — what a coincidence, I am taking inspiration from them for our docs!).
On the other hand, for the Handlers the pattern is akin to a Publish-Subscribe, where when calling
addEventHandler(eventName: QualifiedName, fn: EventHandler)
the developer has to specify the eventName as an argument. While processing a block, if that event is encountered, the Handler is triggered by calling the function fn specified as the second argument.
Once again, quite a clever use of tried and true software design patterns. It really does result in a polished implementation that is quite intuitive.
My first Dev interview!
As part of our Business Development initiative we are establishing various partnerships that often entail the development of software integrations. One of my roles during this process is to hold special meetings (called ‘developer interviews’) that help to make sure we gather all necessary information about how our technology is being implemented. This way I can communicate these things to our community later on.
Simply put, a dev interview is a meeting where the developer advocate (me!) sits down with developers to go over the integration that they are responsible for. During these meetings we discuss the work that needs to be done and highlight any noteworthy customisations or trade-offs that need to be made.
I had my very first of these meetings this morning, and you’ll be able to read an announcement about this upcoming integration in just a few days!
My hope for the dev interview process is that it will lead to interesting case studies that we can include in our documentation as neatly detailed ‘How-tos’ (referred to as Recipes in our docs). This is something I am very much looking forward to. Most importantly, dev interviews give me an excuse to get my hands dirty and do some coding 😈🤓.
Join our Community!
To find out more about Subsquid, join us on our Discord server and chat to one of our helpful Subsquid team members or other users. For those who want some more background information on Subsquid and what it’s all about, our website and Medium page are the perfect places to start.
Website, Twitter, Discord, Telegram, Subsocial, Reddit, Facebook, Instagram
