Explaining dYdX Indexer in detail

Detailing dYdX Indexer

As we approach the launch of the v4 mainnet, we want to give you a glimpse into what the dYdX team is building. This article dives deep into the indexer. The indexer is a read-only layer that exists between the chain itself and end-users. The purpose of the indexer is to transform and provide data in a more user-friendly format.

A good way to think of the indexer is similar to the role of Infura or Alchemy in the Ethereum ecosystem. However, unlike Infura/Alchemy, and like everything else in dYdX v4, the indexer is fully open-source and anyone can run it!

What Is the Indexer?

As part of the dYdX ecosystem tools, we want to ensure that customers have access to high-performance data queries when using the dYdX v4 exchange. The Cosmos SDK full node provides many APIs which can be used to request on-chain data. However, these full nodes are optimized for submitting and executing blocks, not for serving high-frequency, low-latency requests from network/mobile clients.

• Hong Kong’s new anti-money laundering regulations have taken effect. Here are the key compliance points:
• Origin and Development of ZKP: From the 1980s to Present
• 【Exclusive from ChainDD】Analysis of 525-page EU Regulatory Regulations: Step-by-Step Guide to Writing White Papers, Only 3 Types of Cryptocurrencies Approved

This is why we wrote software for an indexing service. The indexer is a read-only service that provides off-chain data to clients via a REST API and websockets. Its purpose is to store and provide data that exists on the dYdX chain in a more user-friendly way.

In other words, the purpose of the indexer is to index and provide data to clients in a more efficient, performant, and web2-friendly way. For example, the indexer will serve websockets which provide updates on order book and fill states. These clients will include front-end applications (mobile and web), market makers, institutions, and any other parties that want to query dYdX chain data through traditional web2 APIs.

On-Chain & Off-Chain Data

The indexer will run two separate ingestion/storage processes using data from v4 full nodes: one for on-chain data, one for off-chain data. Currently, the throughput of state changes for on-chain data is expected to be between 10-50 events per second. On the other hand, the expected throughput of state changes for off-chain data is between 500-1,000 events/second. This represents a difference in throughput requirements of 10-100x. By separately handling these data types, v4 aims to allow different services to scale better based on their throughput requirements.

On-Chain Data

On-chain data is all data that can be replicated by reading transactions submitted on the dYdX chain. All on-chain data has been validated by consensus. This data includes:

• Account balance (USDC)

• Account position (open contracts)

• Order executions

• Industry

• Liquidation

• Deleveraging

• Partial and full order fills

• Funding rate payments

• Trading fees

• Historical oracle prices (used for funding calculation and settlement spot prices)

• Placing and canceling long orders

• Conditional placing and canceling

Off-chain data

Off-chain data is data that is stored in the memory of each v4 node. It is not written to the blockchain or stored in application state. This data cannot be queried via the gRPC API on a v4 node and cannot be derived from data stored in blocks. It is essentially temporary data on a v4 node that is lost when data is cleared from memory storage on restart. This includes:

• Short-term order placements and cancellations

• Order books for each perpetual trading pair

• Indexing orders before they reach the chain

Indexer architecture

The indexer is made up of a series of services that obtain information from v4 full nodes and provide that information to various clients. Kafka topics are used to pass events/data to services within the indexer. The following outlines the key services that make up the Indexer.

Ender (on-chain ingestion)

Ender is the on-chain data ingestion service for the indexer. It uses data from the “to-ender” Kafka topic (which queues up all on-chain events per block) and each payload will contain all event data for the entire block.

Ender obtains all state changes from that block and applies them to a Postgres database so the indexer can store all on-chain data. Ender also creates and sends websocket events via “to-websocket-?” Kafka topics for any websocket events that need to be emitted.

Vulcan (Off-chain ingestion)

Vulcan is Indexer’s off-chain data ingestion service. It will use data from the “to-vulcan” Kafka topic, which queues all off-chain events, carrying payloads including active order book updates, order placement updates, order cancellation updates, and optimistic fills.

This data will be stored in Redis cache. Vulcan will update Redis with any new outstanding order updates, set the status of cancelled orders to cancelled-suspended, and update the order book according to the received payloads.

Whenever partially filled orders are cancelled, Vulcan will also update Postgres to update the order status in Postgres. Vulcan will also create and send websocket events using “to-websocket-?” for any websocket events that need to be emitted.

Comlink (API server)

Comlink is an API server that will expose REST API endpoints to read on-chain and off-chain data. For example, users can request their USDC balance or the size of a specific position through Comlink and receive a formatted JSON response.

As a clear goal set by the dYdX team, we are designing the v4 API to closely match the v3 exchange API [1] since dYdX users are already familiar with these APIs. Over time, we have time to collect feedback and iterate on these APIs with v3 and believe that they are reasonable at the product level.

Roundtable

Roundtable is a periodic job service that provides the necessary exchange aggregation calculations. Examples of these calculations include: 24-hour trading volumes for each market, open interest, PnL classified by account, candlesticks, etc.

Socks (Websocket service)

Socks is Indexer’s websockets service that allows real-time communication between clients and Indexer. It will use data from ender, vulcan, and roundtable and send websocket messages to connected clients.

Hosting and deploying the indexer

In order to create an end-to-end decentralized product, Indexer will be open-source. This will include comprehensive documentation about all services and systems as well as infrastructure-as-code for running the indexer on popular cloud providers. Specific responsibilities for third-party operators who wish to host the indexer typically include initial deployment and ongoing maintenance. Initial deployment will involve:

• Setting up AWS infrastructure to leverage the open-source repository.

• Deploying the indexer code to retrieve data from full nodes and publish that information via APIs and websockets.

Datadog (provides useful metrics and monitoring for the Indexer service) and Bugsnag (provides real-time alerts for errors or issues that require manual intervention). Maintenance of the Indexer will involve:

• Migrating and/or upgrading the Indexer to new open-source versions

• Monitoring Bugsnag and Datadog for any issues and alerting internal teams to resolve them

• Debugging and fixing any issues using the runbook provided by dYdX

dYdX suggests that at least one DevOps engineer is required to perform the necessary responsibilities for deploying and maintaining the Indexer. The operator will need to use the following services:

• AWS

• ECS-Fargate

• RDS-Postgres database

• EC2

• Lambda

• Elasticache Redis

• EC2 ELB – Load Balancer

• Cloudwatch – Logs

• KMS – Data Encryption Manager

• Terraform Cloud – For deploying to the cloud

• Bugsnag – Error awareness

• Datadog – Metrics and Monitoring

• Pagerduty – Alerts The operator should be able to host the open-source Indexer for public access in a highly available manner (i.e., high uptime). Requirements include having accounts for the aforementioned services and employing appropriate personnel to perform deployment and maintenance responsibilities.

About dYdX

At dYdX, our mission is to democratize financial opportunity. We believe the v4 software will represent a significant step forward in serving that mission. The global economic events of the past year have further reinforced the demand for open, transparent, and permissionless financial products. We are excited for v4 to better serve these needs.

Original author: dydx

Original source: dydx

Original title: v4 deep dive: indexer

Translation: Will, IBCL

We will continue to update Blocking; if you have any questions or suggestions, please contact us!