The magic behind the optimism sequencer L2 Derivation Principle.

Unlocking the Power of Positivity Understanding the L2 Derivation Principle in the Optimism Sequencer

Author: joohhnnn

How does opstack derive Layer2 from Layer1?

Before reading this article, I strongly recommend that you first read the introduction to the derivation section from optimism/specs (source[2]). If you feel confused after reading this article, don’t worry, it’s normal. But still, please remember this feeling, because after reading our analysis in this article, please come back and read it again, and you will find that the official article is really concise, summarizing all the key points and details.

• Oops! LastPass Data Breach Leaves 25 Crypto Users Crying with $4.4 Million Loss in Just One Day
• ZK Coprocessor from 0 to 1 What can it actually do?
• Analysis of the niche but not mainstream game track – LBS game track

Now let’s get into the main topic of the article. We all know that the running nodes of layer2 can obtain data from the DA layer (layer1) and construct complete layer2 block data. Today we will explain how this process is implemented in the codebase.

Questions you need to have

If you were asked to design such a system now, how would you design it? What problems would you have? Here are some questions I listed. Thinking with these questions will help you better understand the whole article.

• How does the entire system work when you start a new node?

• Do you need to query all the block data of l1 one by one? How to trigger the query?

• After obtaining the data of the l1 block, what data do you need?

• During the derivation process, how does the state of the blocks change from unsafe to safe and then to finalized?

• What are the obscure data structures in the official specs, such as batch/channel/frame? (You can understand them in detail in the previous chapter 03-how-batcher-works)

What is Derivation?

Before understanding derivation, let’s first talk about the basic rollup mechanism of optimism, here we simply take a transfer transaction on l2 as an example.

When you send a transfer transaction on the optimism network, this transaction will be “forwarded” to the sequencer node, which will sort the transactions and then package and broadcast the blocks, which can be understood as block generation. We call this block containing your transaction Block A. The state of Block A at this time is unsafe. Then, when the sequencer reaches a certain time interval (e.g. 4 minutes), the batcher module in the sequencer will send all the transactions (including your transfer transaction) collected in the previous four minutes as a single transaction to l1, and l1 will produce block X. The state of Block A is still unsafe at this time. When any node executes the program in the derivation part, the node fetches the data of block X from l1 and updates the local l2 unsafe Block A. The state of Block A is now safe. After two epochs of l1 (64 blocks), the l2 node marks block A as a finalized block.

And derivation is to bring the role into the l2 node mentioned above, and gradually transform the obtained unsafe block into a safe block through continuous parallel execution of the derivation program, while gradually transforming the blocks that are already safe into a finalized state.

Code Layer Deep Dive

Hoho captain, let’s dive deep?

Get the data of batch transactions sent by the batcher

First, let’s see how to check whether there is data of batch transactions in a new l1 block. Here, let’s first sort out the modules needed, and then check these modules.

• First, determine the block number of the next l1 block

• Parse the data of the next block

Determine the block number of the next block

op-node/rollup/derive/l1_traversal.go

By querying the block height of origin.Number + 1, you can get the latest l1 block. If this block does not exist, i.e., it matches error and ethereum.NotFound, it means that the current block height is the latest block and the next block has not been generated on l1 yet. If the retrieval is successful, record the latest block number in l1t.block.

Parse the data of the block

op-node/rollup/derive/calldata_source.go

First, use InfoAndTxsByHash to get all the transactions of the retrieved block, and then pass the transactions, our batcherAddr, and our config into the DataFromEVMTransactions function. Why do we pass these parameters? Because when filtering these transactions, we need to ensure the accuracy (authority) of the batcher address and the recipient address. After receiving these parameters in DataFromEVMTransactions, loop through each transaction to filter the accuracy of the address and find the correct batch transactions.

From data to safeAttribute, making the unsafe block safe

In this part, first, the data parsed in the previous step is transformed into a frame and added to the frames array of the FrameQueue. Then, extract a frame from the frames array, and initialize the frame into a channel and add it to the channelbank, waiting for the frames in this channel to be added. After the frames in the channel are completed, extract the batch information from the channel and add the batch to the BatchQueue. Add the batch in the BatchQueue to the AttributesQueue to construct safeAttributes, and update the safeblock in the enginequeue. Finally, the EL layer update of safeblock is completed by calling the ForkchoiceUpdate function.

data -> frame

op-node/rollup/derive/frame_queue.go

This function retrieves the data from the previous step using the NextData function. Then, it parses the data and adds it to the frames array inside the FrameQueue. It returns the first frame in the array.

frame -> channel

op-node/rollup/derive/channel_bank.go

The NextData function is responsible for reading the first raw data from the current channel bank’s first channel and returning it. It also calls the NextFrame function to retrieve the frame and load it into the channel.

channel -> batch

op-node/rollup/derive/channel_in_reader.go

The NextBatch function is mainly responsible for decoding the previous raw data into data with a batch structure and returning it. The WriteChannel function provides a function and assigns it to nextBatchFn. The purpose of this function is to create a reader, decode data with a batch structure from the reader, and return it.

*Note❗️In this case, the batch generated by the NextBatch function is not used directly, but added to the batchQueue first for unified management and use. Additionally, the NextBatch function here is actually called by the func (bq BatchQueue) NextBatch() function under the op-node/rollup/derive/batch_queue.go directory.

batch -> safeAttributes

Additional Information:
1. In Layer 2 blocks, the first transaction in the block is always an “anchor transaction,” which can be understood as containing some L1 information. If this Layer 2 block is also the first block in the epoch, it will also include a “deposit” transaction from Layer 1.
2. The term “batch” used here should not be confused with the batch transactions sent by the batcher. For example, we name the batch transactions sent by the batcher as batchA, while the batch discussed here is named as batchB. BatchA and batchB are related, where batchA may contain a large number of transactions that can be constructed as batchB, batchBB, batchBBB, and so on. BatchB corresponds to the transactions in a Layer 2 block, while batchA corresponds to a large number of transactions in multiple Layer 2 blocks.

op-node/rollup/derive/attributes_queue.go

• The NextAttributes function takes the current L2 safe block header and passes it along with the batch we obtained from the previous step to the createNextAttributes function, which constructs the safeAttributes.

• In createNextAttributes, it is important to note that the PreLianGuaireLianGuaiyloadAttributes function is called internally. This function is mainly responsible for handling the anchor transactions and deposit transactions. Finally, the function concatenates the batch transactions with the transactions returned by the PreLianGuaireLianGuaiyloadAttributes function and returns them.

The createNextAttributes function internally calls PreLianGuaireLianGuaiyloadAttributes.

“`   
func (aq *AttributesQueue) NextAttributes(ctx context.Context, l2SafeHead eth.L2BlockRef) (*eth.PayloadAttributes, error) {       
// Get a batch if we need it       
if aq.batch == nil {           
batch, err := aq.prev.NextBatch(ctx, l2SafeHead)           
if err != nil {               
return nil, err           
}           
aq.batch = batch       
}       
// Actually generate the next attributes       
if attrs, err := aq.createNextAttributes(ctx, aq.batch, l2SafeHead); err != nil {           
return nil, err       
} else {           
// Clear out the local state once we will succeed           
aq.batch = nil           
return attrs, nil       
}   
}
“`

“`   
func (aq *AttributesQueue) createNextAttributes(ctx context.Context, batch *BatchData, l2SafeHead eth.L2BlockRef) (*eth.PayloadAttributes, error) {               ……       
attrs, err := aq.builder.PreLianGuaireLianGuaiyloadAttributes(fetchCtx, l2SafeHead, batch.Epoch())        ……       
return attrs, nil   
}
“`

In this step, the engine queue sets the safehead to safe. However, this does not mean that the block is actually safe. It still needs to be updated using ForkchoiceUpdate in EL.

The tryNextSafeAttributes function internally checks the relationship between the current safehead and unsafehead. If everything is normal, it triggers the consolidateNextSafeAttributes function to set the safeHead in the engine queue to the safe block constructed from the previously obtained safeAttributes. It also sets needForkchoiceUpdate to true to trigger the subsequent ForkchoiceUpdate to change the block status in EL to safe and truly convert the unsafe block into a safe block. Finally, the postProcessSafeL2 function adds the safehead to the finalizedL1 queue for subsequent finalize use.

    func (eq *EngineQueue) tryNextSafeAttributes(ctx context.Context) error {
        …… 
        if eq.safeHead.Number < eq.unsafeHead.Number {
            return eq.consolidateNextSafeAttributes(ctx)
        }
        …… 
    }

    func (eq *EngineQueue) consolidateNextSafeAttributes(ctx context.Context) error {
        ……
        LianGuaiyload, err := eq.engine.PayloadByNumber(ctx, eq.safeHead.Number+1)
        ……
        ref, err := LianGuaiyloadToBlockRef(LianGuaiyload, &eq.cfg.Genesis)
        ……
        eq.safeHead = ref
        eq.needForkchoiceUpdate = true
        eq.postProcessSafeL2()
        ……
        return nil
    }

Finalize the safe blocks

The safe blocks are not truly secure, they still need further finalization known as “finalized”. When a block’s status transitions to “safe”, the derived source L1 (batcher transaction) starts the calculation. After two L1 epochs (64 blocks), this safe block can be updated to the “finalized” state.

op-node/rollup/derive/engine_queue.go

The function tryFinalizeLianGuaistL2Blocks internally validates the blocks in the “finalized queue” for 64 blocks. If the validation passes, it calls tryFinalizeL2 to complete the setting of “finalized” in the engine queue and update the “needForkchoiceUpdate” flag.

    func (eq *EngineQueue) tryFinalizeLianGuaistL2Blocks(ctx context.Context) error {
        ……
        eq.log.Info("processing L1 finality information", "l1_finalized", eq.finalizedL1, "l1_origin", eq.origin, "previous", eq.triedFinalizeAt) //const finalityDelay untyped int = 64
        // Sanity check we are indeed on the finalizing chain, and not stuck on something else.
        // We assume that the block-by-number query is consistent with the previously received finalized chain signal
        ref, err := eq.l1Fetcher.L1BlockRefByNumber(ctx, eq.origin.Number)
        if err != nil {
            return NewTemporaryError(fmt.Errorf("failed to check if on finalizing L1 chain: %w", err))
        }
        if ref.Hash != eq.origin.Hash {
            return NewResetError(fmt.Errorf("need to reset, we are on %s, not on the finalizing L1 chain %s (towards %s)", eq.origin, ref, eq.finalizedL1))
        }
        eq.tryFinalizeL2()
        return nil
    }

    func (eq *EngineQueue) tryFinalizeL2() {
        if eq.finalizedL1 == (eth.L1BlockRef{}) {
            return // if no L1 information is finalized yet, then skip this
        }
        eq.triedFinalizeAt = eq.origin // default to keep the same finalized block
        finalizedL2 := eq.finalized // go through the latest inclusion data, and find the last L2 block that was derived from a finalized L1 block
        for _, fd := range eq.finalityData {
            if fd.L2Block.Number > finalizedL2.Number && fd.L1Block.Number <= eq.finalizedL1.Number {
                finalizedL2 = fd.L2Block
                eq.needForkchoiceUpdate = true
            }
        }
        eq.finalized = finalizedL2
        eq.metrics.RecordL2Ref("l2_finalized", finalizedL2)
    }

Trigger Loop

In op-node/rollup/driver/state.go, the eventLoop function is responsible for triggering the execution entry point of the entire loop process. It mainly indirectly executes the Step function in op-node/rollup/derive/engine_queue.go.

func (eq *EngineQueue) Step(ctx context.Context) error {
	if eq.needForkchoiceUpdate {
		return eq.tryUpdateEngine(ctx)
	}
	// Trying unsafe LianGuaiyload should be done before safe attributes
	// It allows the unsafe head can move forward while the long-range consolidation is in progress.
	if eq.unsafeLianGuaiyloads.Len() > 0 {
		if err := eq.tryNextUnsafeLianGuaiyload(ctx); err != io.EOF {
			return err
		}
		// EOF error means we can't process the next unsafe LianGuaiyload. Then we should process next safe attributes.
	}
	if eq.isEngineSyncing() {
		// Make pipeline first focus to sync unsafe blocks to engineSyncTarget
		return EngineP2PSyncing
	}
	if eq.safeAttributes != nil {
		return eq.tryNextSafeAttributes(ctx)
	}
	outOfData := false
	newOrigin := eq.prev.Origin()
	// Check if the L2 unsafe head origin is consistent with the new origin
	if err := eq.verifyNewL1Origin(ctx, newOrigin); err != nil {
		return err
	}
	eq.origin = newOrigin
	eq.postProcessSafeL2()
	// make sure we track the last L2 safe head for every new L1 block
	// try to finalize the L2 blocks we have synced so far (no-op if L1 finality is behind)
	if err := eq.tryFinalizeLianGuaistL2Blocks(ctx); err != nil {
		return err
	}
	if next, err := eq.prev.NextAttributes(ctx, eq.safeHead); err == io.EOF {
		outOfData = true
	} else if err != nil {
		return err
	} else {
		eq.safeAttributes = &attributesWithLianGuairent{
			attributes: next,
			LianGuairent: eq.safeHead,
		}
		eq.log.Debug("Adding next safe attributes", "safe_head", eq.safeHead, "next", next)
		return NotEnoughData
	}
	if outOfData {
		return io.EOF
	} else {
		return nil
	}
}

Summary

The entire derivation function seems very complex, but if you break down each step, you can still grasp the understanding well. The reason why the official speculation is not easy to understand is that its concepts, such as batch, frame, channel, etc., are easy to confuse. Therefore, if you still feel confused after reading this article, it is recommended to go back and take a look at our 03-how-batcher-works.

References

[1]

joohhnnn: https://learnblockchain.cn/people/4858

[2]

source: https://github.com/ethereum-optimism/optimism/blob/develop/specs/derivation.md#deriving-payload-attributes

[3]

Chapter 1: https://learnblockchain.cn/article/6589

[4]

Chapter 2: https://learnblockchain.cn/article/6755

[5]

Chapter 3: https://learnblockchain.cn/article/6756

[6]

Chapter 4: https://learnblockchain.cn/article/6757

[7]

Chapter 5: https://learnblockchain.cn/article/6758

Here you go! Enjoy your blockchain journey with these chapters and their corresponding links. Happy reading!

We will continue to update Blocking; if you have any questions or suggestions, please contact us!