To get a deeper understanding of the lifecycle of a Diem transaction, we will follow a transaction on its journey from being submitted to a Diem node to being committed to the Diem Blockchain. We will then “zoom-in” on each logical component of a validator node and take a look at its interactions with other components.
A Diem client constructs a raw transaction (let us call it T5raw) to transfer 10 Diem Coins from Alice’s account to Bob’s account. The raw transaction includes the following fields. Each field is linked to its glossary definition.
- Alice's account address.
- A module (or program) that indicates the actions to be performed on Alice's behalf. In this case, it contains:
- A Move bytecode peer-to-peer transaction script.
- A list of inputs to the script (for this example the list would contain Bob's account address and the payment amount in Diem Coins).
- Maximum gas amount Alice is willing to pay for this transaction. Gas is a way to pay for computation and storage. A gas unit is an abstract measurement of computation.
- Gas price (in microdiem/gas units) — The amount in Diem Coins Alice is willing to pay per unit of gas, to execute the transaction.
- Expiration time of the transaction.
- Sequence number — 5. The sequence number for an account indicates the number of transactions that have been submitted from that account. In this case, 5 transactions have been submitted from Alice’s account, including T5raw).
- A transaction with sequence number 5 can only be committed on-chain if the account sequence number is 5.
- Chain ID — An identifier that distinguishes the Diem Mainnet from networks used for other purposes including test networks.
The client signs transaction with Alice's private key. The signed transaction T5 includes the following:
- The raw transaction.
- Alice's public key.
- Alice's signature.
To describe the lifecycle of transaction T5, we will assume that:
- Alice and Bob have accounts on the Diem Blockchain.
- Alice's account has 110 Diem Coins.
- The current sequence number of Alice's account is 5 (which indicates that 5 transactions, including T5raw, have already been sent from Alice's account).
- There are a total of 100 validator nodes — V1 to V100 on the network.
- The client submits transaction T5 to validator V1. A transaction can be submitted directly to a validator-operated FullNode or a Diem FullNode. Diem FullNodes forward transactions to validator V1.
- Validator V1 is a proposer/leader for the current round.
In this section, we will describe the lifecycle of transaction T5, from its submission by the client to being committed to the Diem Blockchain.
For the relevant steps, we've included a link to the corresponding inter-component interactions of the validator node. After you are familiar with all the steps in the lifecycle of the transaction, you may want to refer to the information on the corresponding inter-component interactions for each step.
Note: The arrows in all the visuals in this article originate on the component initiating an interaction/action and terminate on the component on which the action is being performed. The arrows do not represent data read, written, or returned.
Figure 1.1 Lifecycle of a Transaction
1 — The client submits transaction T5 to a validator node V1 through the JSON-RPC service (Client → JSON-RPC service).
2 — JSON-RPC service transmits transaction T5 to V1 mempool (JSON-RPC service → Mempool JRS.2, MP.1)
3 — Mempool will use the virtual machine (VM) component to perform validation checks, such as signature verification, checking that Alice's account has sufficient balance, checking that transaction T5 is not being replayed using the sequence number, and so on. (Mempool → VM)
4 — The mempool will hold T5 in an in-memory buffer. Mempool may already contain multiple transactions sent from Alice's address.
5 — Using the shared-mempool protocol, V1 will share the transactions (including T5) in its mempool with other validator nodes and place transactions received from them into its own (V1) mempool. (Mempool → Other Validators MP.2).
6 — As validator V1 is a proposer/leader, it will pull a block of transactions from its mempool and replicate this block as a proposal to other validator nodes via its consensus component. (Consensus → Mempool MP.3, CO.1)
7 — The consensus component of V1 is responsible for coordinating agreement among all validators on the order of transactions in the proposed block (Consensus → Other Validators CO.2). Refer to our technical paper State Machine Replication in the Diem Blockchain for details of our proposed consensus protocol DiemBFT.
9 — The execution component manages the execution of transactions in the virtual machine (VM). Note that this execution happens speculatively before the transactions in the block have been agreed upon. (Execution → VM EX.2, VM.3)
10 — After executing the transactions in the block, the execution component appends the transactions in the block (including T5) to the Merkle accumulator (of the ledger history). This is an in-memory/temporary version of the Merkle accumulator. The necessary part of the proposed/speculative result of executing these transactions is returned to the consensus component to agree on. (Consensus → Execution CO.3, EX.1). The arrow from "consensus" to "execution" indicates that the request to execute transactions was made by the consensus component. (For consistent use of arrows throughout this article, the arrows do not represent the flow of data).
11 — V1 (the consensus leader) attempts to reach consensus on the proposed block's execution result with the other validator nodes participating in consensus. (Consensus → Other Validators CO.3)
12 — If the proposed block's execution result is agreed upon and signed by a set of validators that have the quorum of votes, validator V1's execution component reads the full result of the proposed block execution from the speculative execution cache and commits all the transactions in the proposed block to persistent storage with their results. (Consensus → Execution CO.4, EX.3), (Execution → Storage EX.4, ST.3)
13 — Alice's account will now have 100 Diem Coins, and its sequence number will be 6. If T5 is replayed by Bob, it will be rejected as the sequence number of Alice's account (6) is greater than the sequence number of the replayed transaction (5).
In the previous section, we described the typical lifecycle of a sample transaction from being submitted to being committed to the Diem Blockchain's distributed database. Now let's look in more depth at the inter-component interactions of a validator node as it processes transactions and responds to queries. This information will be most useful to those who:
- Would like to get an overall idea of how the system works under the covers.
- Are interested in eventually contributing to the Diem Core software.
For our narrative, we will assume that a client submits a transaction TN to a validator VX. For each validator component, we will describe each of its inter-component interactions in subsections under the respective component's section. Note that subsections describing the inter-component interactions are not listed strictly in the order in which they are performed. Most of the interactions are relevant to the processing of a transaction, and some are relevant to clients querying the blockchain (queries for existing information on the blockchain).
Let us look at the core logical components of a validator node:
Figure 1.2 Client Service
JSON-RPC Service (JRS) is the sole external interface of the validator. Any request made by a client to the validator goes to the JRS first.
A client submits a transaction to the client service of a validator node VX.
JRS forwards the transaction to validator node VX's mempool
. The mempool will accept the transaction TN only if the sequence number of TN is greater than or equal to the current sequence number of the sender's account (note that the transaction will not be passed to consensus until the sequence number matches the sequence number of the sender’s account).
When a client performs a read query on the Diem Blockchain (for example, to get the balance of Alice's account), JRS interacts with the storage component directly to obtain the requested information.
Figure 1.3 Virtual Machine
The Move virtual machine (VM) verifies and executes transaction scripts written in Move bytecode.
When mempool requests the VM to validate a transaction via
VM::ValidateTransaction(), the VM loads the transaction sender's account from storage and performs verifications, some of which have been described in the list below. View the entire list of verifications here.
- Checks that the input signature on the signed transaction is correct (to reject incorrectly signed transactions).
- Checks that the sender's account authentication key is the same as the hash of the public key (corresponding to the private key used to sign the transaction).
- Verifies that the sequence number for the transaction is greater than or equal to the current sequence number for the sender's account. Completing this check prevents the replay of the same transaction against the sender's account.
- Verifies that the program in the signed transaction is not malformed, as a malformed program cannot be executed by the VM.
- Verifies that the sender's account balance contains at least the maximum gas amount multiplied by the gas price specified in the transaction, which ensures that the transaction can pay for the resources it uses.
The execution component utilizes the VM to execute a transaction via
It is important to understand that executing a transaction is different from updating the state of the ledger and persisting the results in storage. A transaction TN is first executed as part of an attempt to reach agreement on blocks during consensus. If agreement is reached with the other validators on the ordering of transactions and their execution results, the results are persisted in storage and the state of the ledger is updated.
When mempool receives a transaction from other validators via shared mempool or from the JRS, mempool invokes VM::ValidateTransaction() on the VM to validate the transaction.
For implementation details refer to the Virtual Machine README.
Figure 1.4 Mempool
Mempool is a shared buffer that holds the transactions that are “waiting” to be executed. When a new transaction is added to the mempool, the mempool shares this transaction with other validator nodes in the system. To reduce network consumption in the “shared mempool,” each validator is responsible for delivering its own transactions to other validators. When a validator receives a transaction from the mempool of another validator, the transaction is added to the mempool of the recipient validator.
- After receiving a transaction from the client, the JRS proxies the transaction to the validator node’s mempool.
- The mempool for validator node VX accepts transaction TN for the sender's account only if the sequence number of TN is greater than or equal to the current sequence number of the sender's account.
- The mempool of validator node VX shares transaction TN with the other validators on the same network.
- Other validators share the transactions in their respective mempools with VX’s mempool.
- When the transaction is forwarded to a validator node and once the validator node becomes the leader, its consensus component will pull a block of transactions from its mempool and replicate the proposed block to other validators. It does this to arrive at a consensus on the ordering of transactions and the execution results of the transactions in the proposed block.
- Note that just because a transaction TN was included in a proposed consensus block, it does not guarantee that TN will eventually be persisted in the distributed database of the Diem Blockchain.
When mempool receives a transaction from other validators, mempool invokes VM::ValidateTransaction() on the VM to validate the transaction.
For implementation details refer to the Mempool README.
Figure 1.5 Consensus
The consensus component (consensus) is responsible for ordering blocks of transactions and agreeing on the results of execution by participating in the consensus protocol with other validators in the network.
When validator VX is a leader/proposer, the consensus component of VX pulls a block of transactions from its mempool via:
Mempool::GetBlock(), and forms a proposed block of transactions.
If VX is a proposer/leader, its consensus component replicates the proposed block of transactions to other validators.
- To execute a block of transactions, consensus interacts with the execution component. Consensus executes a block of transactions via
Execution:ExecuteBlock()(Refer to Consensus → execution)
- After executing the transactions in the proposed block, the execution component responds to the consensus component with the result of executing these transactions.
- The consensus component signs the execution results and attempts to reach agreement on this result with other validators participating in consensus.
If enough validators vote for the same execution result, the consensus component of VX informs execution via
Execution::CommitBlock() that this block is ready to be committed.
For implementation details refer to the Consensus README.
Figure 1.6 Execution
The execution component coordinates the execution of a block of transactions and maintains a transient state that can be voted upon by consensus. If these transactions are successful, they are committed to storage.
- Consensus requests execution to execute a block of transactions via:
- Execution maintains a “scratchpad,” which holds in-memory copies of the relevant portions of the Merkle accumulators. This information is used to calculate the root hash of the current state of the Diem Blockchain.
- The root hash of the current state is combined with the information about the transactions in the proposed block to determine the new root hash of the accumulator. This is done prior to persisting any data, and to ensure that no state or transaction is stored until agreement is reached by a quorum of validators.
- Execution computes the speculative root hash and then the consensus component of VX signs this root hash and attempts to reach agreement on this root hash with other validators.
When consensus requests execution to execute a block of transactions via
Execution::ExecuteBlock(), execution uses the VM to determine the results of executing the block of transactions.
If a quorum of validators agrees on the block execution results, the consensus component of each validator informs its execution component via
Execution::CommitBlock() that this block is ready to be committed. This call to the execution component will include the signatures of the agreeing validators to provide proof of their agreement.
Execution takes the values from its “scratchpad” and sends them to storage for persistence via
Storage::SaveTransactions(). Execution then prunes the old values from the “scratchpad” that are no longer needed (for example, parallel blocks that cannot be committed).
For implementation details refer to the Execution README.
Figure 1.7 Storage
The storage component persists agreed upon blocks of transactions and their execution results to the Diem Blockchain. A block of transactions (which includes transaction TN) will be saved via storage when:
- There is agreement between more than a quorum (2f+1) of the validators participating in consensus on all of the following:
- The transactions to include in a block
- The order of the transactions
- The execution results of the transactions to be included in the block
Refer to Merkle accumulators for information on how a transaction is appended to the data structure representing the Diem Blockchain.
When AC or mempool invokes
VM::ValidateTransaction() to validate a transaction,
VM::ValidateTransaction() loads the sender's account from storage and performs the read-only validity checks on the transaction.
When the consensus component calls
Execution::ExecuteBlock(), execution reads the current state from storage combined with the in-memory “scratchpad” data to determine the execution results.
- Once consensus is reached on a block of transactions, execution calls storage via
Storage::SaveTransactions()to save the block of transactions and permanently record them. This will also store the signatures from the validator nodes that agreed on this block of transactions.
- The in-memory data in “scratchpad” for this block is passed to update storage and persist the transactions.
- When the storage is updated, every account that was modified by these transactions will have its sequence number incremented by one.
- Note: The sequence number of an account on the Diem Blockchain increments by one for each committed transaction originating from that account.
For client queries that read information from the blockchain, JRS directly interacts with storage to read the requested information.
For implementation details refer to the Storage README.