Bitcoin is a peer-to-peer electronic cash system that enables multiple parties to transact directly with one another, without needing to rely on trusted third party as a mediator. The system maintains a distributed ledger, formed of blocks of many transactions, and utilises a Proof-of-Work (PoW) consensus algorithm to append new blocks to the chain. The native token of the blockchain, denominated in Satoshis, can be transferred between users through new transactions whereby digital signatures allow the movement of funds to be validated.

The blockchain ledger is maintained by a network of nodes. Each node is responsible for aggregating and ordering users’ transactions into blocks and attempting to find a PoW solution for their block by expending computational power. The nodes are incentivised to do this through a block reward, which itself contains a block subsidy and transaction fees. The block subsidy halves approximately every 4 years, meaning that over time the incentive to create blocks is increasingly made up of the fees paid in each transaction.

Once a transaction has been included in the blockchain, or has reached a certain depth due to subsequent blocks built on top, it becomes computationally infeasible to alter the its data or claim an alternative record existed in its place. This is a statement of the immutability property of Bitcoin-like chains. This property should be distinguished from indelibility, or the property that data can be removed from the blockchain. Bitcoin-like chains are immutable due to the PoW process, but they are not indelible since data can be pruned from the blockchain. The data that can typically be pruned from the blockchain is anything that cannot be used in a future spending transaction, such as in transactions whose outputs have already been spent. The data stored using the most common embedding methods Bartoletti and Pompianu (2017) can also be pruned in this way.

A Bitcoin block (B) is used to collect and order a batch of transactions. These blocks are produced by Bitcoin nodes in regular time intervals of approximately 10 min and comprise three core components as shown in Figure 1.

A Bitcoin transaction (Tx) is used to convey a transfer or value between users or to record data on the blockchain. Each transaction has a unique identifier defined as the double SHA-256 hash of the transaction contents TxID≔H(Tx).

In Bitcoin, the use of a Merkle tree to encode the set of transactions in a block allows for efficient lightweight verification that a given transaction is part of a block. The mechanism used to perform such a check, given a candidate transaction Tx _i , located at index i in a block B, and the Merkle root R _B of the block, is known as a Merkle proof (π) Merkle (1980). This proof comprises a set of hash values used to recover a candidate Merkle root R′. If R′ = R _B then the proof is valid and Tx _i is contained in B. Note that the entire data of Tx _i is required to verify such a proof.

As described previously, every Bitcoin transaction Tx has a unique identifier TxID associated with it. Because this identifier is derived from the double-hash of the transaction, this means that the entire transaction must be possessed in order to verify whether any subset of the transaction content exists on the blockchain.

The first aspect of the proposed solution is that a secondary identifier can be generated for each transaction. In general this can be thought of as a “meta” transaction identifier, and we can use the notation MTxID to refer to it. We propose that such a secondary transaction identifier must be some function F of the full transaction data, such that MTxID≔F(Tx). This ensures that all of the data within Tx can be identified by MTxID. However, to fulfil our stated goals, we must also ensure that only a subset of the contents of Tx are required to verify that something is contained within Tx. Furthermore, we must ensure that MTxID is unique to Tx.

The proposed mechanism to generate a secondary transaction identifier that is unique and only requires a subset of Tx to verify is to use a Merkle tree. In this approach, we define the secondary identifier as MTxID = R, where R is the root of a Merkle tree T. In our construction, the leaves of T are comprised of the contents of Tx split into discrete segments, such that the entirety of Tx is collectively contained within the set of leaves of T.

This approach ensures the uniqueness of MTxID because the root is derived from successively hashing the leaves of the Merkle tree using a one-way cryptographic hash function (e.g., SHA-256). Changing any of the data within the leaves, or the order of the leaves of T, will necessarily alter the resulting MTxID identifier. An example structure of such a transaction-based Merkle tree T is shown in Figure 3.

As outlined in other Merkle-based protocols for blockchain data applications such as Bruschi et al. (2021), the use of a Merkle tree inherently allows for compact and efficient proofs of existence. The proposed system extends this property to individual fields of blockchain transactions, such that any item of data within any blockchain transaction can be independently verified in isolation. This presents a signficant efficiency gain when compared with existing blockchain Merkle proofs where the entire transaction is required.

In our scheme, a transaction Tx is segmented into a set of M data packets D ₁, …D _M . There are many possible ways the transaction data can be split and we here outline a few options as examples.

The proposed mechanism can be implemented as a layer 1 protocol simply be replacing the TxIDs with MTxIDs within the base consensus algorithm of the blockchain used. However, in a layer 2 setting where we do not modify the underlying consensus protocol, we require an alternative way to bind the secondary transaction identifier MTxID to the original identifier TxID. This can be achieved either by including TxID as an additional leaf in the transaction Merkle tree or by prepending the TxID to each leaf in the transaction Merkle tree. Both options ensure that the entire transaction Tx must be processed when MTxID is first generated and that each other leaf is inextricably linked to the full original transaction data.

The steps to generate an MTxID for a given transaction Tx are outlined in the Algorithm 1. In this algorithm the function Merklize() generates a typical binary Merkle tree from a set of leaf data items, whose Merkle root is labelled R.

Algorithm 1

Transaction Identifier Generation.

Input: Transaction Tx

The generation algorithm is intended to be used by a node when it first receives a new transaction in the blockchain network. In the layer 1 setting, this algorithm would be used to directly generate MTxID as the canonical transaction identifier for a given transaction Tx used for the consensus algorithm and block generation process.

When used at layer 2, this generation algorithm can be used by any node voluntarily using our proposed mechanism to prepare transactions for potential data redaction at a future time. If the node wishes to include a transaction in its own block, it should construct the block according to a modified block generation process (see Algorithm 3) that incorporates the above algorithm to ensure that the MTxID is bound to the canonical TxID for this transaction, such that the node can later redact data contained in Tx without sacrificing the ability to prove the existence of the non-redacted content of Tx, as required. If the node does not wish to include Tx in its own block, it may still store the corresponding MTxID locally for similar purposes.

The corresponding protocol for verifying a secondary transaction identifier, given MTxID and Tx, is detailed in Algorithm 2. The root of the generated tree T′ is labelled R′.

Algorithm 2

Transaction Identifier Verification.

Input: Transaction Identifier MTxID, Transaction Tx  

This algorithm is used to verify that a given MTxID has been correctly generated for a given Tx, and therefore requires the full data of the transaction. For this reason, Algorithm 2 is only directly relevant to blockchain nodes in a layer 1 implementation of the mechanism and during the consensus process, while a new transaction is propagating through the network for inclusion on the blockchain and when verifying a new block that contains the transaction. Algorithm 2 may nonetheless be used at any time in a layer 2 setting by a node or a third party external to the blockchain network in possession of the full transaction data Tx to verify it has been correctly associated with an MTxID by nodes voluntarily using our proposed mechanism as a layer 2 protocol.

The transaction identifier verification process should not be conflated with the process of verifying, in the absence of the full transaction data Tx, whether an individual data portion D _i of the transaction has been included on the blockchain. This process is instead referred to as a lightweight proof of existence verification and is outlined in Section 4.4.

Previously, deleting any part of a transaction would compromise the ability of a blockchain node to prove the existence of data contained within the rest of the transaction. The introduction of MTxIDs now enables data to be redacted from the blockchain by allowing nodes to simply delete particular leaves from a transaction Merkle tree T. Deleting these leaves does not prohibit the node from providing compact proofs of existence for the remainder of the transaction, or even verifying future spending relationships with the transaction in question. However, the addition of an MTxID for each transaction is only part of the solution. Specifically, we need a mechanism to establish trust that a given MTxID has been generated correctly for the corresponding transaction. Without having a publicly known and reliable attestation of the MTxIDs for each transaction, we cannot use them to prove with confidence that a particular data item actually exists on the blockchain.

To prove that a data element D _i is included in a transaction on the blockchain, we must prove both of the following:

• there must be a valid proof π _M connecting D _i to a secondary identifier MTxID; and

If both conditions are met, we can be convinced that D _i exists at the claimed location on the blockchain.

The first condition is met simply by obtaining π _M , which is the set of hashes corresponding to a standard Merkle proof Merkle (1980) that proves the leaf data D _i is a member of the set of leaves of the Merkle tree corresponding the Merkle root R = MTxID for the transaction in question.

The second condition is easily met in a layer 1 setting, where MTxID is simply the canonical transaction identifier for the transaction, assuming that the consensus algorithm has been modified appropriately to ensure this. In this case, the proof π _B is simply the standard Merkle proof that connects MTxID to the Merkle root R _B of a valid block B in the blockchain, in the same way that there exists a Merkle proof in the Bitcoin consensus algorithm that currently connects each transaction identifier TxID to a root R _B .

In a layer 2 setting however, a participating node must perform additional steps during the block generation process to ensure that a valid π _B will exist for a given MTxID. In the following subsections, we outline how a node may generate a secondary block Merkle tree, in addition to the standard Merkle tree of a Bitcoin block, and include the root R _M of this secondary tree in a coinbase transaction during block generation. By including both R _B and this additional root R _M when generating a block, the participating node establishes the required information for a valid proof π _B in the layer 2 setting. The use of π _B to verify the existence of a transaction data portion on the blockchain is then outlined in Section 4.4.

To achieve such a proof, we introduce a secondary block Merkle tree denoted by T _M and whose root is R _M . For a block B containing N transactions, the leaves of T _M correspond to the set of secondary transaction identifiers MTxID ₁, …, MTxID _N for the transactions Tx ₁, …, Tx _N contained with the block. In other words, T _M is effectively a tree of Merkle trees. The proposed redaction mechanism relies on a node including the root R _M in the coinbase transaction of blocks they successfully generate.

The structure of a secondary block Merkle tree is outlined in Figure 4. As shown, the order of the transactions used to derive T _M must be the same as the order of the transactions as they appear in block B and its original block Merkle tree T _B . The exception is the coinbase transaction, which does not appear in the secondary tree. The coinbase transaction is necessarily excluded from T _M because our redaction protocol requires R _M to be stored in the coinbase transaction itself.

To facilitate the redaction of data after it has been included on the blockchain in a block B, we require only that the node responsible for generating B also includes the root R _M in the coinbase transaction. This represents a minor modification of the standard process of generating Bitcoin blocks that a node participating in this redaction scheme. The details of the updated block-generation protocol is shown in Algorithm 3. The modified process requires that the node generates the secondary Merkle tree T _M and includes its root R _M in the coinbase transaction. A suitable option is to include the root in the coinbase script. The node can then generate the standard block Merkle tree T _B and generate the final block B by finding a valid proof of work solution. Crucially, the proposed solution operates at a layer above the base blockchain protocol and does not require any change to the underlying rule set used to maintain and validate the blockchain.

Algorithm 3

Block Generation for Redactable Data.

Input: Transaction set T ≔ { T x 1 , … T x N }

The core goal of the mechanism outlined in Section 3 is to prepare a transaction Tx, when it is added to the blockchain, such that a subset of its data can be verified as included on the blockchain at a later time in the absence of the full original transaction data. In other words, to allow for the verification that a data portion D _i ∈ D ₁, …, D _M has been included in the blockchain without access to one or more of the remaining data portions D _j≠i of the original Tx. As outlined previously, we require two proofs π _M and π _B to satisfy our conditions that prove the data portion exists on the blockchain in the absence of the full transaction data.

The first proof π _M is the Merkle proof connecting the data portion D _i to the secondary identifier MTxID and can be checked simply using a standard Merkle proof verification. This check is performed by verifying the Merkle proof π _M for the leaf D _i against the Merkle root MTxID, which we denote by C ₁ ← MerkleVerify(D _i , MTxID, π _M ), setting C ₁ to True if the check is successful and False otherwise.

In the layer 2 context we are considering, the second proof π _B = (π _MTxID , π _TxID ) in fact comprises two further Merkle proofs π _MTxID and π _TxID , which connect MTxID to the secondary root R _M and TxID to the standard root R _B respectively. In order to verify the second condition using π _B , we must perform a total of three checks C ₂, C ₃, C ₄. Checks C ₂, C ₃ are simply Merkle proof verifications similar to C ₁ and using the same notation. The final check C ₄ is to confirm that both R _M and R _B have been included in the same block header H _B by a participating node. In combination, checks C ₂ to C ₄ prove that MTxID corresponds to the same transaction as TxID included in the block B.

The full lightweight proof of existence process is outlined in Algorithm 4. This mechanism allows anybody to verify that a data portion D _i has been included on the blockchain as part of Tx without requiring the other data portions D _j≠i comprising the original transaction.

Algorithm 4

Layer 2 Lightweight Proof of Existence.

Input: D _i , MTxID, TxID, R _M , R _B , block header H _B , proof π _M , proofs π _B = (π _MTxID , π _TxID )

This proof of existence protocol is lightweight because it only requires the portion of the transaction D _i that is to be verified. For instance, if a large transaction 100 Megabytes in size has been split into 1 Kilobyte portions the verifier only needs to retrieve 0.001% of the transaction data. This represents a significant storage and bandwidth saving for any application that needs to verify small items of data stored within large transactions. The flexibility and granularity of data portions D _i that can be verified in this manner is constrained by the choice of how the transaction Tx was originally split into packets in Algorithm 3 by the participating blockchain node.

It is anticipated that the lightweight proof of existence verification process may be performed by any blockchain node or third party that needs to verify the existence of a portion of data on the blockchain. An example application using this feature would be a video streaming service to allow users to check the metadata of a video file stored on the blockchain without needing to download the entire media file itself.

In both layer 1 and layer 2 implementations of the Merklized transaction protocol, it should be noted that lightweight verifications are expected to be performed at a later time after the transaction has been initially added to the blockchain. This means that transactions are still expected to be processed in the usual manner at the time they are added to the blockchain. However, by implementing the transaction Merklization process during block generation, whether at layer 1 or 2, individual fields can be later verified by any party in a lightweight manner.

A key benefit of the proposed transaction Merklization protocol is that individual data elements within transactions can be verified as existing on the blockchain without possessing the full transaction data. This feature is crucial to allow nodes to selectively delete some portions of transaction data without sacrificing the ability to verify others.

This is particularly advantageous in the event that a node is compelled by law to delete or redact some data from its local copy of the blockchain database. Typically, if a node is required to delete or redact a portion of transaction data it must delete the full transaction, making it impossible to preserve other parts of the transaction not subject to the redaction. This can render nodes unable to prove legally-compliant data exists on the blockchain in the case that it is stored in the same transaction as data which must be redacted.

Conversely, our mechanism allows nodes to redact only the portions of data to which legal deletion or regulations such as GDPR apply. The use of Merklized transactions, and the extensions outlined for support as a layer 2 protocol, ensure that the remaining portions of the transaction data can still be proven as part of the blockchain database even after the node has removed other components as required.

In this regime, the redaction of a data portion from a transaction simply involves a node deleting the specific leaves from a transaction Merkle tree T that correspond to the data that must be redacted. The deletion of a subset of leaves from a given transaction by one node does not preclude other nodes from storing and serving the remaining leaves from the same tree. This is because the Merkle tree structures used allow any node to prove the existence of any of the remaining leaves without needing to hold or access the deleted leaves. In practice, this may be done in response to a request for deletion to comply with the right to be forgotten or GDPR laws Ateniese et al. (2017); Shah et al. (2019). Each such request may only apply to different nodes on a jurisdictional or regional basis. Crucially, when implemented as a layer 2 protocol this allows individual nodes to act unilaterally in redacting data as required by the legal jurisdiction in which they operate, without affecting the operations of other nodes in the network. This therefore allows each node to comply with local rules and regulations regarding data protection and privacy without impinging on other nodes which may not be subject to the same legilsation across the world.

The proposed protocol introduced redactability for blockchain data that can be implemented on top of the base Bitcoin protocol. This means that there is no requirement to modify the underlying blockchain infrastructure to facilitate better GDPR compliance for nodes. Moreover, the mechanism can be implemented independently by any individual node, whether in their own interest or as a public service to the network. This also presents a significant privacy benefit for the end users of blockchain networks who can now demand that their data is removed from public blockchain ledgers.

Our solution is also efficient because it does not rely on any heavy cryptographic primitives and is based primarily on Merkle trees which are a scalable data structure allowing compact proofs of existence. The efficiency of this method allows partial transaction verification that can allow a user to check individual aspects of a transaction, such as the transaction value or the number of outputs, without downloading the entire transaction using Merkle proofs. This presents a new way for blockchain analytics to be conducted using lower bandwidths suitable for Internet of Things devices or low-powered user hardware.

As discussed, the key efficiency benefit afforded by the proposed protocol is during the lightweight verification processes, where only a minimal portion of the transaction data is required to verify its existence. In extreme cases, this may represent a significant saving in terms of storage and bandwidth required to obtain the data necessary for verification.

When assessing the overall scalability of the solution as compared with current blockchain protocols, we must consider the complexity of the process used to generate an MTxID and to generate the corresponding secondary root R _M used to encode these identifiers in a block by participating nodes.

The time complexity of computing a standard transaction identifier TxID is determined by the time complexity of the hash function used to generate the identifier. In the case of Bitcoin, the SHA-256 hash function is used, whose complexity scales as O ( n ) , where n is the number of 512-bit message blocks contained in the full data of Tx Rachmawati et al. (2018). The complexity associated with generating the MTxID for a given transaction will therefore depend on the size of the data packets D _i into which the transaction is split. If it is split into packets of approximately 512-bits, the process of generating a MTxID will scale as O ( n + l o g ( n ) ) , where the additional log(n) arises due to the definition of MTxID as a Merkle tree requiring additional hashes to be computed upon the leaf hash values. This shows the additional time complexity required to compute MTxID is small compared to the existing complexity for generating a TxID, and therefore does not significantly impact the performance of nodes when processing transactions.

The generation of the additional root R _M to be included in the coinbase transaction has the same complexity as generating the standard root R _B , which scales as O ( N ) where N is the total number of transactions contained in the block. However, this should not increase the time taken to generate a block significantly as both roots can be generated in parallel and either computation represents a small fraction of the total verification process for transactions which is generally dominated by the verification of elliptic curve digital signatures.

The protocol in its current form is limited because only the transactions in blocks mined by participating miners are eligible for redaction. In addition, the protocol still requires a level of trust in the nodes themselves to faithfully create the correct secondary tree T _M for each of their blocks. Here we propose potential extensions to improve these aspects.