Sending transactions
Project: Let's build Bitcoin
In the previous blog post, we visualized the blockchain as a graph using Graphwiz.
In this post, we will extend Learn Coin to support sending transactions. We will explore the following topics:
- What does it mean to send a new transaction?
- Lock transaction outputs in Learn Coin with a Public Key
- Specify public keys for genesis block and for miners
- Add JSON RPC for clients to send transactions
- Keep pending transactions as local state in the LearnCoin node
- Extend client CLI to send new transactions, show account balances, and get transaction info
- Explain existing security problems
Let’s begin!
What does it mean to send a new transaction?
Sending a transaction refers to a process where the network includes the transaction in one of the future mined blocks.
The process starts with a client requesting a new transaction to be processed by some learn coin node. The node propagates the transaction to the whole network, ensuring that all miners would learn about it, and hopefully include it in the next block. Miners get a fee for each transaction as an incentive.
When a transaction is included in a block, we say that it was confirmed once. For each next mined block, the transaction gets an additional confirmation. For example, if five blocks are mined after a transaction is included in some block, we say it has 6 confirmations. The likelyhood of a transaction being reverted is significantly smaller with each additional confirmation.
Lock transaction outputs with a public key
To transfer coins from transaction inputs to new transaction outputs, we need a locking script which only the owner knows how to unlock. The locking script is empty in our current implementation, so let’s fix that by introducing a new public key concept and adding it to the locking script.
// learncoin/src/public_key.rs
#[derive(Debug, Clone, Hash, Serialize, Deserialize, Eq, PartialEq)]
pub struct PublicKey(String);
We pretend that a public key is any string for the time being, even though this is not really the case. But don’t worry about it yet. We will talk about the maths behind the security in future blog posts.
Locking script is still empty, so let’s add a public key field to it.
We will also update the TransactionOutput to require a locking script.
// learncoin/src/transaction.rs
...
#[derive(Debug, Clone, Serialize, Deserialize, Eq, PartialEq)]
pub struct LockingScript {
public_key: PublicKey,
}
// ...
impl TransactionOutput {
pub fn new(amount: i64, locking_script: LockingScript) -> Self {
Self {
locking_script,
amount,
}
}
// ...
}
Learn Coin constructs transaction ID hash by getting a string value of its data, so we have to make sure that we update the Display trait to include the locking script too.
impl Display for TransactionOutput {
fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
write!(f, "{}:{}", self.locking_script, self.amount)
}
}
What public key to use in the genesis block?
We have changed the TransactionOutput::new constructor to take a locking_script now, so our code won’t compile until we fix all call sites.
There are few important public keys that we need to choose:
- for locking transaction outputs in the genesis block;
- for locking transaction outputs of the coinbase transactions in blocks mined by miners.
Let’s use genesis-address public key to lock transaction outputs for the genesis block.
What public key should miners use?
Miners get the public key to lock transaction outputs of the coinbase transaction via the block-template protocol. We have already included this in our implementation of the block-template protocol, but we still need to choose the value for it. Let’s extend the server CLI to take the public key for miners as an argument.
// learncoin/src/commands/server_command.rs
pub fn server_command() -> App<'static> {
// ...
.arg(
Arg::new("miner-public-key")
.long("miner-public-key")
.about("PUBLIC_KEY to lock the transaction output of the coinbase transaction.")
.takes_value(true)
.required(true)
.default_value("genesis-address"),
)
We need to propagate this information to the LearnCoinNode and ensure it’s sent to the miner via the block-template protocol.
I’ll leave this to you as an exercise.
Transactions must include the minimum height of the current block
As we already know, transaction IDs are constructed as a double hash of the transaction data. As long as the transaction data is unique, transaction IDs should be unique too. However, it’s very common for many coinbase transactions to have the same data, e.g. when the same miner finds multiple blocks because it locks the transaction outputs with the same public key. This may lead to different transactions having the same ID.
To fix this, we extend transaction data with a minimum block height of the block that can include the transaction. Miners would use the next block height to ensure the data for each coinbase transaction is unique.
// learncoin/src/transaction.rs
// ...
pub struct Transaction {
// ...
minimum_block_height: u32,
}
The block height is used to compute the ID of the transaction, so let’s extend the hash_transaction_data function.
impl Transaction {
// ...
fn hash_transaction_data(
block_height: u32,
inputs: &Vec<TransactionInput>,
outputs: &Vec<TransactionOutput>,
) -> TransactionId {
let data = format!(
"{}-{}-{}",
block_height,
inputs
.iter()
.map(TransactionInput::to_string)
.collect::<Vec<String>>()
.join(""),
outputs
.iter()
.map(TransactionOutput::to_string)
.collect::<Vec<String>>()
.join("")
);
let first_hash = Sha256::digest(data.as_bytes());
let second_hash = Sha256::digest(first_hash.as_slice());
TransactionId(second_hash)
}
}
What if someone sends multiple transactions that have the same data?
This can happen in our current implementation because transactions are not validated yet. However, once we introduce validation, transactions may only spend UTXOs (unspent transaction outputs). Therefore, if there are multiple transactions with the exact same data, that means all would trying to spend the same outputs. As a result, the network would reject all except for one.
Learn Coin Node supports sending new transactions
Before we can send new transactions via the client CLI, we need to extend the JSON RPC protocol first. Let’s add a new method that takes a single transaction inputs and potentially multiple transaction outputs. The client requests to send a new transaction, and the node responds with the transaction ID that is created based on its knowledge about the current blockchain height.
// learncoin/src/peer_message.rs
pub enum JsonRpcMethod {
// ...
// Expects TransactionId in response.
SendTransaction(TransactionInput, Vec<TransactionOutput>),
}
pub enum JsonRpcResult {
// ...
TransactionId(TransactionId),
}
When the learn coin node receives a request to send a new transaction, it adds it to the local set of pending transactions and tells its peers about it.
// learncoin/src/learncoin_node.rs
pub struct LearnCoinNode {
// ...
pending_transactions: HashMap<TransactionId, Transaction>,
}
We will call the process of telling its peers about the transaction relaying, so let’s add a new TransactionRelay message to our protocol.
// learncoin/src/peer_message.rs
pub enum PeerMessagePayload {
// ...
TransactionRelay(Transaction),
}
Finally, let’s implement the on_send_transaction function to handle the request.
impl LearnCoinNode {
// ...
fn on_send_transaction(
&mut self,
input: TransactionInput,
outputs: Vec<TransactionOutput>,
peer_address: &str,
id: u64,
) {
// we will implement this now ...
}
Remember that transactions now include a height of the next block, so we find the height of the blockchain tip first.
// ... continuation
// Safety: Tip must always be in the BlocKIndex.
let tip_height = self
.block_index
.get_block_index_node(&self.active_chain.tip().header().hash())
.unwrap()
.height as u32;
match Transaction::new(tip_height + 1, vec![input], outputs) {
Ok(transaction) => {
// Transaction format is valid.
// ...
}
Err(e) => {
// Transaction format is invalid.
let result = Err(e);
let response = JsonRpcResponse { id, result };
self.network
.send(peer_address, &PeerMessagePayload::JsonRpcResponse(response));
}
}
}
If transaction format is valid, the node responds to the client with success. Additionally, the node adds the transaction to its local pending transaction state and relays it to its peers.
// ... continuation
// Transaction format is valid.
let result = Ok(JsonRpcResult::TransactionId(*transaction.id()));
let response = JsonRpcResponse { id, result };
// TODO: Validation.
self.pending_transactions
.insert(*transaction.id(), transaction.clone());
self.network
.send(peer_address, &PeerMessagePayload::JsonRpcResponse(response));
self.network
.send_to_all(&PeerMessagePayload::TransactionRelay(transaction));
The pending transactions are included in the block-template (we also include the public key for miners).
// learncoin/src/learncoin_node.rs
impl LearnCoinNode {
// ...
fn on_get_block_template(&mut self, peer_address: &str, id: u64, unix_timestamp: u64) {
// ...
let block_template = BlockTemplate {
// ...
transactions: self
.pending_transactions
.values()
.map(Transaction::clone)
.collect::<Vec<Transaction>>(),
public_key: self.miner_public_key.clone(),
}
}
}
Great! Miners can now include transactions in the blocks. However, we never remove pending transactions, so miners would always include the same transactions. Let’s fix that.
Update pending transactions state
When a pending transaction is included in a new block, we simply remove it.
We also have to ensure to return the transaction to the pool when a block is reverted.
The best place to implement this functionality is the maybe_update_active_chain function that already does something similar for blocks.
// learncoin/src/learncoin_node.rs
impl LearnCoinNode {
// ...
fn maybe_update_active_chain(&mut self, candidate_block_hash: &BlockHash) {
// ...
if active_tip.chain_work < candidate.chain_work {
// ...
// Remove blocks from the active chain until the LCA becomes the new tip.
while self.active_chain.tip().header().hash() != lowest_common_ancestor {
// ...
// When a block is removed from the active chain,
// we add transactions back to the pool of pending transactions.
for transaction in removed.transactions() {
self.pending_transactions
.insert(*transaction.id(), transaction.clone());
}
}
// ...
for new_tip_hash in candidate_path {
// ...
// When a block is added to the active chain, we remove transactions from the pool.
for transaction in new_tip_block.transactions() {
self.pending_transactions.remove(transaction.id());
}
}
}
}
Great, the miners don’t include the same transactions in the new blocks anymore.
Handling Transaction Relay messages
We’re almost there. All that’s left for the learncoin node is to process transaction relay messages by adding them to the local state and delegating them to the peers.
impl LearnCoinNode {
fn on_message(...) {
match message {
// ...
PeerMessagePayload::TransactionRelay(transaction) => {
self.on_transaction_relay(transaction)
}
}
}
// ...
fn on_transaction_relay(&mut self, transaction: Transaction) {
// TODO: Validate.
if !self.pending_transactions.contains_key(transaction.id()) {
self.pending_transactions
.insert(*transaction.id(), transaction.clone());
self.network
.send_to_all(&PeerMessagePayload::TransactionRelay(transaction));
}
}
}
The node only processes the transaction if it’s valid, but we are not there yet. Also, if the node already knows about the transaction, it won’t do anything to avoid infinite loops where the same transaction is relayed over the network over and over again.
Extend Client CLI to send new transactions
Let’s add a new command to our client CLI to send transactions.
// learncoin/src/commands/client_command.rs
fn send_transaction() -> App<'static> {
App::new("send-transaction")
.version("0.1")
.about(
"Send a transaction with a single transaction input and multiple transaction outputs.",
)
.arg(
Arg::new("input")
.long("input")
.value_name("TXID:INDEX")
.about("Unspent transaction output formatted as <txid:output_index>")
.takes_value(true)
.required(true),
)
.arg(
Arg::new("outputs")
.long("outputs")
.value_name("Comma-separated list of <PublicKey>:<Amount>")
.takes_value(true)
.required(true)
.multiple_values(true)
.use_delimiter(true),
)
}
Running the following command will spend an output from transaction 437ffefd050015398c6daf80999c9dc165dd5927734c9bc10a42cf610a0ceaee at index 0, then lock 30 coins with a public key alice-address and 17 coins with a public key bob-address.
./learncoin client --server 127.0.0.1:8333 \
-send-transaction \
--input "437ffefd050015398c6daf80999c9dc165dd5927734c9bc10a42cf610a0ceaee:0" \
--outputs "alice-address:30,bob-address:17"
Let’s implement the client-side logic that uses JSON RPC to a send new transaction to the learncoin node.
pub fn run_client_command(matches: &ArgMatches) -> Result<(), Box<dyn Error>> {
// ...
else if let Some(ref matches) = matches.subcommand_matches("send-transaction") {
let transaction_input = matches.value_of("input").unwrap();
let mut tokens = transaction_input.split(":");
let utxo_id = TransactionId::new(
Sha256::from_hex(tokens.next().expect("input format must be <txid:index>")).unwrap(),
);
let output_index = OutputIndex::new(
tokens
.next()
.expect("input format must be <txid:index>")
.parse::<i32>()
.unwrap(),
);
let transaction_input = TransactionInput::new(utxo_id, output_index);
let transaction_outputs: Vec<TransactionOutput> = matches
.values_of_lossy("outputs")
.unwrap()
.into_iter()
.map(|balances| {
let mut tokens = balances.split(":");
let locking_script = LockingScript::new(PublicKey::new(
tokens
.next()
.expect("output format must be list of <pubkey:amount>")
.to_owned(),
));
let amount = tokens
.next()
.expect("output format must be list of <pubkey:amount>")
.parse::<i64>()
.unwrap();
TransactionOutput::new(amount, locking_script)
})
.collect();
client.execute_send_transaction(transaction_input, transaction_outputs)?;
Ok(())
}
}
The above code parses CLI arguments and calls client.execute_send_transaction function, which sends the request to the node.
// learncoin/src/client.rs
impl Client {
// ...
pub fn execute_send_transaction(
&mut self,
input: TransactionInput,
outputs: Vec<TransactionOutput>,
) -> Result<(), String> {
let id = self.send_json_rpc_request(JsonRpcMethod::SendTransaction(input, outputs))?;
match self.wait_for_json_rpc_response(id)? {
JsonRpcResponse { id, result } => match result? {
JsonRpcResult::TransactionId(transaction_id) => {
println!("{}", transaction_id);
Ok(())
}
unexpected => Err(format!("Received unexpected message: {:?}", unexpected)),
},
}
}
}
If the request is successful, we print the transaction ID to the stdout.
This is similar to what send transaction in bitcoin-cli does.
Extend Client CLI to show balances for each address
LearnCoin network can accept transactions and mine blocks, but we don’t have a good way to see how much money each address has.
When we say an address bob-address has 5 coins, we mean there are some (potentially many) unspent transaction outputs that are locked with the bob-address public key whose total amount is 5 coins.
Let’s extend the client CLI print balances associated with each public key.
// learncoin/src/commands/client_command.rs
fn get_balances() -> App<'static> {
App::new("get-balances")
.version("0.1")
.about("Retrieves balances for each public address on the blockchain.")
}
pub fn run_client_command(matches: &ArgMatches) -> Result<(), Box<dyn Error>> {
// ...
else if let Some(ref matches) = matches.subcommand_matches("get-balances") {
client.execute_get_balances()?;
Ok(())
}
}
To execute the above command, the client gets the full blockchain, extracts balances for each public key and prints them to the stdout in non-increasing order.
// learncoin/src/client.rs
impl Client {
// ...
pub fn execute_get_balances(&mut self) -> Result<(), String> {
let (_, active_blocks) = self.get_blockchain()?;
let mut balances = AccountBalances::extract_account_balances(&active_blocks)
// Sort by amount in non-increasing order.
.into_iter()
.collect::<Vec<(PublicKey, i64)>>();
balances.sort_by(|(_, lhs), (_, rhs)| rhs.cmp(lhs));
for (address, balance) in balances {
println!("{}: {}", address, balance);
}
Ok(())
}
}
All that’s left is to implement AccountBalances::extract_acount_balances function.
There are multiple approaches to achieve this, but I decided to use the following algorithm:
- Maintain the current state of balances and transactions after processing each block in the active chain.
- Decrement the balance for each transaction input in each block. We do this by finding the referenced transaction output, which is why we need to maintain the transactions seen so far.
- Increment the balance for each transaction output in each block.
Let’s begin with implementing the high level logic which stores the transactions seen so far and delegates processing of inputs and outputs.
// learncoin/src/commands/account_balances.rs
impl AccountBalances {
pub fn extract_account_balances(active_blocks: &Vec<Block>) -> HashMap<PublicKey, i64> {
let mut transactions = HashMap::new();
let mut balances = HashMap::new();
for block in active_blocks {
for transaction in block.transactions() {
transactions.insert(*transaction.id(), transaction);
for input in transaction.inputs() {
Self::process_input(&transactions, &mut balances, input);
}
for output in transaction.outputs() {
Self::process_output(&mut balances, output);
}
}
}
balances
}
}
To process a transaction input we need to look at two cases:
- A transaction is coinbase, which means it has exactly one input and one output. The input comes from mining the block, so it doesn’t spend any existing transaction outputs. Therefore, we don’t decrement anything in this case.
- A transaction is not a coinbase, which means its transaction inputs spend some transaction outputs. We want to decrement the balances for spent transaction outputs. To do that, we use the local transactions state to lookup the output that is spent and decrement the balance of the public key associated with it.
// learncoin/src/commands/account_balances.rs
impl AccountBalances {
// ...
fn process_input(
transactions: &HashMap<TransactionId, &Transaction>,
balances: &mut HashMap<PublicKey, i64>,
input: &TransactionInput,
) {
// Decrement the balance for spent transaction output.
// Coinbase inputs do not take money from any transaction output, so we ignore
// those.
if !input.is_coinbase() {
// Spend the referenced transaction output.
let txid = input.utxo_id();
match transactions.get(txid) {
None => {
panic!("Invalid blockchain. Unknown transaction: {}", txid)
}
Some(transaction) => {
match transaction
.outputs()
.get(input.output_index().value() as usize)
{
None => {
panic!(
"Invalid blockchain. Unknown transaction output at index: {}",
input.output_index()
)
}
Some(output) => {
let public_key = output.locking_script().public_key();
// Safety: We must have already processed the TransactionOutput,
// so it's fine to assume that the key exists.
let mut balance = balances.get_mut(public_key).unwrap();
*balance -= output.amount();
}
}
}
}
}
}
Processing transaction outputs is simpler. We simply increment the balance for each public key associated with the transaction output by the corresponding amount.
// learncoin/src/commands/account_balances.rs
impl AccountBalances {
// ...
fn process_output(balances: &mut HashMap<PublicKey, i64>, output: &TransactionOutput) {
// Increment the balance for unspent transaction output.
let public_key = output.locking_script().public_key().clone();
// Ensure that the key exists if it's the first time we're seeing the account address.
let balance = balances.entry(public_key).or_insert(0);
*balance += output.amount();
}
}
Extending Client CLI to get transaction info and confirmations
It is useful to query the current state of the transaction, so let’s add that functionality to our client CLI.
We’d like to use the transaction ID returned by the send-transaction command to see how many confirmations the transaction has, as well as its data.
The API should look as follows:
./learncoin client --server 127.0.0.1:8333 get-transaction --id 3ecbc61a55a8b8844037005b97e7442d04ab191fa43af74b465914871e3fb99d
# Print "Confirmations: <num_confirmations>"
# Print "<transaction_data>"
Since we’ve already implemented many client commands in this post, I’ll leave this one to you as an exercise. We don’t have to extend the JSON RPC protocol to implement this. Instead, we can query the whole blockchain and find the corresponding transaction by its ID. Then we can count how many blocks exist in the active blockchain after the one that contains our transaction to count the number of confirmations.
If you’re stuck, you can see the code here.
Sending our first transaction
Let’s run the learncoin network with two nodes and two miners, one miner connected to each node.
Each miner would use a different public key to lock the outputs for mined blocks and we’ll call them bob-address and alice-address.
You may need to run below commands in different terminals or as background processes.
# Start Bob's node.
./learncoin server 127.0.0.1:8333 --miner-public-key bob-address
# Start Alice's node.
./learncoin server 127.0.0.1:8334 --miner-public-key alice-address --peers 127.0.0.1:8333
# Start Bob's miner.
./learncoin miner --server 127.0.0.1:8333
# Start Alice's miner.
./learncoin miner --server 127.0.0.1:8334
After waiting some time for blocks to be mined, Bob and Alice earned some money from mining.
We can verify this by running get-balances command.
./learncoin client --server 127.0.0.1:8333 get-balances
alice-address: 1300
bob-address: 1250
genesis-address: 50
Isn’t this awesome? It’s the first time we are seeing something tangible in the LearnCoin network.
Let’s take it to the next level and spend one of Bob’s transaction inputs by locking 30 coins with a new public key nikola-address.
We also lock 20 coins with bob-address - remember that transaction outputs are always spent in full, so if we want to transfer part of it we must explicitly lock the change.
Otherwise, we would give some miner a nice fee.
./learncoin client --server 127.0.0.1:8333 send-transaction --input "7dcc3ec823d83644072b61432ff1641f44569bf96d7a17e10d253dceabfd29db:0" --outputs "nikola-address:30,bob-address:20"
3ecbc61a55a8b8844037005b97e7442d04ab191fa43af74b465914871e3fb99d
I’ve waited for a bit to for the transaction to be included in the next block before checking the balances again.
./learncoin client --server 127.0.0.1:8333 get-balances
alice-address: 1350
bob-address: 1220
genesis-address: 50
nikola-address: 30
Awesome, the transaction has been processed successfully. Notice that alice has a bit more money because her miner found the last block and got the reward of 50 coins.
Let’s get the transaction info to see how many confirmations it has.
./learncoin client --server 127.0.0.1:8333 get-transaction --id 3ecbc61a55a8b8844037005b97e7442d04ab191fa43af74b465914871e3fb99d
Confirmations: 1
Transaction { id: TransactionId(3ecbc61a55a8b8844037005b97e7442d04ab191fa43af74b465914871e3fb99d), minimum_block_height: 52, inputs: [TransactionInput { utxo_id: TransactionId(30b896fef8e02b1a88a490c1f9303e68a8c1fc339612d73c1ff18f931dd24fd8), output_index: OutputIndex(0), unlocking_script: UnlockingScript }], outputs: [TransactionOutput { locking_script: LockingScript { public_key: PublicKey("nikola-address") }, amount: 30 }, TransactionOutput { locking_script: LockingScript { public_key: PublicKey("bob-address") }, amount: 20 }] }
What about security?
As you may have noticed, our current implementation has many vulnerabilities:
- Transactions can spend any outputs multiple times (double-spending)
- Anyone can spend any transaction outputs, so it’s possible for Bob to spend Alice’s money and vice-versa.
- Transactions can transfer more coins than what exists in the transaction inputs, so it’s possible to create coins out of nothing.
- etc.
The above are serious issues and the coin would have no value unless we address all of the above.
Summary
In this post, we have introduced a public key to lock transaction outputs, but we didn’t implement the unlocking logic yet. We use it only to associate some account address with coins. To show this, we added a couple of commands in the Client CLI.
Here’s the code from this blog post.
In the next blog post, we will implement basic transaction validation to address some vulnerabilities.