Blockchain, cryptography, and

consensus
Christian Cachin
(with Elli Androulaki, Angelo De Caro, Andreas Kind, Mike Osborne, Simon
Schubert, Alessandro Sorniotti, Marko Vukolic and many more)

IBM Research – Zurich

June 2017
Connected markets
‣ Networks connect participants
– Customers, suppliers, banks, consumers

‣ Markets organize trades

– Public and private markets

‣ Value comes from assets

– Physical assets (house, car ...)
– Virtual assets (bond, patent ...)
– Services are also assets

‣ Transactions exchange assets

2
Ledger
‣ Ledger records all business activity as
transactions
– Databases

‣ Every market and network defines a

ledger

‣ Ledger records asset transfers between

participants

‣ Problem — (Too) many ledgers

– Every market has its ledger
– Every organization has its own ledger

3
Multiple ledgers
Ledger ‣ Every party keeps its own
B
Bob Ledger ledger and state
C
Ledger ‣ Problems, incidents, faults
A Alice Charlie
‣ Diverging ledgers

Frank Dave

Ledger Ledger
Eve D
F
Ledger
E
4
Blockchain provides one virtual ledger
‣ One common trusted ledger
Bob
‣ Today often implemented by a
centralized intermediary
Alice Charlie
‣ Blockchain creates one single
One
ledger for all parties
Ledger

‣ Replicated and produced

Frank Dave
collaboratively

Eve ‣ Trust in ledger from

– Cryptographic protection
– Distributed validation
5
Four elements characterize Blockchain

Replicated ledger Cryptography

● History of all transactions ● Integrity of ledger
● Append-only with immutable past ● Authenticity of transactions
● Distributed and replicated ● Privacy of transactions
● Identity of participants

Consensus Business logic

● Decentralized protocol ● Logic embedded in the ledger
● Shared control tolerating disruption ● Executed together with transactions
● Transactions validated ● From simple "coins" to self-enforcing
"smart contracts"
6
Blockchain simplifies complex transactions

Logistics Property records Capital markets

Real-time visibility Digital but unforgeable Faster settlement times

Improved efficiency Fewer disputes Increased credit availability
Transparency & verifiability Transparency & verifiability Transparency & verifiability
Reduced cost Lower transfer fees No reconciliation cost

7
Blockchain scenario features
‣ A given task or problem, but no (central) trusted party available

‣ Protocol among multiple nodes, solving a distributed task

– The writing nodes decide and reach consensus collectively

‣ Key aspects of the distributed task

– Stores data
– Multiple nodes write
– Not all writing nodes are trusted
– Operations are (somewhat) verifiable

‣ If all writing nodes are known → permissioned or consortium blockchain

‣ Otherwise, when writing nodes are not known → permissionless or public blockchain

8
Why blockchain now?
‣ Cryptography has been a key technology in the financial world for decades
– Payment networks, ATM security, smart cards, online banking ...

‣ Trust model of (financial) business has not changed

– Trusted intermediary needed for exchange among non-trusting partners
– Today cryptography mostly secures point-to-point interactions

‣ Bitcoin started in 2009

– Embodies only cryptography of 1990s and earlier
– First prominent use of cryptography for a new trust model (= trust no entity)

‣ The promise of Blockchain – Reduce trust and replace it by technology

– Exploit advanced cryptographic techniques

9
 What is a blockchain?

10
When you get too many people talking
about the same thing it tends to clutter up
things.

Bob Dylan, 1965

11
A state machine
‣ Functionality F
– Operation o transforms a state s to new state s' and may generate a response r
r
o
(s', r) ← F(s, o)
s s'

‣ Validation condition
– Operation needs to be valid, in current state, according to a predicate P()
r
o

s s'

12
P(s,o) = TRUE
Blockchain state machine
‣ Append-only log
– Every operation o appends a "block" of valid transactions (tx) to the log

s s'

‣ Log content is verifiable from the most recent element

‣ Log entries form a hash chain

ht ← Hash( [tx1, tx2, ... ] || ht-1 || t) .

13
Example – The Bitcoin state machine
‣ Bitcoins are unforgeable bitstrings
– "Mined" by the protocol itself (see later)

‣ Digital signature keys (ECDSA) own and transfer bitcoins

– Owners are pseudonymous, e.g., 3JDs4hAZeKE7vER2YvmH4yTMDEfoA1trnC

‣ Every transaction transfers a bitcoin (fraction) from current to next owner

– "This bitcoin now belongs to 3JDs..." signed by the key of current owner
– (Flow linkable by protocol, and not anonymous when converted to real-world assets)

‣ Validation is based on the global history of past transactions

– Signer has received the bitcoin before
– Signer has not yet spent the bitcoin

14
Distributed p2p protocol to create a ledger
Nodes
produce
transactions

Nodes run a
o1 o2 o3 protocol to
construct the
s0 s1 s2 s3 ledger

15
Blockchain protocol features
‣ Only "valid" operations (transactions) are "executed"

‣ Transactions can be simple

– Bitcoin tx are statement of ownership for coins, digitally signed
"This bitcoin now belongs to K2" signed by K1

‣ Transactions can be arbitrary code (smart contracts)

– Embody logic that responds to events (on blockchain) and may transfer assets in
response
– Auctions, elections, investment decisions, blackmail ...

16
 Consensus

17
Three kinds of consensus for blockchain
‣ Decentralized / permissionless
– Bitcoin, Ethereum

‣ Somewhat decentralized
– Ripple, Stellar

‣ Consortium / permissioned
– BFT (Byzantine fault tolerance) consensus

18
Decentralized – Nakamoto consensus/Bitcoin
‣ Nodes prepare blocks
– List of transactions (tx)
– All tx valid

‣ Lottery race
– Solves a hard puzzle
– Selects a random
winner/leader
– Winner's operation/
block is executed and
"mines" a coin

‣ All nodes verify and

validate new block
– "Longest" chain wins
19
Decentralized = permissionless
‣ Survives censorship and suppression
– No central entity

‣ Nakamoto consensus requires proof-of-work (PoW)

– Original intent: one CPU, one vote
– Majority of hashing power controls network
– Gives economic incentive to participate (solution to PoW is a newly "mined" Bitcoin)

‣ Today, total hashing work consumes a lot of electricity

– Estimates vary, 250-1000MW, from a major city to a small country ...

‣ Protocol features
– Stability is a tradeoff between dissemination of new block (10s-20s) and mining rate
(new block on average every 10min)
– Decisions are not final ("wait until chain is 6 blocks longer before a tx is confirmed")
20
Decentralized – deployment
‣ Bitcoin
– Many (100s? 1000s?) of alt-coins and blockchains

‣ Ethereum
– First digital currency with general-purpose smart contract execution

‣ Sawtooth ledger (Intel contribution to Hyperledger)

– PoET consensus (proof of elapsed time)
● Nodes run PoET program in "trusted execution environment" (Intel SGX)

● PoET waits a random amount of time (say, E[wait] = 10min)

● Creates an attested proof of elapsed time

● Rest like in Bitcoin protocol

21
Somewhat (de)centralized – Ripple, Stellar
‣ Nodes decide whom
to trust
– Validator nodes

‣ Quorum-like protocol
– Among different
heterogeneous
validators

‣ Consistent when
"enough" nodes in
path are trusted by
the nodes involved
in the transaction

22
Somewhat (de)centralized
‣ Every node designates other, well-connected nodes that it trusts
– Decentralization

‣ Transactions among two nodes are agreed if validated by "strong" majority in

the overlap of the trusted node sets
– In practice, leads to centralization

‣ Stellar evolved as fork of Ripple protocol

– Issues with ledger forks and protocol correctness have been reported

‣ Open questions
– Relation to Byzantine quorum systems [MR98]
– How to formally specify properties

23
Consortium consensus (quorums & BFT)
‣ Designated set of
homogeneous
validator nodes

‣ BFT/Byzantine agreement
– Tolerates f-out-of-n faulty/
adversarial nodes
– Generalized quorums

‣ Tx sent to consensus
nodes

‣ Consensus validates tx,

decides, and disseminates
result
24
Consortium consensus = permissioned
‣ Central entity controls group membership
– Dynamic membership changes in protocol
– Membership may be decided inline, by protocol itself

‣ Well-understood problem in distributed computing

– BFT and consensus studied since ca. 1985
● Clear assumptions and top-down design

● 700 protocols and counting [AGK+15]

● Textbooks [CGR11]

● Open-source implementations (BFT-SMaRT)

– Many systems already provide crash tolerant consensus (Chubby, Zookeeper, etcd ...)
– Requires Ω(n2) communication (OK for 10-100 nodes, not > 1000s)

‣ Revival of research in BFT protocols

– Focus on scalability and communication efficiency
25
Consortium consensus – under development
‣ Hyperledger Fabric (originally started by IBM)
– Includes PBFT protocol [CL02]

‣ Tendermint, Juno/Kadena, JPMC Quorum, Axoni, Iroha, Chain and others

‣ HoneyBadgerBFT [MXC+16]
– Revisits practical randomized BFT [CKPS01], including amoritzation

‣ Many existing BFT libraries predate blockchain

– BFT-SMaRT, Univ. Lisbon (github.com/bft-smart/library)
– Prime, Johns Hopkins Univ. (www.dsn.jhu.edu/byzrep/prime.html)

26
More variations of consensus
‣ Bitcoin-NG [EGS+16]
– Bitcoin PoW elects a leader, it is responsible for ordering the next K tx

‣ Proof-of-stake (explored by Ethereum)

– Voting power relative to asset holdings (through cryptocurrency held by blockchain)

‣ Hybrid PoW (PeerCensus [DSW16])

– PoW protocol to elect nodes in one consensus group
– Group runs ordinary BFT consensus

‣ Hierarchical & partitioned, randomized [LNB+15]

– Random sub-groups, nodes and tx assigned randomly to sub-groups
– Each sub-group runs ordinary BFT consensus

27
Scalability–performance tradeoff

M. Vukolic: The Quest for Scalable Blockchain Fabric: Proof-of-Work vs. BFT Replication.
28 Proc. iNetSec 2015, LNCS 9591.
 Validation

29
Validation of transactions – PoW protocols
‣ Recall validation predicate P on state s and operation o: P(s, o)

‣ When constructing a block, the node

– Validates all contained tx
– Decides on an ordering within block

‣ When a new block is propagated, all nodes must validate the block and its tx
– Simple for Bitcoin – verify digital signatures and that coins are unspent
– More complex and costly for Ethereum – re-run all the smart-contract code

‣ Validation can be expensive

– Bitcoin blockchain contains the log of all tx – 105GB as of 3/2017
(https://blockchain.info/charts/blocks-size)

30
Validation of transactions – BFT protocols
‣ Properties of ordinary Byzantine consensus
– Weak Validity: Suppose all nodes are correct: if all propose v, then a node may only
decide v; if a node decides v, then v was proposed by some node.
– Agreement: No two correct nodes decide differently.
– Termination: Every correct node eventually decides.

‣ Standard validity notions do not connect to the application!

‣ Need validity anchored at external predicate [CKPS01]

– External validity: Given predicate P, known to every node, if a correct node decides v,
then P(v); additionally, v was proposed by some node.

– Can be implemented with digital signatures on input tx

31
Public validation vs. private state
‣ So far everything on blockchain is public – where is privacy?

‣ Use cryptography – keep state "off-chain" and produce verifiable tx

– In Bitcoin, verification is a digital signature by key that owns coin

– In ZeroCash [BCG+14], blockchain holds committed coins and transfers use zero-
knowledge proofs (zk-SNARKS) validated by P

– Hawk [KMS+16] uses verifiable computation (VC)

● Computation using VC performed off-chain by involved parties

● P checks correctness of proof for VC

‣ Private computation requires additional assumption (MPC, trusted HW ...)

32
Security and privacy
‣ Transactional privacy
– Anonymity or pseudonymity through cryptographic tools
– Some is feasible today (e.g., anonymous credentials in IBM Identity Mixer)

‣ Contract privacy
– Distributed secure cryptographic computation on encrypted data

‣ Accountability & non-repudiation

– Identity and cryptographic signatures

‣ Auditability & transparency

– Cryptographic hash chain

‣ Many of these need advanced cryptographic protocols

33
 Hyperledger Fabric

34
Hyperledger
‣ A Linux Foundation project – www.hyperledger.org
– Open-source collaboration, developing blockchain technologies for business
– Started in 2016: Hyperledger unites industry leaders to advance blockchain technology
– 135 members in Apr. '17

‣ Incubates and promotes blockchain technologies for business

‣ Today 4 frameworks and 4 tools, hundreds of contributors

‣ Hyperledger Fabric was originally contributed by IBM –

github.com/hyperledger/fabric/
– Architecture and consensus protocols originally contributed by IBM Research - Zurich
35
36
37
38
Hyperledger Fabric
‣ Enterprise-grade blockchain fabric and distributed ledger framework
– One of multiple blockchain platforms in the Hyperledger Project
– First "active" platform under the Hyperledger umbrella (since 3/2017)

‣ Developed open-source, by IBM and others (DAH, LSEG ...)

– github.com/hyperledger/fabric
– Initially called 'openblockchain' and contributed by IBM to Hyperledger project
– Actively developed, IBM and IBM Zurich play key roles

‣ Technical details
– Implemented in GO
– Runs smart contracts or "chaincode" within Docker containers
– Transactions Deploy new chaincode / Invoke an operation / Read state
– Implements consortium blockchain using traditional consensus (BFT, ZooKeeper)
39
Hyperledger Fabric V1
‣ Separate the functions of nodes into endorsers and consensus nodes
– Every chaincode may have different endorsers
– Endorsers have state, run tx, and validate tx for their chaincode
– Chaincode specifies endorsement policy
– Consensus nodes order endorsed and already-validated tx
– All peers apply all state changes in order, only for properly endorsed tx

‣ Functions as replicated database maintained by peers [PWSKA00, KJP10]

– Replication via (BFT) atomic broadcast in consensus
– Endorsement protects against unauthorized updates

‣ Scales better – only few nodes execute, independent computations in parallel

‣ Permits some confidential data on blockchain via partitioning state

40  Data seen only by endorsers assigned to run that chaincode
Separation of endorsement from consensus
‣ Validation is by chaincode Per-chaincode
endorsers
‣ Dedicated endorsers
per chaincode

‣ Consensus service
– Only communication
– Pub/sub messaging
– Ordering for endorsed tx
Consensus service
‣ State and hash chain only orders tx
are common
– State may be encrypted

41
Transactions in Fabric V1
‣ Client
– Produces a tx (operation) for some chaincode (smart contract)
‣ Submitter peer
– Execute/simulates tx with chaincode
– Records state values accessed, but does not change state  readset/writeset
‣ Endorsing peer
– Re-executes tx with chaincode and verifies readset/writeset
– Endorses tx with a signature on readset/writeset
‣ Consensus service
– Receives endorsed tx, orders them, and outputs stream of "raw" tx (=atomic broadcast)
‣ All peers
– Disseminate tx stream from consensus service with p2p communication (gossip)
– Filter out the not properly endorsed tx, according to chaincode endorsement policy
– Execute state changes from readset/writeset of valid tx, in order
42
Transaction flow

43
Modular consensus in Fabric V1
‣ "Solo orderer"
– One host only, acting as specification during development (ideal functionality)

‣ Apache Kafka, a distributed pub/sub streaming platform

– Tolerates crashes among member nodes, resilience from Apache Zookeeper inside
– Focus on high throughput

‣ SBFT - Simple implementation of PBFT (6/2017 - under development)

– Tolerates f < n/3 Byzantine faulty nodes among n
– Focus on resilience

44
 Conclusion

45
Conclusion
‣ Blockchain enables new trust models

‣ Many interesting technologies

– Distributed computing for consensus
– Cryptography for integrity, privacy, anonymity

‣ We are only at the beginning

‣ Blockchain = Distributing trust over the Internet

– www.hyperledger.org
– www.ibm.com/blockchain/
– www.research.ibm.com/blockchain/

46
 References
[AGK+15] P.-L. Aublin, R. Guerraoui, N. Knezevic, V. Quéma, M. Vukolic: The Next 700 BFT Protocols. ACM
TOCS, 32(4), 2015.
[BCG+14] E. Ben-Sasson, A. Chiesa, C. Garman, M. Green, I. Miers, E. Tromer, M. Virza: Zerocash:
Decentralized Anonymous Payments from Bitcoin. IEEE S&P 2014.
[CKPS01] C. Cachin, K. Kursawe, F. Petzold, V. Shoup: Secure and Efficient Asynchronous Broadcast
Protocols. CRYPTO 2001.
[CGR11] C. Cachin, R. Guerraoui, L. Rodrigues: Introduction to Reliable and Secure Distributed
Programming (2. ed.). Springer, 2011.
[CSV16] C. Cachin, S. Schubert, M. Vukolic: Non-determinism in Byzantine Fault-Tolerant Replication.
OPODIS 2016.
[CL02] M. Castro, B. Liskov: Practical Byzantine fault tolerance and proactive recovery. ACM TOCS, 20(4),
2002.
[DSW16] C. Decker, J. Seidel, R. Wattenhofer: Bitcoin meets strong consistency. ICDCN 2016.
[EGS+16] I. Eyal, A. Gencer, E.G. Sirer, R. van Renesse: Bitcoin-NG: A Scalable Blockchain Protocol. NSDI
2016.

47
 References
[KMS+16] A. Kosba, A. Miller, E. Shi, Z. Wen, C. Papamanthou: Hawk: The Blockchain Model of
Cryptography and Privacy-Preserving Smart Contracts. IEEE S&P 2016.
[LNB+15] L. Luu, V. Narayanan, K. Baweja, C. Zheng, S. Gilbert, P. Saxena: A Secure Sharding Protocol For
Open Blockchains. ACM CCS 2016.
[MR98] D. Malkhi, M. Reiter: Byzantine Quorum Systems. Distributed Computing, 1998.
[MXC+16] A. Miller, Y. Xia, K. Croman, E. Shi, D. Song: The Honey Badger of BFT Protocols. ACM CCS 2016.
[V16] M. Vukolic: The Quest for Scalable Blockchain Fabric: Proof-of-Work vs. BFT Replication. LNCS 9591,
Proc. iNetSeC 2015.

48
Hyperledger Fabric references
‣ www.hyperledger.org

‣ Docs – hyperledger-fabric.readthedocs.io/en/latest/

‣ Chat – chat.hyperledger.org, all channels like #fabric-*

‣ Designs – wiki.hyperledger.org/community/fabric-design-docs

‣ Architecture of V1 –
github.com/hyperledger/fabric/blob/master/proposals/r1/Next-Consensus-
Architecture-Proposal.md

‣ Code – github.com/hyperledger/fabric

49
50
Consensus?

51
Consensus?

52
Consensus?

53
Consensus?

54
Consensus?

55
Consensus?

56
Consensus?

57
58
Hyperledger Fabric details (v0.6-preview) / 1
‣ Platform-agnostic
– GO, gRPC over HTTP/2

‣ Peers
– Validating peers (all running consensus) and non-validating peers

‣ Transactions
– Deploy new chaincode / Invoke an operation / Read state
– Chaincode is arbitrary GO program running in a Docker container

‣ State is a key-value store (RocksDB)

– Put, get ... no other state must be held in chaincode
– Non-validating peers store state and execute transactions

59
Hyperledger Fabric details / 2
‣ Consensus in BFT model
– Modular architecture supports other consensus protocols
– Currently, Practical Byzantine Fault Tolerance (PBFT) [CL02]
– Non-determinism addressed by Sieve protocol [CSV16]
– Static membership in consensus group

‣ Hash chain computed over state and transactions

60
Hyperledger Fabric details / 3
‣ Membership service issues certificates to peers
– Enrollment certificates (E-Cert, issued by E-CA)
● Assign identity to peer, gives permission to join and issue transactions

– Transaction certificates (T-Cert, issued by T-CA)

● Capability to issue one transaction (or more)

● Unlinkable to enrollment certificate, for anyone except for transaction CA

‣ Pseudonymous transaction authorization

– Controlled by peer, how many Transaction-Signatures with same T-Cert

61
Hyperledger Fabric details (V0.6)
‣ Peers
– Validating peers (all running consensus) and non-validating peers

‣ Membership service issues identity-certificates and transaction-certificates

‣ Transactions
– Deploy new chaincode / Invoke an operation / Read state
– Chaincode is arbitrary GO program running in a Docker container

‣ State is a key-value store (RocksDB)

– Put, get ... no other state must be held in chaincode
– Non-validating peers store state and execute transactions

‣ Hash chain computed over state (and possibly transactions)

62
Non-determinism in BFT replication [CSV16]
‣ Service-replication paradigm needs deterministic state machines
– Agree on order of operations, then every node executes

‣ What if application is given as black-box? Deterministic? Undecidable!

‣ Our approach – filter out inadvertent non-determinism

– Execute operation, compare results, and revert it if "too much" divergence is evident
– When "enough" nodes arrive at the same result, accept it

‣ If application is randomized
– For algorithmic purpose (Monte Carlo): use master-slave approach
– For cryptography and security functions: cryptographic verifiable random functions (VRF)

63