Introduction
Catastrophic mistakes occur in the world of distributed systems. Imagine accidentally deleting a whole bunch of clusters from your blockchain infrastructure. Such incidences can trigger devastating consequences.
The best way to avoid such disasters is keeping appropriate data reserves. When dealing with enterprise blockchain frameworks in container orchestration frameworks, having data in reserve is absolutely critical, especially when implementing a kubernetes blockchain backup strategy and a reliable container orchestration backup strategy.
The combination of these technologies provides a powerful and highly secure platform for processing blockchain transactions. The framework itself is an open source solution hosted under community governance that offers a platform for all distributed ledger related implementations.
This framework has large benefits such as high performance and scalability. It provides a high degree of trust between participants with known identities and rich queries over an immutable ledger.
This guide offers an end-to-end guide to blockchain backup and recovery, protecting and recovering data on blockchain networks that run in containerized environments with no issues, including hyperledger fabric disaster recovery scenarios.
Understanding why data reserves are necessary is the basis of this process.
Why Data Reserves Are Essential
Creating data reserves is not optional but mandatory due to several critical reasons.
System Dependability
Having data reserves enables teams to operate with greater confidence. When using storage that is based on the cloud for backups then a further level of security prevents different forms of data loss caused by:
- •Natural disasters
- •Human error
- •Malicious attacks
For example, if errors happen on pod restarts or crashes on the correct mount path configuration for an application in the deployment specification, data loss is inevitable without proper backups.
System Robustness
Data reserves help to make applications more robust in case of unexpected crashes and errors. Team members may accidentally erase volumes during normal operations. Having reserves allows applications to be restored to the most up-to-date state, reducing downtime and data loss.
Protection and Standards Adherence
Having reserves helps to improve the overall security of applications, and ensures compliance with industry standards by protecting against unexpected data corruption events and malicious attacks. This is especially critical in the regulated industries where data integrity standards are high.
Infrastructure Modifications
Data reserves make it easy to navigate through infrastructure modifications. When migrating blockchain networks to different cloud providers, different storage mechanisms might be necessary in order to reduce costs. Having comprehensive backups makes such transitions smooth and risk-free.
Network Implementation
This guide assumes a pre-existing blockchain network setup on a container orchestration platform with dedicated namespaces for every organization where all its services operate.
The network set-up should have a namespace for each organization, and within each namespace should have certain types of deployments.
Each organizational namespace will typically have:
- •Some peer nodes
- •Ledger databases
- •Certificate authority servers for organizations that need to register new identities
- •Certificate authority server databases
- •Ordering service nodes for organizations that contribute to the ordering service
Data Persistence in Container Environments
Stateful applications need to store and retrieve data to function properly. The application can manage this responsibility itself or it can delegate it totally or partially to another stateful application such as databases.
Ultimately, everything is stored as files or folders. These applications are usually configured to use certain file paths on which they store persistent data.
Container deployments are used to achieve blockchain data persistence using combinations of:
- •Persistent volume claim mounts with persistent volumes
- •Configuration maps
- •Secure storage objects
Sensitive data such as cryptographic keys and certificates are mounted as secure storage objects. Configuration data such as specification files are mounted as configuration maps.
This means that the files and folders that are found in all persistent volume claim mount paths, secure storage objects, and configuration maps make up the full set of data for that deployment.
Protect Your Blockchain Infrastructure Today
Implement comprehensive backup strategies to safeguard your blockchain data and ensure business continuity.
Peer Node Data Organization
Peer nodes store some data locally at default locations and delegate the rest of the data to ledger databases.
The transport layer security folder contains the certificate and key pair for the peer node.
The fabric configuration folder includes specification files that contain the configuration that was used when the peer was started.
The membership service provider folder holds the membership information for peers, enrollment certificate and key, certificate chain from the certificate authority server.
The admin membership service provider folder holds administrator credentials.
The production folder holds parts of the ledger data which include:
- •The blockchain itself
- •Installed chaincode and lifecycle data
- •Private data stores
- •Transient stores included during transaction submission
Ordering Service Data Organization
Ordering service nodes store most data locally at default locations. Several folders are similar to those in peer nodes, but these are used to contain:
- •Ordering service membership information
- •Transport layer security certificate and key pairs
- •Enrollment certificate and key pairs
The ledger folder stores the chains that this ordering service is a part of and waiting operations.
The production folder contains the files specific to a particular consensus algorithm and snapshots.
Certificate Authority Server Data Organization
Certificate authority servers store some data locally at default locations and the remainder in certificate authority server databases.
The fabric certificate authority folder contains:
- •Server certificates
- •Transport layer security certificates
- •Configuration files that are used when the service was started
When restoring you may want to retain and use new transport layer security certificates and the server configuration file for the certificate authority since it represents the latest profile changes but replace remaining contents such as enrollment certificates with content from the previous cluster.
Strategies for Data Preservation
First, ensure you preserve secure storage objects and configuration maps. There are several methods for backing up data that exists in volumes.
One way is by taking snapshots of the volumes that these services use. Another way is to use specialized services or tools specifically designed for this purpose with different features and capabilities.
Choose the right preservation strategy based on your needs and context, some approaches may make it easier to migrate workloads across clusters, and some may not.
Data Recovery Procedures
Recovery strategy depends on the preservation strategy implemented. Refer to the appropriate restoration documentation for your method of choice.
Begin by recreating secure storage objects and configuration maps from preserved data. Next is to put services to sleep and copy preserved data to the pod.
Repeat this process for all services across all namespaces, taking care of restoring normally mounted data.
Start independent services like ledger databases and certificate authority server databases first then proceed with the rest of the services. This staged approach helps to ensure dependencies are met before dependent services are started.
Verifying Preservation Consistency
Blockchain networks are distributed systems, so backup and restore processes present data consistency challenges.
At any given moment the height of the blockchain at peer nodes may vary due to various reasons including:
- •The network speed
- •Geographical proximity
- •The type of peer node
Leader peer nodes are the first to receive blocks from the ordering service, so they may commit blocks first. Some peer nodes may suffer internal crashes which they recover from by repeating steps, which has the natural consequence of committing blocks later than others.
This can be mitigated in two ways. First, create a no transaction window where no transactions are sent to the network. Next, query the state of the ledger on the blockchain using peer command line interface commands to make sure that all the peer nodes are at the same height before starting the preservation process.
After restoration of data, repeat the verification process to make sure that the restoration was done properly. View peer and ordering service logs to ensure that everything is working properly.
This verification step is critical to ensure that the restored network functions identically to the original network before the point of preservation.
Final Thoughts
This guide has delved into how and where blockchain networks on container orchestration platforms store data, the benefits of preserving data and how to restore or migrate networks along with verification tests to ensure success.
This process requires careful planning and execution. Understanding the data organization of different types of nodes is essential to successful preservation and recovery.
Each component stores data in a different way and holistic approaches to preservation must consider all these variations.
Recovery is not a matter of making copies of files to their original locations. It involves careful sequencing, which begins with independent services and progresses to dependent ones.
The distributed nature of blockchain networks adds some complexity and means that additional verification steps must be taken to ensure that all nodes are in sync after recovery.
The distributed nature of Blockchain networks means that there is more complexity, and we need to have robust preservation and recovery procedures in place to ensure business continuity and protect valuable blockchain data in the event of:
- •Accidental deletions
- •Infrastructure migrations
- •Disaster recovery scenarios
Regular testing of these procedures is equally important, to make sure that they work correctly when actually needed.
