An Overview of OPEN State and Other Technical Updates

Hello OPEN Community,

We would like to provide an update on OPEN State, the crucial role it plays in providing interoperability, and an overview of other technical developments that we have been working on. 

The Current Landscape

The number of digital assets and blockchains is growing rapidly. However, blockchains and related solutions remain isolated, operating and developing in their own respective silos. We’re seeing a rise in unique solutions within their own respective environments. They are each providing a unique solution to their approach to decentralized technologies.

Blockchain technology is becoming more appealing amongst the masses. And more and more developers face and will face the problem of interoperability of their application with different blockchains and distributed ledgers.

Interoperability can provide developers with the ability to combine the advantages of optimizations of different consensus mechanisms and virtual machines, in many cases off-loading tasks that are better accomplished on other chains and letting each focus on its core competencies. In the emerging world of blockchains, interoperability can make them more useful, user-friendly, efficient, and scalable. 

Nowadays, there are numerous blockchains that are popular and widely used. It is quite a challenge to integrate all of them into an application because developers need to integrate each blockchain separately into their solution. Also, blockchain ecosystems can differ a lot from each other. 

Thousands of developers could waste a lot of time and energy searching for or creating solutions to this problem. They often need to integrate a batch of different technologies to achieve their goals.

In some cases, you cannot even integrate blockchains directly and need to use some kind of middleware (i.e. side-chains or state-channels). It makes development complicated, long, and expensive. All these technical roadblocks compel us to develop an approach which provides a simple and convenient way to interoperate with different blockchains.

INTEROPERABILITY

CROSS-CHAIN

When we talk about interoperability, we need to be a little more specific about what it means. On high-level, interoperability can be split into two categories:

1. Exchange of digital assets – the ability to transfer and exchange assets originating from different blockchains without trusted intermediaries (e.g. centralized exchanges)
2. Exchange of arbitrary data – the ability to make advanced blockchain-to-blockchain API calls which can go as far as having smart contract code on one chain verify the consensus finality of events on other chains directly

The first option can be constructed atop blockchains which have fairly simple programming capabilities, as users on both sides can easily produce publicly verifiable signatures for actions that enable atomic transfers or transfers which only complete if both sides do their part.

The second option is the more difficult type of interoperability, as most blockchains are passive systems that can’t produce a signature that can be easily verified by others, and thus needs more complex proofs about their state to be understood by outside parties. 

Vitalik Buterin wrote in this paper that there are 3 primary technical approaches to achieving interoperability:

• Centralized or multisig notary schemes, where a party or a group of parties agree to carry out the action on chain B when some event on chain A takes place
• Sidechains/relays (systems inside of one blockchain which can be validated and read events and/or state in other blockchains)
• Hash-locking (setting up operations on chain A and chain B that have the same trigger, usually the revelation of the preimage of a particular hash)

A notary mechanism can be considered to be the technologically simplest way to reach cross-chain interoperability. This approach utilizes trusted intermediaries to attest to chain A that a given event on chain B took place or that a particular claim about chain B is true.

Instead of interaction with intermediaries, relays allow the chains to get information about the target chain by themselves. A relay is a contract on chain A that functions as a light client of chain B, using chain B’s standard verification procedure to verify block headers fed into the contract.

Hash-locking is the most practical technical approach to interoperability and as one might expect, also the most limiting in terms of functionality. A hashed timelock contract (HTLC) is a class of blockchain-based payments that use hashlocks and timelocks to require the receiver of a payment to either acknowledge receipt prior to a deadline or forfeit the ability to claim the payment, returning it to the payer. HTLCs allow for cross-chain atomic swaps and fully funded bi-directional payment channels between assets on certain types of blockchains.

OPEN State

OPEN Platform is not intended to provide cross-chain interoperability. It provides the necessary payment infrastructure such as coin transfers and token transfers. The key feature of Open State is providing transaction tracking options for supported blockchains. There is no need to make complex integrations and no need to use any SDKs. Transaction notifications are implemented via webhook invocation using HTTP protocol, so it provides interoperability between off-chain and on-chain worlds.

Such an infrastructure is a complicated system which consists of several essential parts. These parts should interoperate with each other, and this is a vital point of the whole system. That’s why we need to provide intrinsic interoperability first.

As mentioned above, interoperability between system parts is crucial. Given the system complexity, each part should comply with the single responsibility approach. Each system part should implement a logical atomic set of functionality. It provides efficient scalability and system extensibility. For these purposes the system splits into OPEN API, OPEN Scaffold and OPEN State. Whereas a user interacts with OPEN API and OPEN Scaffold interacts with a blockchain, OPEN State provides general interoperability.

OPEN Platform needs to provide data exchange between the system parts. OPEN State collects all necessary data from OPEN API, blockchains, and smart contracts, storing it in a distributed ledger. Using decentralized storage provides safety and reliability. Also, OPEN State provides data to other system parts.

The way OPEN Platform implements interoperability is described in more detail in the next section.

OPEN PLATFORM

OPEN Platform provides necessary payment infrastructure for application developers to utilize the blockchain, including cryptocurrency technologies. OPEN Platform is a hybrid technology which harmoniously combines on-chain and off-chain approaches.

At the current moment, OPEN Platform consists of several essential parts:

OPEN API provides a convenient API to integrate with OPEN Platform. OPEN API interacts with a blockchain via OPEN Scaffold.
OPEN Scaffold is a smart contract which provides money flow.
OPEN SDK provides necessary tools for application developers.
OPEN State is intended to track and store a wallet’s state.
OPEN CHAIN serves as multipurpose blockchain technology.

OPEN Scaffold is an abstraction which guarantees the ability to create convenient asset flow. To integrate a blockchain with OPEN Platform, the blockchain should support smart contracts. Then it is necessary to create a smart contract template which implements the OPEN Scaffold approach. The smart contract template provides a developer with a set of adjustments for creating a specific smart contract that satisfies the developer’s needs. Then it can be deployed via OPEN API. After deploying, the developer saves the ability to manage the smart contract.

OPEN API is intended to provide the opportunity to deploy OPEN Scaffold into a target blockchain. Users can interact with OPEN API via its UI. Application developers can integrate with OPEN API through OPEN SDK.

OPEN State is partially based on relay’s approach and built atop a fully managed ledger database which provides a transparent, immutable, and cryptographically verifiable transaction log ‎owned by the central trusted authority. OPEN State tracks each and every wallet state change and maintains the complete and verifiable history of changes over time.

OPEN Chain is a high-performance DPoS-based blockchain network which is going to solve the throughput issue of existing blockchain networks. OPEN Chain will be suggested by default for unspecified transactions for load-balancing purposes.

With OPEN Platform, developers can receive convenient, robust and powerful tools for integrating cryptocurrency payments into their applications. All payment history is reliably stored in the distributed ledger and can be provided at any moment.

The hybrid architecture of OPEN Platform provides several advantages:

Usability – to experience the power of interoperability in an intuitive and easy manner, developers should integrate with OPEN Platform.
Interoperability – at the current moment, OPEN Platform provides partial cross-chain interoperability. However, further development will bring the full power of cross-chain interoperability.
Extensibility – the number of integrated blockchains may be increased at any moment. Also, OPEN Platform’s functionality can be extended by adding on new parts of the platform.
Efficiency – OPEN State may provide the ability to decrease the number of transactions in target blockchains.

If we dive deeper into the OPEN Platform mechanism, we can define five interesting points. To help understand this, please view the diagram below:

 

1. OPEN API gets a user’s request to deploy OPEN Scaffold smart contracts.
2. OPEN API deploys Scaffold smart contracts into required blockchains.
3. After successful smart contract deployment, OPEN API transfers necessary data to OPEN State.
4. OPEN State creates the subscription on smart contract events and sets up a webhook based on the received data. 
5. After the event is received, OPEN State performs a webhook on the appropriate URL.

OPEN STATE

OPEN State provides an open source solution for tracking wallets, utilizes the reactive approach,  and has been developed with a beautiful architecture. For a developer to accept cryptocurrency, they need to implement many different things. Firstly, there might be many different options, such as Bitcoin and Ethereum. For each currency, the developer needs to implement its integration. With regards to the integration, there are also many options such as native integration or payment gateway integration. Each native integration is usually very complicated and might be vulnerable.

Many backend services these days are REST-ful (i.e. they operate over HTTP), so the underlying protocol is fundamentally blocking and synchronous. Microservice calls or API calls often involve calling other services and then even more services depending on the results from the first calls. With so much IO going on, if you were to wait for one call to complete before sending the next request, your poor client would give up in frustration before you managed to assemble a reply. So external service calls, especially complex orchestrations of dependencies between calls, are a good thing to optimize. In OPEN State, as we call external APIs over HTTP such as getting the state of blocks in Ethereum, we utilize Kotlin coroutines. Additionally, our storage libraries provide reactive drivers.

The architecture of tracking wallets provides an easy way to extend and multiply the number of integrated blockchain networks. And it is not only powerful from the performance perspective, but also extensible. Furthermore, it is already integrated with Ethereum and in the near future, it will support Bitcoin and BNB (Binance blockchain).

Blockchain tracking

The architecture of wallet tracking in OPEN State is implemented in such a way that makes it easier to add new integrations into blockchains. To do this, you just need to create your own class, inherit from a blockchain, and implement its methods. After that, it is imperative to mark it as a bean, so that later your implementation will be noticed by the Spring context.

There are two jobs involved in wallet tracking:

• Blockchain processor
• Blockchain checker

Blockchain processor

This processor wakes up at a certain time with a fixed delay after each execution. Utilizing a coroutine scope, it receives one random blockchain from the Redis queue that needs to be processed. After, it will immediately try to lock this blockchain for processing by adding a value in Redis. The idea behind this solution is that more than one OPEN State instance will be launched on the servers and to not allow several jobs to do the processing at the same time. This solution allows you to not load servers and guarantees the correct processing of blocks. If the processor fails to take over(lock) for processing, then we would consider that this blockchain is already being processed by another job. Otherwise, we get the number of the last processed block for the current blockchain from Redis. After, through this block number, we get all transactions and find wallets that need to be tracked, saving them in MongoDB. Furthermore, after processing, we increase the number of the currently processed blocks in Redis. At this point, it is worth emphasizing that after each block is processed, we update the lock that we mentioned earlier, since on Redis this is implemented through a timeout. That is, if the processor does not complete the work at the specified time, then the blockchain will be automatically removed from the locks in Redis. Accordingly, another processor will take over. This whole process is performed in a loop until this job processes til the last block(last> current). After iterating these operations, it is logical to release the lock for the current blockchain by deleting the value in Redis.

The structure of storage in Redis is as follows:

It is worth noting that the blockchain used below is the name of the class that is inherited from the parent blockchain class. For example, the already integrated Ethereum – in this case, it will be EthereumBlockchain.

• [queue]: blockchain – Used to store blockchain queues. Blockchain checker adds when a new block appears and the blockchain processor reads.
• [lock: blockchain]: timestamp – Stored for locking blockchain described in blockchain processor.
• [blockchain: current]: value – Current processed block number stored
• [blockchain: last]: value – Last known block number stored

Blockchain checker

This job wakes up at a certain time with a fixed delay after each execution. This process is needed to periodically check for new unprocessed blocks. Firstly, when launching the Kotlin coroutine, it will go through each blockchain with which OPEN State was integrated. Also, for each blockchain, it will request from the implementation (by calling the getLastBlockNumber method which obliges to implement this method) the last block number in the system. In addition, when receiving information about the last known block from Redis, it compares these block numbers. If these values are not equal, then we update the value of the last block in Redis for a blockchain. Additionally, we add the current blockchain in the queue in Redis so that the Blockchain processor will be processed later. Otherwise, if these values are equal, we simply end the processing of the current blockchain, since we already know the current block number and there is no need to trigger Redis.

Webhook Invocation

What is a webhook invocation?

A webhook invocation is the URL to which an HTTP request should be sent if this webhook wallet participates in a blockchain transaction (i.e. fund transfer).

This architectural decision was made to provide the system with flexibility, a guarantee of webhook delivery, and even to load distribution to instances.

Single Responsibility principle states that every object should have one responsibility and this responsibility must be completely encapsulated. Following this principle, it was divided into several processors, each one responsible for its specific task. Processors in this context are jobs that wake up at a certain time and perform a task:

• Blockchain processor
• Webhook processor
• Failover processor
• After the blockchain processor has processed the block and found a transaction in it that involves a wallet in the system, it adds to a special queue for webhooks in Redis which will be further processed by special jobs.
• The structure of storing webhooks in Redis is as follows – since Redis stores data in the form of “key and value” and provides a queue system, we utilize these capabilities:

1. The first queue stores information about the wallet. The key will be [queue: webhook] and the value will be a json object that stores the wallet address and blockchain type. It would be correct to mention that the values for the key will be unique, since this will allow webhooks to be performed for a specific wallet consistently.
2. The second queue stores transactions in the wallet, which needs to send a webhook. For example, the key will be [address: type: webhook] and the value is the transaction hash.

Webhook processor

A webhook processor makes only one webhook invocation for one wallet at a time. This job wakes up at a certain time with a fixed delay after each execution. This allows for the avoidance of spam and overloading of remote servers. In addition, this starts several instances of OPEN State to make webhook invocation faster. Logically, the architecture of a webhook invocation can be divided into 2 main parts:

• Webhook queue
• Webhook invocation

The webhook invocation algorithm is presented in the diagram below:

Webhook queue

All detected transactions in a blockchain processor are added to a webhook queue. A webhook queue consists of 2 parts:

• A wallet execution queue contains a set of objects with information about a wallet attempts counter and execution timestamp. There is only one such object for a wallet in the queue. A task in a queue can be rescheduled (i.e. changed execution timestamp) or removed if webhooks are indicated for all transactions of a wallet. The queue is implemented as ZSet in Redis and an added timestamp is used as a score.
• A list of transactions is created for each wallet in the queue. It is implemented as a list in Redis. All transactions from a blockchain processor are added to the end of the list. Processed transactions are removed from the list  and processing starts from the first record in the list.

Webhook invocation

The webhook invocation processor takes the task that is closest to the current timestamp from the webhook’s queue and tries to execute it. Every task is executed. It retries task execution if the webhook invocation fails. The webhook invocation processor can be configured with the following properties:

• Webhook invocation processor execution delay – amount of milliseconds between each processor invocation.
• Lock time out – amount of milliseconds to lock an object in the wallet queue.
• Short time period attempts count – max count of attempts for webhook invocation retries in a short time period. The delay is equal to the number of seconds taken from a Fibonacci row for each attempt. Default value is 10.
• Long time period attempts count – max count of attempts for webhook invocation retries in a long time period. Each attempt would run once per day during a specified count of days. Default value is 17.

The webhook invocation executes as follows:

• Get the next task from the webhook queue and lock it. If the lock fails, this means that task was already executed by another process and WIP will try to get the next task from the queue.
• Get the first record from the transactions list associated with the current wallet and invoke an HTTP request.
• If the HTTP response is successful (2xx HTTP response code), remove the transaction from the list. If the list becomes empty, remove the wallet from the webhook queue and delete the associated list of transactions.
• If the webhook fails, it would be re-scheduled to execute later after a delay. The delay is calculated as follows: fb(n) * second. After reaching a short time period attempts count, the webhook would be re-scheduled to run once per day. After reaching a long time period attempts count, the webhook invocation would be blocked for the wallet.
• If all attempts fail, the webhook task would be removed from the queue and a list of transactions for the wallet would be moved to a special collection in MongoDB. The wallet’s transactions would not be processed until a webhook url or a webhook status is changed.
• Save webhook invocation information to MongoDB (number of attempts, last received HTTP response code, message from remote server, and an invocation timestamp).
• Remove lock for wallet.

Webhook invocation payload

A webhook payload would be sent via HTTP POST request and include the following data:

• blockchain – Name of blockchain
• walletAddress – Address of wallet that received transaction
• transaction – Information about transaction

A transaction object would include the following data:

• hah – transaction hash
• blockHash – hash of block
• blockHeight – height of block
• amount – amount of transaction
• date – date of transaction
• to – address of receiver wallet
• from – address of sender wallet

Example of payload:

{ “blockchain”: “Ethereum”, “walletAddress”:
“0x9D86b1B2554ec410ecCFfBf111A6994910111340”, “transaction”: { “hash”: “0x9D86b1B2554ec410ecCFfBf111A6994910111340”, “blockHash”:
“0x3aabcf5ca3ec5e459d9aa1314d67c6595beac810”, “blockHeight”: 124414145,
“amount”: 0.314141, “date”: “2020-01-12T00:22:43.511Z”, “to”:
“0x9D86b1B2554ec410ecCFfBf111A6994910111340”, “from”:
“0x3aabcf5ca3ec5e459d9aa1314d67c6595beac810” } }

Failover processor

This processor is needed in order to periodically check that the webhook processor has not exceeded the processing time.

This job works in the following ways: 

1. Takes an item from the [lock: webhook] queue in Redis, which will already be sorted by date (added in descending order). 
2. Next, it checks that the webhook execution time for this wallet has not been exceeded. If so, it will count that the webhook processor has not processed this webhook and queues it for processing [queue: webhook] .

This processor guarantees delivery if the webhook processor does not finish processing the webhook.

For more details on OPEN State, please refer to our GitHub repository. Stay tuned for more updates!