overview

data propagates through the millix network from one node to another. in normal protocol operations this process occurs in bulk between nodes that are peers, or connected to each other. outside of the protocol data, can be shared when a data request is made via the millix API to a specific node.

broadly speaking, there are two types of data that are propagated: transaction data informing nodes of new transactions and the lineage of transactions required to verify transactions, and network data defining the nodes operating on the millix network. this document explains the process of nodes receiving and storing transaction data during normal protocol operations.

key identifier

each node operates with an active wallet. the active wallet defines what addresses, balances and transactions appear in the node user interface and the private keys that the node uses to send millix. the wallet has a prime address, which is the first address in it's list of available addresses. the prime address is used as the key identifier of the wallet. the key identifier is contained within each address available to the wallet, allowing it to easily recognize transactions (deposits or payments) that affect the wallet balance.

data shards

millix is designed to scale to very large amounts of transactions in terms of speed and storage. millix is designed with sharding to accommodate a larger amount of transactions on the network than a single node would be willing or able to store. data shards act like data buckets. to prevent any bucket from overflowing, and to allow a large volume of data, more shards can continually be added to the millix network. each shard is intended to be limited to a maximum number of transactions representing disk space that is reasonable for a modest device to store without burden.

when a new transaction is created, it is randomly assigned to an existing, established shard on the millix network. to ensure that all transactions are stored online somewhere and available to be retrieved, each node is assigned a specific shard that it is obligated to support. the node is required to store all the transactions that have been assigned to the shard they support. in this way, each node is a full node for that shard. it does not prune transaction records from that shard.

nodes with larger disks can be configured to support more shards than it is obliged to support. this increases the node's efficiency in verifying transactions and increases its earning potential via verification fees.

shard database organization

the main millix database is a database schema called millix. this database contains wallet data, addresses, keys, configuration etc that allows the node to operate on the millix network. within the millix database are seven tables storing transaction data. consider this copy of the transaction tables to be shard zero.

each additional shard is represented by a database file, called a schema, dedicated to storing transaction data. the file is named using the shard id. each shard schema contains a structural copy of the seven transaction tables. the shard can be stored on the device itself or on an external drive or network drive. the path of each shard database file is stored in the millix database. each node only has the shard database file for the shards that it is assigned to and that it is configured to support, or has supported in the past.

to run efficiently, each node should store all of the transaction data associated with payments made from, or received to, the addresses in it's active wallet. this is how the node knows it's balances and can verify it's own transactions with the desired level of trust.

problem

imagine there are 50 established shards in use on the millix network. each transaction is assigned to a randomly selected shard from that pool of 50. if node Alpha is only supporting shard one, and shard Beta sends a transaction that is assigned to shard two to node Alpha, node Alpha would have to create the shard two database file to store the transaction it received. over time, node Alpha would accumulate all 50 shard databases. even if this didn't occur naturally, bad actors could flood the network with transactions to nodes and cause it to occur unnaturally. this may be harmless, but it is undesirable and would make shard backups more burdensome.

solution

all new transactions that are propagated on the millix network are first inserted into the transaction tables contained in the millix database. once they are stored in the millix database they await one of three potential dispositions:

1. the key identifier of the transaction is associated with the active wallet in use by the node and is assigned to a shard that the node is not configured to support. the transaction remains in shard zero until the node includes it in an audit point, spend its or it expires.

2. the key identifier of the transaction is not associated with the active wallet in use by the node. it will remain in shard zero in the millix database where it can be used to verify transactions, until it becomes eligible to be pruned from the database based on the node's pruning configuration.

3. regardless of whether the transaction is associated with the active wallet, the transaction is assigned to a shard that the node is configured to support. a shard database file already exists on the node to store transactions assigned to the shard. the transaction is copied to the appropriate shard database to be permanently stored and is removed from the millix database.

in this way, the transaction tables within the millix database acting as shard zero function identically to other shard databases in regards to storing and fetching records for verification and reporting. the only difference is the speed and selectivity that records in shard zero are pruned from the tables. the size of shard zero is defined by the pruning policy and the number of transactions related to the key identifier of the node's active wallet. considering the protocol rule that expires transaction's unspents, there may only be a few days worth of millix transactions stored in shard zero, plus all it's own transactions. this keeps the size of the millix database manageable, while allowing the node to be productive verifying transactions that rely on transactions assigned to shards that the node is not configured to store data for.