overview

the objective is to approach infinite data scale and data reliability as a network while requiring very small data storage obligations for a device. the total amount of storage available on the network to store transactions, and it's related data, must exceed what a single device is capable of providing. this is accomplished by spreading data across fragmented copies (shards) of the entire network data set.

many shards will be available, and depending on the amount of network data and the number of nodes on the network, additional shards can be added and recognized by the network at any time. there are two types of shards:

• protocol shards: protocol shards will contain groups of records that share the same transaction_id value. nodes will be required to host at least one protocol shard, which will be assigned during node installation. the node will announce it's protocol shards to other nodes so they know who to ask for data from a specific shard. nodes can choose to host additional protocol shards for economic opportunity. when data is being synched to the node, it is only interested in records associated with shard.shard_id where status = 1.

• custom shards: custom shards allow any node to create their own shard to contain transaction data or transaction_output_attribute data. this extends the capability of millix to be able to store large pieces of data with a transaction or to act as an economic incentive to store data redundantly across multiple nodes. for example, an organization can create and host custom shards to store their transactions on known nodes operating at performance levels that meet their specific operational goals.

as it is not obligatory for nodes to store custom shards, it is up to the custom shard creator to provide or attract and incentivize other nodes to store the shard. this can be accomplished with the node_attribute table. any node that connects to a node that is hosting a custom shard can see opportunities to participate in. this enables a decentralized data storage market economy.

creating a shard

any node can create a new protocol shard or custom shard by inserting to their local shard and shard_attribute table. no permission or authority is required to create new shards. since the shard tables are not synced to nodes, this by itself doesn't inform the network about the shard. it's not until the shard is selected to be supported by a node (initially this must be the node that created the shard as it is the only node with visibility to the node) that the existence of the shard becomes visible to other nodes.

to avoid shard_id conflict, the shard_id is a payload hash of the shard definition including a signature (shard_key) from the node originating the shard. although someone operating two nodes could use the same private key to create the same shard definition at the same time on two devices, the impact would be meaningless. there would be two shard records of equal definition and shard_id but a unique key index on table shard.shard_id would prevent duplicate records, and the sequence of the discarded record would not matter.

announcing a shard

the catalog of available shards and their attributes can propagate through the network via the node_attribute table. every time a node connects to another node, they view the node_attribute records for that node and can store it in their local node_attribute table.

assigning a shard to a node

every node is assigned to a protocol shard and has the obligation of storing the raw transaction details associated to that shard_id. millix aspires to have redundant copies of every transaction since genesis. a baseline protocol shard disk size needs to be established (20 gb?), as well as the minimum number of nodes that is acceptable to be assigned to each protocol shard (20 public nodes with different IP?). using the baseline assumptions and the state of the network, users can determine when new protocol shards are needed.

when a new shard is created it must be supported by a critical mass of public nodes (10?) before it is eligible to be the assigned shard for newly installed nodes. perhaps new node are not assigned a protocol shard until they have a history of node connections and can consider the shards available and eligible to be assigned to itself.

deactivating a shard

when a shard reaches its stated maximum number of records or disk space it is set to status = 0 by each node supporting it. the state will be propagated via node_attribute and, once propagated fully, no nodes will consider the deactivated shard when randomly selecting a shard to associate to a transaction or as an assigned shard on a new node installation.

shard tables

shard

shard_id (hash of node_id_origin, shard_date, shard_type, node_signature)
shard_name
shard_type
schema_name
schema_path
is_required
record_count
disk_size
updated
node_id_origin
shard_date
node_signature (signed with private key)
status (1=active, 0=inactive or static
create_date

shard_attribute_type

attribute_type_id
attribute_type
status
create_date

example attribute type

fee_ask_request_byte
fee_offer_byte_minute
fee_offer_request_byte
record_count_max
disk_size_max
audit_point_*
consensus_*

shard_attribute

shard_attribute_id
shard_id
attribute_type_id
value
status
create_date

transaction tables

shard_id is denormalized across all the shard eligible tables to allow record navigation in any direction, to any related table, from any selected row. add the following columns:

audit_point.shard_id
audit_verification.shard_id
transaction.shard_id
transaction_input.shard_id
transaction_output.shard_id
transaction_output_attribute.shard_id
transaction_parent.shard_id
transaction_signature.shard_id

when a transaction_id is inserted to a shard, all the associated records must also be inserted and present on the same shard.