Test Network Lifecycle and Formation

This document outlines the technical architecture and lifecycle of the Shardy Test Network, detailing how nodes discover each other, form a consensus lattice, and maintain decentralized state.

1. Network Topology & Stack

The Shardy Network is built on a hybrid communication model designed for high-throughput task distribution and low-latency state synchronization.

• Orchestrator-Worker (WebSocket): Primary control plane for task dispatch, ACK/progress, and results.
• P2P Backbone (libp2p): Gossip for presence, transactions, and block announcements; stream-based block/snapshot sync.
• Node-to-Node (WebRTC): Optional data path negotiated via orchestrator signaling.

2. The Formation Process (Node Onboarding)

A node’s entry into the Shardy Lattice follows a strict “Proof of Performance” sequence:

A. Hardware Profiling (Benchmarking)

Before joining, every node must run a local benchmark (runBenchmarkCall).

• GPU Profiling: Measures GFLOPS and VRAM availability via WebGPU.
• RAM/CPU Check: Estimates stable allocation limits.

The result is a HardwareProfile that determines the node’s eligibility for specific task tiers.

B. Bootstrapping

The node contacts the Orchestrator’s /api/libp2p/bootstrap endpoint to receive multi-addresses. It then initializes a libp2p host, subscribes to gossip topics, and starts presence publishing.

C. The Lattice Join

The node establishes a WebSocket connection at /ws/worker-join, completes the hello challenge handshake, then sends profile_v2. The Orchestrator registers the node in-memory and persists it in the state store.

3. Consensus and State Management

Shardy operates as a “Shard-State” machine where every operation is backed by consensus.

A. The Core Components

• Mempool: Collects transactions (task_create, task_ack, task_progress, task_result, task_verified, stake_set, worker_presence).
• Consensus Engine: Slot-based block proposal/vote/commit.
• State Machine: Applies blocks and updates tasks, workers, balances, and events.

B. Decentralized Verification

Results generated by nodes are verified via:

1. Redundancy: Multiple nodes execute the same shard.
2. Consensus Log: Verification results are submitted as transactions and finalized in blocks.
3. Watchdog: Detects ACK/execution timeouts and triggers retry/reassign/dead-letter actions.

C. Consensus Sync (Current)

The network supports fast synchronization without full history:

• Block announce topic: shardy.block.announce.v1.
• Block sync stream: /shardy/blocks/sync/1.0.0.
• Snapshot sync stream: /shardy/state/snapshot/1.0.0.
• HTTP fallback: /api/consensus/status and /api/consensus/blocks.

4. Network Mesh Campaigns

Beyond individual tasks, the Shardy Network supports Network Mesh tests to evaluate the P2P throughput of the lattice.

1. Start Campaign: POST /api/network-mesh with fileCount and fileSizeMb.
2. Control Signal: Orchestrator sends network_test_control_start to accepted nodes.
3. WebRTC Swarm: Nodes negotiate direct links (signaled through orchestrator).
4. Telemetry Aggregation: Nodes send network_test_heartbeat; reports are available via /api/network-mesh/:id/report.

5. Persistence and Reliability

• Orchestrator Persistence: Uses RocksDB (default) or SQLite (dev-only) to persist state and block history.
• Worker Persistence: Utilizes OPFS (Origin Private File System) for “Crash-Safe” task processing. If a browser process dies, the node can resume from the last saved state upon restart.
• State Rehydration: Upon Orchestrator restart, it reloads all active tasks and worker assignments from the database, ensuring the network can recover from a total infrastructure reboot without data loss.