Dynamic Sharding Strategies for Distributed Transactional Systems
Author(s): Arunkumar Sambandam
Publication #: 2602006
Date of Publication: 11.12.2023
Country: United States
Pages: 1-20
Published In: Volume 9 Issue 6 December-2023
Abstract
Distributed transactional systems rely on sharding to divide data across multiple nodes in order to achieve scalability and parallel execution. Sharding allows concurrent processing of transactions and helps distribute workload across the cluster. However, conventional sharding strategies typically use static or hash based partition placement that does not adapt to runtime access patterns. As workloads evolve, related data items often become scattered across multiple shards. This fragmentation forces transactions to access several shards for a single operation, resulting in increased coordination, additional communication, and longer execution paths. When transactions span multiple shards, the system must perform extra synchronization and distributed commit processing. These steps introduce waiting time and resource contention across nodes. As concurrency grows, cross shard interactions become more frequent, increasing overhead and limiting the number of transactions that can be processed simultaneously. Although computational resources may remain available, the time spent on coordination and communication reduces overall processing efficiency. Consequently, systems experience declining throughput as cluster size and workload intensity increase. Adding more nodes does not always translate into proportional performance gains because communication overhead dominates execution time. Empirical observations indicate that static sharding frequently leads to excessive inter shard traffic and inefficient resource utilization. The inability to adapt partition placement to access locality results in repeated distributed operations that restrict transaction processing capacity. These limitations highlight the need for improved sharding strategies that can sustain high transaction rates in large scale environments. This paper addresses the problem of throughput degradation in distributed transactional systems and focuses on improving transaction processing capacity by minimizing inter shard communication and enhancing data locality during shard management.
Keywords: Telemetry, Congestion, Monitoring, Correlation, Distributed, Networks, Scalability, Utilization, Overhead, Diagnostics, Synchronization, Performance, Efficiency, Latency, Throughput.
Download/View Count: 8
Share this Article