DeltaFS: A Scalable No-Ground -Truth Filesystem For Massively -Parallel Computing

High-Performance Computing (HPC) is known for its use of massive concurrency. But it can be challenging for a parallel filesystem's control plane to utilize cores when every client process must globally synchronize and serialize its metadata mutations with those of other clients. We present DeltaFS, a new paradigm for distributed filesystem rnetadata. DeltaFS allows jobs to self-commit their munespace changes to logs, avoiding the cost of global synchronization. Followup jobs selectively merge logs produced by previous jobs as needed, a principle we term No Ground Truth which allows.for efficient data sharing. By avoiding unnecessary synchronization of ineladata operations, DeltaFS improves metadata operation throughput up to 93x leveraging parallelism on the nodes where job processes run. This speedup grows as job size increases. DeltaFS enables efficient inter-job communication, reducing overall workflow runtime by significantly improving client metadata operation latency up to 49x and resource usage up to 52x.