Distributed Clustering Approach by Apache Pyspark Based on SEER for Clinical Data

Data clustering is a thoroughly studied data mining issue. As the amount of information being analyzed grows exponentially, there are several problems with clustering diagnostic large datasets like the monitoring, microbiology, and end results (SEER) carcinoma feature sets. These traditional clustering methods are severely constrained in terms of speed, productivity, and adaptability. This paper summarizes the most modern distributed clustering algorithms, organized according to the computing platforms used to process vast volumes of data. The purpose of this work was to offer an optimized distributed clustering strategy for reducing the algorithm's total execution time. We obtained, preprocessed, and analyzed clinical SEER data on liver cancer, respiratory cancer, human immunodeficiency virus (HIV)-related lymphoma, and lung cancer for large-scale data clustering analysis. Three major contributions and their effects were covered in this paper: To begin, three current Pyspark distributed clustering algorithms were evaluated on SEER clinical data using a simulated New York cancer dataset. Second, systemic inflammatory response syndrome (SIRS) model inference was done and described using three SEER cancer datasets. Third, employing lung cancer data, we suggested an optimized distributed bisecting k-means method. We have shown the outcomes of our suggested optimized distributed clustering technique, demonstrating the performance enhancement.