Research and Improvement of Parallelization of FP - Growth Algorithm Based on Spark

Aiming at the problem of load imbalancing when the existing FP-Growth algorithm groups the data set, an optimization algorithm DGFP-Growth for dynamic dividing data set is proposed First, before the dynamic grouping, the load estimation method is proposed The position of each item in the frequent item list and the size of its support are used to determine the load weight. Then under the grouping strategy, these items are assigned to the corresponding group according to their load weights to ensure that the load is balanced within each group. Experiments show that the optimization algorithm proposed in this paper can effectively solve the problem of parallel load imbalancing, and improve the overall efficiency of the cluster between 5% and 15%.