Coal pillar is a crucial element formed during underground coal mining, playing a significant role in ensuring coal mining operations are conducted safely. Historically, research on coal pillars has primarily focused on regular-shaped pillars, with limited attention given to special-shaped pillars. Therefore, this paper aims to investigate the special-shaped coal pillar formed after the excavation of the newly constructed 3307 headgate in Dayang Coal Mine, China. The primary object of this paper is to elucidate the mechanism behind the instability failure of special-shaped coal pillars and verify the influence of various geological conditions on their stability. To accomplish these goals, numerical simulation techniques were employed to analyze the stress distribution patterns and plastic zone failure characteristics of special-shaped coal pillar. The investigation revealed the presence of stress concentration phenomena within special-shaped coal pillar, where the stress peak values were directly correlated with the width of the pillar. Moreover, it was observed that as the width of coal pillar decreased, the stress curve gradually transitioned from a bimodal shape to a unimodal shape. Additionally, the ratio of the plastic zone within coal pillar increased as the width decreased, with pronounced tensile failures occurring at sharp corners. Building upon theoretical analysis and numerical simulation results, the interior of special-shaped coal pillar was classified into four distinct zones: damaged zone, progressive zone, stable zone, and special zone. Based on this zoning approach, a stability control technology was developed, employing a combination of "pier pillar support + flexible formwork concrete + grouting reinforcement + high-strength cable and bolt + U-steel." Subsequently, an industrial experiment was conducted to validate the efficacy of this control technology. Field monitoring results demonstrated that the newly excavated roadway exhibited minimal deformation and damage, thereby satisfying the requirements for safe and efficient mine production over the anticipated service life. The failure characteristics and bearing mechanism of special-shaped coal pillars were analyzed, and a mechanical model was established, allowing exploration of the influence of different geological conditions on pillar stability.The stress evolution and plastic zone development characteristics of special-shaped coal pillar during the whole cycle were studied.A zoning criterion for special-shaped coal pillar was proposed, along with corresponding control measures, which were successfully applied to the site, effectively controlled the deformation of roadway.