Railway embankments in seismically active areas are prone to earthquake-induced damage. In many instances globally, such damage has led to substantial economic losses. Serviceability assessment of these embankments is pivotal in ascertaining better performance during earthquakes. This work presents a physics-based approach to assess the serviceability of railway embankments subjected to strong ground motions. A series of nonlinear dynamic analyses are performed to evaluate the failure mechanism, progression of the failure plane, accumulation of plastic strain, and deformations of a railway embankment using the framework of smoothed particle hydrodynamics (SPH). The embankment and its underlying foundation are treated as a layered domain, and peak acceleration within each layer is determined through the site-specific nonlinear ground response analysis. The vulnerability assessment of the embankment is carried out considering the vertical displacement of the crest, accumulation of plastic strain, and post-failure scenario under site-specific ground motion characteristics. The vulnerability of the embankment is further quantified through fragility analysis by considering various damage levels. Fragility analysis is carried out using incremental dynamic analysis (IDA) against peak ground acceleration (PGA) of input ground motions as the key hazard indicator. The robustness of the developed vulnerability evaluation framework is also scrutinized through a sequence of stochastic analyses, considering the variability in ground conditions to enhance engineering assessment. The embankment is seen to experience a maximum vertical deformation of 0.05 m at the crest when initial signs of plastic strain development are observed, with deformation increasing to around 0.1 m for moderate damage levels and reaching up to 0.2 m at the point of slope failure. Fragility curves reveal that the right edge of the embankment reaches the first damage level at a PGA of approximately 0.12 g, followed by higher damage levels at PGA's as high as 0.8 g, for a 100% probability of extensive damage. Stochastic analysis shows that the probability of maximum vertical displacement exceeding deterministic values is about 78.47%, with maximum deviations of 2.599 m. For plastic strain, the probability of exceeding deterministic values is 78.49%, with maximum deviations of 13.08. These findings underscore the importance of considering site-specific conditions and the variability of soil properties in seismic assessments to ensure accurate and reliable serviceability evaluations of railway embankments.