Evaluation of a Cyber Risk Assessment Approach for Cyber-Physical Systems: Maritime- and Energy-Use Cases

In various domains such as energy, manufacturing, and maritime, cyber-physical systems (CPSs) have seen increased interest. Both academia and industry have focused on the cybersecurity aspects of such systems. The assessment of cyber risks in a CPS is a popular research area with many existing approaches that aim to suggest relevant methods and practices. However, few works have addressed the extensive and objective evaluation of the proposed approaches. In this paper, a standard-aligned evaluation methodology is presented and empirically conducted to evaluate a newly proposed cyber risk assessment approach for CPSs. The approach, which is called FMECA-ATT&CK is based on failure mode, effects and criticality analysis (FMECA) risk assessment process and enriched with the semantics and encoded knowledge in the Adversarial Tactics, Techniques, and Common Knowledge framework (ATT&CK). Several experts were involved in conducting two risk assessment processes, FMECA-ATT&CK and Bow-Tie, against two use cases in different application domains, particularly an autonomous passenger ship (APS) as a maritime-use case and a digital substation as an energy-use case. This allows for the evaluation of the approach based on a group of characteristics, namely, applicability, feasibility, accuracy, comprehensiveness, adaptability, scalability, and usability. The results highlight the positive utility of FMECA-ATT&CK in model-based, design-level, and component-level cyber risk assessment of CPSs with several identified directions for improvements. Moreover, the standard-aligned evaluation method and the evaluation characteristics have been demonstrated as enablers for the thorough evaluation of cyber risk assessment methods.