Onsite video mining for construction hazards identification with visual relationships

Widely-used video monitoring systems provide a large corpus of unstructured image data on construction sites. Although previous developed vision-based approaches can be used for hazards recognition in terms of detecting dangerous objects or unsafe operations, such detection capacity is often limited due to lack of semantic representation of visual relationships between/among the components or crews in the workplace. Accordingly, the formal representation of textural criteria for checking improper relationships should also be improved. In this regard, an Automated Hazards Identification System (ARTS) is developed to evaluate the operation descriptions generated from site videos against the safety guidelines extracted from the textual documents with the assistance of the ontology of construction safety. In particular, visual relationships are modeled as a connector between site components/operators. Moreover, both visual descriptions of site operations and semantic representations of safety guidelines are coded in the three-tuple format and then automatically converted into Horn clauses for reasoning out the potential risks. A preliminary implementation of the system was tested on two separate onsite video clips. The results showed that two types of crucial hazards, i.e., failure to wear a helmet and walking beneath the cane, were successfully identified with three rules from Safety Handbook for Construction Site Workers. In addition, the high-performance results of Recall@50 and Recall@100 demonstrated that the proposed visual relationship detection method is promising in enriching the semantic representation of operation facts extracted from site videos, which may lead to better automation in the detection of construction hazards.