Currently, the condition of a sewer pipe is assessed by an inspector monitoring live video supplied from a remotely controlled closed-circuit television (CCTV) camera. As the inspector guides the video camera through the pipe, she/he will look for different types of defects/anomalies, including structural, operational, construction features, and miscellaneous defects. Based on the National Association for Sewer Service Companies (NASSCO) standard, there are 224 different defects/sub-defects which can occur within a given inspection. Given the significant number of defects/sub-defects, assigning defect codes and their corresponding severities is prone to subjectivity and hence may impact the overall accuracy of the inspection interpretations. Inaccurate interpretations could mislead decision makers while selecting the proper intervention actions to sustain critical sewers. In an effort to speed up the overall inspection process and enhance the interpretation accuracy, this research aims at utilizing artificial intelligence and computer vision tools to detect and classify defects in accordance with existing standards; this research is a continuation of AECOM X Google Hack-a-thon's proof of concept application. The smart and automated tool relies on enormous data obtained from the City of Toronto, multiyear program to build a reliable database. The initial results of the prototype showed promising detection and classification capabilities of defects and sub-defects including circumferential crack (CC), longitudinal fracture (FL), encrustation attached deposits (DAE), tab break-in (TB), tab break-in intruding (TBI), and obstruction intruding (OBI). The average accuracy achieved for the six anomalies was 85% where the maximum and minimum accuracy levels were 94% and 75%. This tool, once completed, will elevate the sewer inspection process by speeding up the inspection validation, enhancing accuracy, and maintaining consistency, thereby assisting in making proper decisions when selecting the required intervention actions.