Driven by the demand for ubiquitous connectivity, the increasing interaction between in-vehicle and external devices makes it easier for attack vectors and vulnerabilities to penetrate vehicles. As a typical cyber-physical system, the communications among actuators and sensors are mostly completed through in-vehicle networks, which does not concern cybersecurity threats of its original design. To effectively prevent potential security risks, deploying an intrusion detection system is a feasible and practical solution to detect and identify abnormalities in network traffic. But for development, it must balance embedded device resources with model complexity. This study proposes an anomaly detection method based on spectral residuals and depth-separable convolutional neural networks for in-vehicle networks. Specifically, the spectral residual operation is used to remove redundancies in the signal input to highlight the abnormal points, while the lightweight convolutional block is designed to tackle the challenge of sophisticated decisions. First, we design an image builder to transform signal sequence data into matrix-like structures for easy change of the intrusion detection problem to an image classification problem. Then, we construct the lightweight convolutional network, optimized for vehicular signals, to achieve high detection performance without the unnecessary complexity of the MobileNet model architecture. Experimental results on two public datasets demonstrate that our algorithm successfully detects various attacks, while having no unacceptable resource consumption and compute pipeline congestion. More importantly, compared with other advanced anomaly detection models on an in-vehicle network dataset, the results illustrate its superiority in detecting unknown attacks.