With the rapid development of Internet technology and communication technology, more and more computer systems and networks have been maliciously attacked by intruders. Network security has been seriously threatened to a certain extent, and network security technology has also attracted more and more attention from the public. As a security protection technology for actively monitoring network data, intrusion detection technology effectively compensates for the defects of traditional security protection technologies such as firewalls and data encryption, and has become an important research field in network security. Based on this, it is very important to design the security mechanism of the system to prevent unauthorized access to system resources and data. This paper uses a DBSCAN algorithm for anomaly detection clustering algorithm. Algorithms that can be used for massive data processing have become a research hotspot in anomaly detection. Normal behavior profiles are formed on audit records and adjusted in time as program behavior changes. Experimental results show that, compared with other algorithms, anomaly detection based on the DBSCAN algorithm can improve the detection rate of the data set, and significantly improve the accuracy of anomaly detection.