Recently, widespread use of digital speech communication has spawned a multitude of Voice over IP (VoIP) applications. These applications require the ability to identify speakers in real time. One of the challenges in accurate speaker recognition is the inability to detect anomalies in network traffic generated by attacks on VoIP applications. This paper presents L 2E, an innovative approach to detect anomalies in network traffic for accurate speaker recognition. The L 2E method is capable of online speaker recognition from live packet streams of voice packets by performing fast classification over a defined subset of the features available in each voice packet. The experimental results show that L 2E is highly scalable and accurate in detecting a wide range of anomalies in network traffic.