Abstract: Detecting anomalies in videos is a complex problem with a myriad of applications in video surveillance. However, large and complex datasets that are representative of real-world deployment of surveillance cameras are unavailable. Anomalies in surveillance videos are not well defined and the standard and existing metrics for evaluation do not quantify the performance of algorithms accurately. We provide a large scale dataset, A Day on Campus (ADOC), with 25 event types, spanning over 721 instances and occurring over a period of 24 hours. This is the largest dataset with localized bounding box annotations that is available to perform anomaly detection. We design a novel metric to evaluate the performance of methods and we perform an evaluation of the state-of-the-art methods to ascertain their readiness to transition into real-world surveillance scenarios.