Unmanned aerial vehicles (UAVs) have brought many practical benefits during the last decades. Moreover, as technology advances, UAVs become more optimal in size and range. However, the threat posed by these devices is also increasing if people misuse them for illegal activities (such as terrorism, drug trafficking, etc.), which poses a high risk to security for different organizations and governments. Hence, detection and monitoring of drones are crucial to prevent security breaches. However, the small size and similarity to wild birds in the complex background of drones pose a significant challenge. This paper addresses the detection of small drones in real surveillance videos using standard deep learning-based object recognition methods. Our method approaches the drone detection problem by training the YOLOv4 model with modifications in the network structure, training strategy, and pre-anchor boxes for better small object detection. We also integrate the Seq-NMS post-processing phase to increase detection reliability and reduce false alarms. The experimental results show that our approach can perform better than the previous methods.