The Video Event Reconstruction and Analysis (VERA) System - Shooter Localization from Social Media Videos
Junwei Liang, Jay D. Aronson, Alexander Hauptmann
Carnegie Mellon University

(11/2019) Presented at CES 2020.

(11/2019) Press coverage:
, , , , , , , , ,

(09/2019) Received Best Demo award at CBMI2019.

We introduce the VERA system, enabled by established machine learning techniques and physics models, that can localize the shooter location only based on a couple of user-generated videos that capture the gunshot sound.

We estimate shooter distance for each camera and estimate shooter direction for each pair of cameras.

Demo Video
  1. For full technical details, please refer to our technical report. Please report on Github if you find any detail missing in the report.
  2. We have released all of our source code on Github. We encourage researchers and engineers to help continue improving this system.
If you find this system helpful to your project/report/research, please cite the following papers:
How A Super-Sonic Bullet Generates Two Sounds

User Guide
Step 1. Upload Your Videos
Click "Create New Collection" button and then click "Upload New Videos" to upload your videos to a video collection. Given this collection a name and then click "Create Collection" to finish this process. Note that currently you cannot add new videos after this step.
Step 2. Video Synchronization
In this step, we will synchronize the videos and put them into a global timeline. First click "Run Video Synchronization" to run automatic synchronization. After it is finished, you will see the videos in a global timeline control panel:

You can modify the global time offset of each video here and then click "save offsets" button to save the new time offsets. However, to ensure more accurate video synchronization result, we recommend users to conduct pairwise verification at the video frame-level, by clicking the "Manual Refine" button and go to the global synchronization page.
In the global synchronization page, you will see "Click each video to confirm results pairwisely" on the right. Click each video to go into the pairwise verification papge. In the verification papge, on the left it is the video you just selected, on the right is the list of all other videos. For each of the other video, click "Load Video" button to load the video on the webpage. When you click "Play Sync from Left Video", both videos will play based on the current offset. You can adjust the offset and click "Save Offset" to save the new offset. For accurate frame-level synchronization, click "Frame View" and "Get Frames" for both videos. Click "Show Current Video Frame" to jump to the frame ID that your videos stop at. Click "Prev" or "Next" on top of the frames until you find a perfect visual synchronization. Then click "Get Offset from FramePair" to get the current synchronization into the offset. Click "Save Offset". When you are satisfy with the offset, remember to click "Correct".
After pairwise synchronization, go back to the global synchronization page. Click "Refine Based on Confirmed Pairs" to run global synchronization based on your confirmed pairwise results.
Step 3. Gunshot Marking
In this step, we will first add information for each gunshot. Click "New Gunshot" and then enter the information. Then for each of the gunshot added, we will need to mark the bullet shockwave sound (if it is supersonic) and the muzzle blast sound. In the "Mark the Gunshot in Videos" section, click "Mark Gunshot" of each video. If the video is too long and you are not sure where the gunshot is, click "Run Gunshot Detection" and it will run our gunshot detection model on this video. The gunshot segment will be labeled red on the timeline. To aide the marking, we provide spectrogram and power graph for users to mark the sounds. Enter the start and end time of the video, click "Show Spectrogram" and the graph will be generated:

Click on the graph to mark the time of the sounds. Click "Save" when you are done. You will need to mark each gunshot for each of the videos.
Step 4. Video Localization per Gunshot
In this final step, we will visualize possible shooter locations for each gunshot. You will need to mark each camera location on the map at the time of the gunshot.

Release Log