lab/cupido
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add offline tracking pipeline for video backlog

Browse source 
The 2024 video set in all_video_info_merged.xlsx covers 63 (date, machine)
sessions — 129 video instances — that have no auto-detectable targets, so
ROI placement requires manual reference-point selection. This commit adds
the three-stage pipeline that lets a user click for an hour, then walk
away while the tracker grinds overnight:

  1. build_video_inventory.py — scan /mnt/ethoscope_data/videos/ and join
     against the xlsx, producing data/metadata/video_inventory.csv

  2. pick_targets.py — interactive matplotlib/Tk picker. User clicks
     TOP/CORNER/LEFT (the L-shape ethoscope expects); after the third
     click the 6 ROI rectangles are drawn on top of the frame so geometry
     can be verified before saving. Also supports marking a video
     'unusable' (FOV wrong) so it's permanently skipped, frame stepping
     by ±1s/±5%/midpoint, point editing in --redo mode, and a crosshair
     cursor that survives matplotlib's per-motion cursor reset.

  3. track_videos.py — headless batch tracker. Reads the JSON sidecars,
     builds 6 ROIs from the HD-mating-arena geometry, runs MultiFlyTracker
     against the merged.mp4 via MovieVirtualCamera, writes SQLite DBs to
     data/tracked/. Idempotent (skips done DBs), parallel via --jobs,
     subclasses MovieVirtualCamera so frames stay BGR (MultiFlyTracker
     calls cvtColor(BGR2GRAY) without checking channel count).

Plus auto_detect_targets.py (fallback that runs ethoscope's auto-detector
in case any videos do have visible target dots), monitor_tracking.py
(progress + ETA from data/tracked/ ground truth, --watch for live view),
and tracking_geometry.py (single source of truth for the affine math
shared by picker and tracker).

requirements-tracking.txt pins the extra deps (opencv-python, openpyxl,
gitpython, netifaces, mysql-connector-python) — these are only needed
for the tracking pipeline, not the existing analysis notebooks.

Verified end-to-end on one of the user-picked videos: ~4000 rows/ROI in
a 120s slice, fly bounding boxes in the expected 800-2000 px² band.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Giorgio Gilestro 2026-04-27 17:25:26 +01:00
parent e7e4db264d
commit e4da7691d5
11 changed files with 1296 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

26
README.md
View file

@ -46,6 +46,32 @@ The key insight: not all "trained" flies may have actually learned. The trained
**Read `docs/bimodal_hypothesis.md` for the detailed analysis plan and code sketches.**
## Offline Tracking Pipeline (added Apr 2026)
For tracking new videos that have **no auto-detectable targets**, the pipeline
is split in two stages so you can sit at the screen and click for an hour, then
let the tracker grind through overnight.
```bash
# extra deps (ethoscope src must be at /home/gg/Code/ethoscope_project/...)
pip install -r requirements-tracking.txt
# 1) build the inventory (xlsx ↔ /mnt/ethoscope_data/videos/)
python scripts/build_video_inventory.py
# 2) interactive: click TOP, CORNER, LEFT on each video (one frame per video)
python scripts/pick_targets.py # process all not-yet-picked
python scripts/pick_targets.py --redo # re-pick already-picked videos
# keys: r=reset n=skip f=jump frame q/ESC=quit ENTER=save
# 3) batch tracking (idempotent, can run in background)
python scripts/track_videos.py --jobs 4 # parallel
# output → data/tracked/*_tracking.db (SQLite, same schema as data/raw/)
```
See `tasks/todo.md` "Offline Tracking" section for the full plan, and
`data/metadata/video_inventory.csv` for the list of videos to process.
## Folder Structure
```