Add offline tracking pipeline for video backlog

The 2024 video set in all_video_info_merged.xlsx covers 63 (date, machine)
sessions — 129 video instances — that have no auto-detectable targets, so
ROI placement requires manual reference-point selection. This commit adds
the three-stage pipeline that lets a user click for an hour, then walk
away while the tracker grinds overnight:

  1. build_video_inventory.py — scan /mnt/ethoscope_data/videos/ and join
     against the xlsx, producing data/metadata/video_inventory.csv

  2. pick_targets.py — interactive matplotlib/Tk picker. User clicks
     TOP/CORNER/LEFT (the L-shape ethoscope expects); after the third
     click the 6 ROI rectangles are drawn on top of the frame so geometry
     can be verified before saving. Also supports marking a video
     'unusable' (FOV wrong) so it's permanently skipped, frame stepping
     by ±1s/±5%/midpoint, point editing in --redo mode, and a crosshair
     cursor that survives matplotlib's per-motion cursor reset.

  3. track_videos.py — headless batch tracker. Reads the JSON sidecars,
     builds 6 ROIs from the HD-mating-arena geometry, runs MultiFlyTracker
     against the merged.mp4 via MovieVirtualCamera, writes SQLite DBs to
     data/tracked/. Idempotent (skips done DBs), parallel via --jobs,
     subclasses MovieVirtualCamera so frames stay BGR (MultiFlyTracker
     calls cvtColor(BGR2GRAY) without checking channel count).

Plus auto_detect_targets.py (fallback that runs ethoscope's auto-detector
in case any videos do have visible target dots), monitor_tracking.py
(progress + ETA from data/tracked/ ground truth, --watch for live view),
and tracking_geometry.py (single source of truth for the affine math
shared by picker and tracker).

requirements-tracking.txt pins the extra deps (opencv-python, openpyxl,
gitpython, netifaces, mysql-connector-python) — these are only needed
for the tracking pipeline, not the existing analysis notebooks.

Verified end-to-end on one of the user-picked videos: ~4000 rows/ROI in
a 120s slice, fly bounding boxes in the expected 800-2000 px² band.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

tasks/todo.md

@@ -51,6 +51,68 @@ See `docs/bimodal_hypothesis.md` for detailed methodology.
 - [ ] Consider converting pixel distances to physical units (need calibration)
 - [ ] The second notebook (`flies_analysis.ipynb`) re-runs from DB extraction - consider deprecating
 
+## Phase: Offline Tracking of 2024 Video Backlog (added 2026-04-27)
+
+### Recap
+
+Tracked so far (5 sessions, all from 2025-07-15, machines 076/145/268). The DBs in
+`data/raw/` use tracker `ConstrainedMultiFlyTracker` and template
+`HD_Mating_Arena_6_ROIS.json` (2 flies × 6 ROIs per video).
+
+The metadata file `../all_video_info_merged.xlsx` indexes a different set of
+experiments: 7 dates from 2024-09-17 → 2024-10-21, 16 ethoscope machines,
+63 unique (date, machine) sessions = 484 ROI-rows. **None of the already-tracked
+sessions are in this xlsx — these are fresh recordings to track.**
+
+Inventory: see `data/metadata/video_inventory.csv` (built by
+`scripts/build_video_inventory.py`).
+- 1163 video sessions on disk under `/mnt/ethoscope_data/videos/`
+- 63/63 xlsx (date, machine) sessions have video on disk
+- 129 video instances need tracking (some (date, machine) have 2-4 recordings/day)
+
+### Plan
+
+The HD-mating-arena videos have no auto-detectable targets — the user must
+manually click 3 reference points (L-shape: top, corner, left) per video. Once
+all targets are picked, tracking can run in the background.
+
+- [x] **Step 1 — Inventory**: `scripts/build_video_inventory.py` →
+      `data/metadata/video_inventory.csv`. 63 (date,machine) sessions match
+      the xlsx, all videos found, 129 video instances need tracking.
+- [x] **Step 2 — Manual target picker**: `scripts/pick_targets.py`. Loops over
+      videos with `in_xlsx & ~already_tracked & no JSON yet`; per video, shows
+      a representative frame, captures 3 clicks (top, corner, left), saves
+      `data/targets/<video_basename>.json`. Skips videos already done.
+- [x] **Step 3 — Background tracker**: `scripts/track_videos.py`. Reads target
+      JSONs, builds 6 ROIs from the HD-mating-arena geometry, runs
+      `MovieVirtualCamera` + `MultiFlyTracker` + `SQLiteResultWriter`, writes
+      `data/tracked/<basename>_tracking.db`. Idempotent. Smoke-tested
+      end-to-end: 90s of video → ~3000 rows/ROI, areas in 800-2000 band.
+- [x] **Step 4 — Tracking deps**: `requirements-tracking.txt`.
+
+### Still TODO
+- [ ] User to run `pick_targets.py` (interactive — needs DISPLAY) on the 129
+      pending videos.
+- [ ] Run `track_videos.py --jobs 4` against the resulting JSONs.
+- [ ] (Optional) `auto_detect_targets.py` exists as a fallback for videos that
+      DO have visible targets (saves clicks). Confirmed not useful on the
+      2025-07-15 batch — these arenas don't have black target dots — but worth
+      trying on 2024 batches before falling back to manual.
+- [ ] Decide what to do with the 4 (date, machine) sessions that have 3-4
+      recordings/day instead of 2 (e.g. ETHOSCOPE_086 on 2024-09-17 has 4).
+      One of them is at lower resolution (1280x960) — likely an aborted take.
+
+### Open questions / risks
+
+- Some (date, machine) combos have 3-4 recordings (e.g. ETHOSCOPE_086 on
+  2024-09-17). Need to figure out which is the real "test" video vs aborted
+  takes — possibly use video duration or filename pattern.
+- One mismatched-resolution file: `1280x960@25fps-20q` instead of
+  `1920x1088@25fps-28q` — flag for inspection.
+- The original `ConstrainedMultiFlyTracker` is no longer in the ethoscope repo;
+  `MultiFlyTracker` is its likely successor. Validate output schema matches
+  what the existing analysis pipeline expects (`load_roi_data.py`, etc.).
+
 ## Discovered During Work
 
 (Add new items here as they come up during analysis)