From e4da7691d5d83c8161986efef720eb25b6af298d Mon Sep 17 00:00:00 2001 From: Giorgio Date: Mon, 27 Apr 2026 17:25:26 +0100 Subject: [PATCH] Add offline tracking pipeline for video backlog MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2024 video set in all_video_info_merged.xlsx covers 63 (date, machine) sessions — 129 video instances — that have no auto-detectable targets, so ROI placement requires manual reference-point selection. This commit adds the three-stage pipeline that lets a user click for an hour, then walk away while the tracker grinds overnight: 1. build_video_inventory.py — scan /mnt/ethoscope_data/videos/ and join against the xlsx, producing data/metadata/video_inventory.csv 2. pick_targets.py — interactive matplotlib/Tk picker. User clicks TOP/CORNER/LEFT (the L-shape ethoscope expects); after the third click the 6 ROI rectangles are drawn on top of the frame so geometry can be verified before saving. Also supports marking a video 'unusable' (FOV wrong) so it's permanently skipped, frame stepping by ±1s/±5%/midpoint, point editing in --redo mode, and a crosshair cursor that survives matplotlib's per-motion cursor reset. 3. track_videos.py — headless batch tracker. Reads the JSON sidecars, builds 6 ROIs from the HD-mating-arena geometry, runs MultiFlyTracker against the merged.mp4 via MovieVirtualCamera, writes SQLite DBs to data/tracked/. Idempotent (skips done DBs), parallel via --jobs, subclasses MovieVirtualCamera so frames stay BGR (MultiFlyTracker calls cvtColor(BGR2GRAY) without checking channel count). Plus auto_detect_targets.py (fallback that runs ethoscope's auto-detector in case any videos do have visible target dots), monitor_tracking.py (progress + ETA from data/tracked/ ground truth, --watch for live view), and tracking_geometry.py (single source of truth for the affine math shared by picker and tracker). requirements-tracking.txt pins the extra deps (opencv-python, openpyxl, gitpython, netifaces, mysql-connector-python) — these are only needed for the tracking pipeline, not the existing analysis notebooks. Verified end-to-end on one of the user-picked videos: ~4000 rows/ROI in a 120s slice, fly bounding boxes in the expected 800-2000 px² band. Co-Authored-By: Claude Opus 4.7 --- .gitignore | 9 + README.md | 26 ++ requirements-tracking.txt | 11 + scripts/auto_detect_targets.py | 119 ++++++++ scripts/build_video_inventory.py | 150 ++++++++++ scripts/config.py | 8 + scripts/monitor_tracking.py | 155 ++++++++++ scripts/pick_targets.py | 467 +++++++++++++++++++++++++++++++ scripts/track_videos.py | 218 +++++++++++++++ scripts/tracking_geometry.py | 71 +++++ tasks/todo.md | 62 ++++ 11 files changed, 1296 insertions(+) create mode 100644 requirements-tracking.txt create mode 100644 scripts/auto_detect_targets.py create mode 100644 scripts/build_video_inventory.py create mode 100644 scripts/monitor_tracking.py create mode 100644 scripts/pick_targets.py create mode 100644 scripts/track_videos.py create mode 100644 scripts/tracking_geometry.py diff --git a/.gitignore b/.gitignore index 50e96cf..02d5434 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,15 @@ data/raw/*.db data/processed/*.csv +# Offline-tracking outputs (reproducible from videos + target JSONs) +data/tracked/*.db +data/tracked/*.db-wal +data/tracked/*.db-shm +data/tracked/*.db-journal +data/targets/*.json +data/metadata/video_inventory.csv +data/logs/*.log + # Generated figures (reproducible from scripts) figures/*.png diff --git a/README.md b/README.md index bf88c6f..9d9ff17 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,32 @@ The key insight: not all "trained" flies may have actually learned. The trained **Read `docs/bimodal_hypothesis.md` for the detailed analysis plan and code sketches.** +## Offline Tracking Pipeline (added Apr 2026) + +For tracking new videos that have **no auto-detectable targets**, the pipeline +is split in two stages so you can sit at the screen and click for an hour, then +let the tracker grind through overnight. + +```bash +# extra deps (ethoscope src must be at /home/gg/Code/ethoscope_project/...) +pip install -r requirements-tracking.txt + +# 1) build the inventory (xlsx ↔ /mnt/ethoscope_data/videos/) +python scripts/build_video_inventory.py + +# 2) interactive: click TOP, CORNER, LEFT on each video (one frame per video) +python scripts/pick_targets.py # process all not-yet-picked +python scripts/pick_targets.py --redo # re-pick already-picked videos +# keys: r=reset n=skip f=jump frame q/ESC=quit ENTER=save + +# 3) batch tracking (idempotent, can run in background) +python scripts/track_videos.py --jobs 4 # parallel +# output → data/tracked/*_tracking.db (SQLite, same schema as data/raw/) +``` + +See `tasks/todo.md` "Offline Tracking" section for the full plan, and +`data/metadata/video_inventory.csv` for the list of videos to process. + ## Folder Structure ``` diff --git a/requirements-tracking.txt b/requirements-tracking.txt new file mode 100644 index 0000000..b52aad2 --- /dev/null +++ b/requirements-tracking.txt @@ -0,0 +1,11 @@ +# Extra dependencies needed only for the offline-tracking pipeline +# (build_video_inventory.py, pick_targets.py, auto_detect_targets.py, +# track_videos.py). Not needed for the existing analysis notebooks. +# +# install with: pip install -r requirements-tracking.txt +opencv-python>=4.8 +openpyxl>=3.1 +gitpython>=3.1 +netifaces>=0.11 +mysql-connector-python>=8.0 +pyserial>=3.5 diff --git a/scripts/auto_detect_targets.py b/scripts/auto_detect_targets.py new file mode 100644 index 0000000..077ac41 --- /dev/null +++ b/scripts/auto_detect_targets.py @@ -0,0 +1,119 @@ +"""Try auto-detection of L-shape targets on each video and save JSON sidecars. + +Useful for: +- videos that DO have visible black-circle targets (saves manual clicks); +- as a smoke test of the whole pipeline before running the picker. + +Failure is silent — videos that fail auto-detection are simply not written +to disk, leaving them for the manual `pick_targets.py` tool. + +Output JSON has the same shape as the manual picker's so `track_videos.py` +can consume either. +""" + +from __future__ import annotations + +import argparse +import datetime as dt +import json +import logging +import sys +from pathlib import Path + +import cv2 +import numpy as np +import pandas as pd + +# ethoscope source tree +sys.path.insert(0, "/home/gg/Code/ethoscope_project/ethoscope/src/ethoscope") + +from config import INVENTORY_CSV, TARGETS_DIR # noqa: E402 + +from ethoscope.roi_builders.target_roi_builder import TargetGridROIBuilder # noqa: E402 + + +def detect_one(video_path: Path, frame_idx: int) -> tuple[list[list[int]], int] | None: + """Run ethoscope target detection on one frame; return (points, frame_idx) or None.""" + cap = cv2.VideoCapture(str(video_path)) + if not cap.isOpened(): + return None + n = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) + if n > 0 and frame_idx >= n: + frame_idx = max(0, n - 1) + cap.set(cv2.CAP_PROP_POS_FRAMES, frame_idx) + ok, frame = cap.read() + cap.release() + if not ok or frame is None: + return None + + # The detector expects a single-channel image (grey) like ethoscope cameras produce. + if frame.ndim == 3: + gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) + else: + gray = frame + + # We don't actually need a fully-configured grid here — _find_target_coordinates + # alone gives us the 3 reference points. + builder = TargetGridROIBuilder(n_rows=2, n_cols=3) + try: + ref = builder._find_target_coordinates(gray) + except Exception as e: + logging.debug(f"detection failed for {video_path.name}: {e}") + return None + if ref is None: + return None + return [[int(p[0]), int(p[1])] for p in ref], frame_idx + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--frame", type=int, default=125) + parser.add_argument("--limit", type=int, default=None) + parser.add_argument("--video", type=str, default=None, + help="run on a single video path (skips inventory)") + parser.add_argument("--overwrite", action="store_true", + help="overwrite existing JSON sidecars") + args = parser.parse_args() + + TARGETS_DIR.mkdir(parents=True, exist_ok=True) + + if args.video: + videos = [Path(args.video)] + else: + if not INVENTORY_CSV.exists(): + sys.exit("Inventory missing — run build_video_inventory.py first.") + inv = pd.read_csv(INVENTORY_CSV) + todo = inv[inv["in_xlsx"] & ~inv["already_tracked"]] + videos = [Path(p) for p in todo["mp4_path"].tolist()] + if args.limit: + videos = videos[: args.limit] + + n_ok = n_fail = n_skip = 0 + for v in videos: + out = TARGETS_DIR / f"{v.stem}.json" + if out.exists() and not args.overwrite: + n_skip += 1 + continue + result = detect_one(v, args.frame) + if result is None: + n_fail += 1 + print(f" fail: {v.name}") + continue + points, used_frame = result + out.write_text(json.dumps({ + "video_path": str(v), + "frame_index": int(used_frame), + "reference_points": points, + "order": ["top", "corner", "left"], + "picked_at": dt.datetime.now().isoformat(timespec="seconds"), + "method": "auto", + }, indent=2)) + n_ok += 1 + print(f" ok: {v.name} → {points}") + + print(f"\nDone. ok={n_ok} fail={n_fail} skipped(existing)={n_skip}") + + +if __name__ == "__main__": + logging.basicConfig(level=logging.WARNING, format="%(levelname)s %(message)s") + main() diff --git a/scripts/build_video_inventory.py b/scripts/build_video_inventory.py new file mode 100644 index 0000000..3c083e7 --- /dev/null +++ b/scripts/build_video_inventory.py @@ -0,0 +1,150 @@ +"""Build an inventory of videos available on disk and join with the metadata xlsx. + +Scans /mnt/ethoscope_data/videos////*.mp4 +and produces a CSV mapping each (date, machine_name) row in +all_video_info_merged.xlsx to the corresponding merged.mp4 path on disk. + +Output: data/metadata/video_inventory.csv with columns: + machine_uuid, machine_name, session_date, session_time, mp4_path, + in_xlsx (bool), already_tracked (bool) +""" + +from __future__ import annotations + +import re +from pathlib import Path + +import pandas as pd + +from config import DATA_RAW, INVENTORY_CSV, VIDEO_INFO_XLSX, VIDEOS_ROOT + +SESSION_RE = re.compile(r"^(\d{4}-\d{2}-\d{2})_(\d{2}-\d{2}-\d{2})$") + + +def scan_videos(videos_root: Path) -> pd.DataFrame: + """Walk videos_root and return one row per merged.mp4 found. + + Args: + videos_root: Root directory containing ///. + + Returns: + DataFrame with columns: machine_uuid, machine_name, session_date, + session_time, session_datetime, mp4_path. + """ + rows = [] + for uuid_dir in sorted(videos_root.iterdir()): + if not uuid_dir.is_dir(): + continue + for machine_dir in uuid_dir.iterdir(): + if not machine_dir.is_dir() or not machine_dir.name.startswith("ETHOSCOPE_"): + continue + for session_dir in machine_dir.iterdir(): + if not session_dir.is_dir(): + continue + m = SESSION_RE.match(session_dir.name) + if not m: + continue + date_str, time_str = m.group(1), m.group(2) + # Prefer *_merged.mp4 if present + merged = sorted(session_dir.glob("*_merged.mp4")) + if not merged: + merged = sorted(session_dir.glob("*.mp4")) + if not merged: + continue + rows.append( + { + "machine_uuid": uuid_dir.name, + "machine_name": machine_dir.name, + "session_date": date_str, + "session_time": time_str, + "session_datetime": f"{date_str}_{time_str}", + "mp4_path": str(merged[0]), + } + ) + return pd.DataFrame(rows) + + +def already_tracked_set(data_raw: Path) -> set[tuple[str, str]]: + """Return the set of (date, time) sessions for which a tracking DB exists. + + DBs are named like: + 2025-07-15_16-03-10___1920x1088@25fps-28q_merged_tracking.db + """ + out = set() + for db in data_raw.glob("*_tracking.db"): + m = re.match(r"^(\d{4}-\d{2}-\d{2})_(\d{2}-\d{2}-\d{2})_", db.name) + if m: + out.add((m.group(1), m.group(2))) + return out + + +def main() -> None: + print(f"Scanning {VIDEOS_ROOT} ...") + videos_df = scan_videos(VIDEOS_ROOT) + print(f" found {len(videos_df)} video sessions on disk") + + print(f"Loading metadata xlsx: {VIDEO_INFO_XLSX}") + meta = pd.read_excel(VIDEO_INFO_XLSX) + meta["session_date"] = meta["date"].dt.strftime("%Y-%m-%d") + + # The xlsx has one row per (date, machine, ROI) — collapse to unique sessions + meta_sessions = ( + meta[["session_date", "machine_name"]].drop_duplicates().reset_index(drop=True) + ) + print(f" xlsx contains {len(meta_sessions)} unique (date, machine) sessions") + + # Mark which video sessions are referenced by the xlsx + xlsx_keys = set(zip(meta_sessions["session_date"], meta_sessions["machine_name"])) + videos_df["in_xlsx"] = videos_df.apply( + lambda r: (r["session_date"], r["machine_name"]) in xlsx_keys, axis=1 + ) + + # Mark which already have tracking DBs in data/raw/ + tracked = already_tracked_set(DATA_RAW) + videos_df["already_tracked"] = videos_df.apply( + lambda r: (r["session_date"], r["session_time"]) in tracked, axis=1 + ) + + INVENTORY_CSV.parent.mkdir(parents=True, exist_ok=True) + videos_df.sort_values(["session_date", "machine_name", "session_time"]).to_csv( + INVENTORY_CSV, index=False + ) + + # Coverage report + in_xlsx = videos_df["in_xlsx"] + needed = videos_df[in_xlsx & ~videos_df["already_tracked"]] + n_xlsx_sessions = len(meta_sessions) + n_with_video = videos_df[in_xlsx].drop_duplicates( + ["session_date", "machine_name"] + ).shape[0] + + # xlsx sessions that have no video on disk + found_keys = set( + zip( + videos_df.loc[in_xlsx, "session_date"], + videos_df.loc[in_xlsx, "machine_name"], + ) + ) + missing = sorted(xlsx_keys - found_keys) + + print() + print("=" * 70) + print(f"Wrote inventory: {INVENTORY_CSV}") + print(f" total video sessions on disk: {len(videos_df)}") + print(f" xlsx unique sessions: {n_xlsx_sessions}") + print(f" xlsx sessions with video: {n_with_video}") + print(f" xlsx sessions missing video: {len(missing)}") + print(f" already tracked (DB exists): {videos_df['already_tracked'].sum()}") + print(f" TO TRACK (in_xlsx & ~tracked, video instances): {len(needed)}") + + if missing: + print() + print("xlsx sessions with NO matching video on disk:") + for d, m in missing[:20]: + print(f" {d} {m}") + if len(missing) > 20: + print(f" ... and {len(missing) - 20} more") + + +if __name__ == "__main__": + main() diff --git a/scripts/config.py b/scripts/config.py index 0593c7e..a3462b2 100644 --- a/scripts/config.py +++ b/scripts/config.py @@ -7,3 +7,11 @@ DATA_RAW = PROJECT_ROOT / "data" / "raw" DATA_METADATA = PROJECT_ROOT / "data" / "metadata" DATA_PROCESSED = PROJECT_ROOT / "data" / "processed" FIGURES = PROJECT_ROOT / "figures" + +# Offline-tracking pipeline paths +VIDEOS_ROOT = Path("/mnt/ethoscope_data/videos") +VIDEO_INFO_XLSX = PROJECT_ROOT.parent / "all_video_info_merged.xlsx" +INVENTORY_CSV = DATA_METADATA / "video_inventory.csv" +TARGETS_DIR = PROJECT_ROOT / "data" / "targets" +TRACKING_OUTPUT_DIR = PROJECT_ROOT / "data" / "tracked" +LOGS_DIR = PROJECT_ROOT / "data" / "logs" diff --git a/scripts/monitor_tracking.py b/scripts/monitor_tracking.py new file mode 100644 index 0000000..9ffa891 --- /dev/null +++ b/scripts/monitor_tracking.py @@ -0,0 +1,155 @@ +"""Live progress + ETA for the offline tracker batch. + +Counts ground-truth (DBs on disk) rather than parsing log lines, so it works +whether the batch is running fresh or was resumed after a crash. Errors are +parsed out of any *.log files in data/logs/. + +Usage: + python monitor_tracking.py # one snapshot, exit + python monitor_tracking.py --watch # refresh every 10 s + python monitor_tracking.py --watch 30 # refresh every 30 s +""" + +from __future__ import annotations + +import argparse +import json +import re +import time +from datetime import datetime, timedelta +from pathlib import Path + +from config import LOGS_DIR, TARGETS_DIR, TRACKING_OUTPUT_DIR + + +def count_target_jsons() -> tuple[int, int, list[str]]: + """Return (n_pickable, n_unusable, unusable_video_stems).""" + pickable = 0 + unusable_stems: list[str] = [] + for j in TARGETS_DIR.glob("*.json"): + try: + d = json.loads(j.read_text()) + except Exception: + continue + if d.get("unusable"): + unusable_stems.append(j.stem) + elif d.get("reference_points"): + pickable += 1 + return pickable, len(unusable_stems), unusable_stems + + +def count_tracked_dbs() -> tuple[int, datetime | None, str | None]: + """Return (n_dbs, mtime_of_newest, name_of_newest).""" + dbs = list(TRACKING_OUTPUT_DIR.glob("*_tracking.db")) + if not dbs: + return 0, None, None + newest = max(dbs, key=lambda p: p.stat().st_mtime) + return len(dbs), datetime.fromtimestamp(newest.stat().st_mtime), newest.stem + + +def parse_recent_errors(log_dir: Path, tail_lines: int = 5000) -> list[str]: + """Scan the most recent *.log file for lines reporting errors.""" + if not log_dir.exists(): + return [] + logs = sorted(log_dir.glob("*.log"), key=lambda p: p.stat().st_mtime) + if not logs: + return [] + latest = logs[-1] + try: + with latest.open() as f: + tail = f.readlines()[-tail_lines:] + except Exception: + return [] + out = [] + for line in tail: + if re.search(r":\s*error\b", line) or " error: " in line.lower(): + out.append(line.rstrip()) + return out + + +def db_completion_history() -> list[float]: + """Return mtimes of all tracking DBs, sorted ascending. Used for rate.""" + return sorted(p.stat().st_mtime for p in TRACKING_OUTPUT_DIR.glob("*_tracking.db")) + + +def fmt_duration(seconds: float) -> str: + if seconds < 60: + return f"{int(seconds)} s" + if seconds < 3600: + return f"{int(seconds // 60)} min" + h = int(seconds // 3600) + m = int((seconds % 3600) // 60) + return f"{h} h {m} min" + + +def snapshot() -> str: + pickable, unusable, _ = count_target_jsons() + tracked, last_mtime, last_name = count_tracked_dbs() + history = db_completion_history() + errors = parse_recent_errors(LOGS_DIR) + + lines = [f"tracking progress @ {datetime.now():%Y-%m-%d %H:%M:%S}"] + lines.append(f" pickable JSONs: {pickable}") + lines.append(f" unusable JSONs: {unusable} (skipped by tracker)") + pct = (tracked / pickable * 100) if pickable else 0 + lines.append( + f" DBs on disk: {tracked} / {pickable} ({pct:.0f}%)" + ) + lines.append(f" errors in log: {len(errors)}") + + # Rate from the last 10 completions, when available. + if len(history) >= 2: + window = history[-min(10, len(history)) :] + span = window[-1] - window[0] + if span > 0: + rate_per_hour = (len(window) - 1) / span * 3600 + lines.append(f" rate (last {len(window) - 1}): {rate_per_hour:.1f} videos/hour") + remaining = max(0, pickable - tracked) + if rate_per_hour > 0 and remaining > 0: + eta_sec = remaining * 3600 / rate_per_hour + eta_at = datetime.now() + timedelta(seconds=eta_sec) + lines.append( + f" ETA remaining: {fmt_duration(eta_sec)} " + f"(done by {eta_at:%H:%M %a})" + ) + + if last_mtime is not None and last_name is not None: + ago = (datetime.now() - last_mtime).total_seconds() + lines.append( + f" most recent DB: {last_name[:60]}... ({fmt_duration(ago)} ago)" + ) + + if errors: + lines.append("") + lines.append(f" recent errors ({min(5, len(errors))} of {len(errors)}):") + for e in errors[-5:]: + lines.append(f" {e[:120]}") + + return "\n".join(lines) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--watch", nargs="?", type=int, const=10, default=None, + help="refresh every N seconds (default 10 if flag given without value)", + ) + args = parser.parse_args() + + if args.watch is None: + print(snapshot()) + return + + try: + while True: + # Clear screen and reprint + print("\033[2J\033[H", end="") + print(snapshot()) + print(f"\n(refreshing every {args.watch}s — Ctrl-C to exit)") + time.sleep(args.watch) + except KeyboardInterrupt: + print() + + +if __name__ == "__main__": + main() diff --git a/scripts/pick_targets.py b/scripts/pick_targets.py new file mode 100644 index 0000000..a5eea07 --- /dev/null +++ b/scripts/pick_targets.py @@ -0,0 +1,467 @@ +"""Interactive target picker for offline tracking (matplotlib/Tk GUI). + +Loops through videos that need tracking and lets the user click 3 reference +points per video in L-shape order: + + 1) TOP target (above the corner) + 2) CORNER target (the right-angle vertex) + 3) LEFT target (to the left of the corner) + +These three points are the same reference layout used by ethoscope's +`TargetGridROIBuilder`: dst_points = [(0, -1), (0, 0), (-1, 0)] in unit +coordinates. Saving them as a JSON sidecar lets the offline tracker build the +6-ROI HD mating arena grid without needing auto-target detection. + +Output JSON sidecar: data/targets/.json + { + "video_path": "/mnt/.../*.mp4", + "frame_index": , + "reference_points": [[x0, y0], [x1, y1], [x2, y2]], + "order": ["top", "corner", "left"], + "picked_at": "" + } + +Keys (in the picker window): + LEFT-CLICK add a point (top → corner → left) + r reset clicks for current video + d skip this video for THIS run only (no JSON written) + u mark this video unusable (FOV wrong etc.); skipped forever + . / , advance / rewind by 25 frames (≈ 1 s @ 25 fps) + ] / [ advance / rewind by 5% of the video (~3 min in a 1 h video) + # jump to the middle of the video + enter save the 3 points and move on + q / ESC quit picker + +After the 3rd click, the 6 ROI rectangles are drawn over the frame so you +can sanity-check the geometry before pressing ENTER. + +With --redo, if a JSON sidecar exists, its points are pre-loaded so you can +nudge them rather than restart from scratch. + +Why matplotlib instead of cv2.imshow: + OpenCV's bundled GUI uses Qt, which needs XKeyboard + a fonts directory and + is fragile over SSH X11-forwarding. matplotlib's TkAgg backend uses pure + Tk/X11 and works out of the box on any DISPLAY (and gives free pan/zoom + via the toolbar — useful for clicking small targets precisely). +""" + +from __future__ import annotations + +import argparse +import datetime as dt +import json +import os +import sys +from pathlib import Path + +# Force TkAgg BEFORE importing matplotlib. We override even if MPLBACKEND is +# already set, because the script is unusable with a non-interactive backend. +os.environ["MPLBACKEND"] = "TkAgg" + +import cv2 # noqa: E402 +import matplotlib # noqa: E402 +import matplotlib.pyplot as plt # noqa: E402 +import numpy as np # noqa: E402 +import pandas as pd # noqa: E402 + +# matplotlib.backend_bases exposes the cursor identifiers under different +# names depending on version: `Cursors` enum on 3.5+, lowercase `cursors` +# instance on older releases. Both have the same integer attributes. +try: + from matplotlib.backend_bases import Cursors as _Cursors # 3.5+ +except ImportError: + try: + from matplotlib.backend_bases import cursors as _Cursors # older + except ImportError: + _Cursors = None + +# Verify we ended up on an interactive backend; bail loud (with a concrete +# explanation) if not. matplotlib silently falls back to 'agg' when its +# requested backend can't load, which is hard to debug without help. +_backend = matplotlib.get_backend() +if _backend.lower() in ("agg", "headless", "template", "pdf", "svg", "ps"): + diag = [] + try: + import tkinter as _tk + try: + _tk.Tk().destroy() + diag.append("tkinter import + Tk() instantiation: OK") + except Exception as e: + diag.append(f"tkinter imported but Tk() failed: {e!r}") + except Exception as e: + diag.append(f"tkinter import FAILED: {e!r}") + diag.append(" → on Manjaro/Arch, run: sudo pacman -S tk") + print( + f"ERROR: matplotlib loaded the non-interactive backend {_backend!r}.\n" + f" Expected 'TkAgg'. Diagnostic info:\n" + f" DISPLAY = {os.environ.get('DISPLAY')!r}\n" + f" MPLBACKEND = {os.environ.get('MPLBACKEND')!r}\n" + f" matplotlib ver = {matplotlib.__version__}\n" + + "\n".join(f" {d}" for d in diag), + file=sys.stderr, + ) + sys.exit(2) + +from config import INVENTORY_CSV, TARGETS_DIR # noqa: E402 +from tracking_geometry import compute_roi_polygons # noqa: E402 + +# Strip default matplotlib keybindings that would conflict with ours. +for k in ("keymap.home", "keymap.save", "keymap.quit", "keymap.fullscreen", + "keymap.pan", "keymap.zoom", "keymap.back", "keymap.forward"): + try: + plt.rcParams[k] = [] + except KeyError: + pass + +CLICK_LABELS = ("TOP", "CORNER", "LEFT") +CLICK_COLORS = ("red", "lime", "deepskyblue") + + +def grab_frame( + video_path: Path, frame_idx: int +) -> tuple[np.ndarray, int, int] | None: + """Return (RGB frame, actual_frame_idx, n_frames) from the video, or None. + + Clamps frame_idx to [0, n_frames-1] so callers can step blindly. + """ + cap = cv2.VideoCapture(str(video_path)) + if not cap.isOpened(): + return None + n = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) + if n > 0: + frame_idx = max(0, min(frame_idx, n - 1)) + cap.set(cv2.CAP_PROP_POS_FRAMES, frame_idx) + ok, frame = cap.read() + cap.release() + if not ok or frame is None: + return None + return cv2.cvtColor(frame, cv2.COLOR_BGR2RGB), frame_idx, n + + +def pick_one( + video_path: Path, + frame_idx: int, + status_prefix: str, + initial_points: list[tuple[float, float]] | None = None, +) -> dict | None: + """Show the picker UI for a single video; return the result dict or None.""" + grabbed = grab_frame(video_path, frame_idx) + if grabbed is None: + print(f" ! cannot read {video_path}") + return None + frame, frame_idx, n_frames = grabbed + # Big-step size for ] / [ : 5% of total length, ~3 min in a 1h video. + big_step = max(1, int(round(0.05 * n_frames))) if n_frames > 0 else 250 + + fig, ax = plt.subplots(figsize=(14, 8)) + try: + fig.canvas.manager.set_window_title("pick targets") + except Exception: + pass + # Use a crosshair cursor over the axes so it's obvious where the click + # will land. matplotlib's toolbar resets the cursor to POINTER (arrow) on + # every mouse-move when no tool is active, so we intercept set_cursor: + # whenever it asks for POINTER, we substitute SELECT_REGION (crosshair). + # Tool modes (zoom/pan) keep their native cursors. + if _Cursors is not None: + _orig_set_cursor = fig.canvas.set_cursor + + def _set_cursor_with_crosshair(cursor): + if cursor == _Cursors.POINTER: + cursor = _Cursors.SELECT_REGION + return _orig_set_cursor(cursor) + + fig.canvas.set_cursor = _set_cursor_with_crosshair + try: + fig.canvas.set_cursor(_Cursors.SELECT_REGION) + except Exception: + pass + else: + # Last-ditch: just set the Tk widget's cursor once and hope the + # toolbar doesn't immediately overwrite it. + try: + fig.canvas.get_tk_widget().config(cursor="tcross") + except Exception: + pass + img_artist = ax.imshow(frame) + ax.set_axis_off() + fig.tight_layout() + + state = { + "points": list(initial_points) if initial_points else [], + "action": None, # 'save' | 'skip' | 'quit' | 'unusable' + "frame": frame, + "frame_idx": frame_idx, + "drawn": [], # artists drawn on top of the image + } + + def update_title(): + nb = len(state["points"]) + nxt = ( + f"click {CLICK_LABELS[nb]}" + if nb < 3 + else "ENTER=save | r=reset d=skip u=unusable q=quit | . , [ ] # = step frame" + ) + ax.set_title( + f'{status_prefix} frame {state["frame_idx"]} | {nxt}', + fontsize=10, + ) + + def redraw_points(): + for a in state["drawn"]: + try: + a.remove() + except Exception: + pass + state["drawn"].clear() + for i, (x, y) in enumerate(state["points"]): + color = CLICK_COLORS[i] + label = CLICK_LABELS[i] + (cross,) = ax.plot(x, y, marker="+", color=color, markersize=22, mew=2) + (ring,) = ax.plot( + x, y, marker="o", color=color, markersize=22, + fillstyle="none", mew=2, + ) + txt = ax.text( + x + 14, y - 14, label, + color=color, fontsize=10, weight="bold", + ) + state["drawn"].extend([cross, ring, txt]) + if len(state["points"]) >= 2: + (line1,) = ax.plot( + [state["points"][0][0], state["points"][1][0]], + [state["points"][0][1], state["points"][1][1]], + color="white", linewidth=0.7, alpha=0.6, + ) + state["drawn"].append(line1) + if len(state["points"]) == 3: + (line2,) = ax.plot( + [state["points"][1][0], state["points"][2][0]], + [state["points"][1][1], state["points"][2][1]], + color="white", linewidth=0.7, alpha=0.6, + ) + state["drawn"].append(line2) + # ROI overlay — draw the 6 computed rectangles on top of the frame + try: + polys = compute_roi_polygons(state["points"]) + except Exception as e: + polys = [] + print(f" (ROI preview failed: {e})") + for j, poly in enumerate(polys): + # Close the polygon by repeating the first point + xs = list(poly[:, 0]) + [poly[0, 0]] + ys = list(poly[:, 1]) + [poly[0, 1]] + (line,) = ax.plot( + xs, ys, color="yellow", linewidth=1.5, alpha=0.9, + ) + state["drawn"].append(line) + cx = float(np.mean(poly[:, 0])) + cy = float(np.mean(poly[:, 1])) + lbl = ax.text( + cx, cy, str(j + 1), + color="yellow", fontsize=14, weight="bold", + ha="center", va="center", + ) + state["drawn"].append(lbl) + update_title() + fig.canvas.draw_idle() + + def reload_frame(new_idx: int): + grabbed = grab_frame(video_path, new_idx) + if grabbed is None: + return + new_frame, new_idx, _ = grabbed + state["frame"] = new_frame + state["frame_idx"] = new_idx + img_artist.set_data(new_frame) + # Keep clicked targets + ROI overlay in place across frame-stepping — + # press 'r' to clear them explicitly. + redraw_points() + + def on_click(event): + if event.inaxes is not ax: + return + if event.button != 1: # left click only + return + if event.xdata is None or event.ydata is None: + return + # Skip clicks fired while the toolbar's pan/zoom is active. + toolbar = getattr(fig.canvas, "toolbar", None) + if toolbar is not None and getattr(toolbar, "mode", ""): + return + x, y = float(event.xdata), float(event.ydata) + if len(state["points"]) < 3: + state["points"].append((x, y)) + else: + # 3 points already there — replace the nearest one. Lets the user + # nudge pre-loaded targets in --redo mode, or correct a bad click. + dists = [(x - px) ** 2 + (y - py) ** 2 for px, py in state["points"]] + i_nearest = min(range(3), key=dists.__getitem__) + state["points"][i_nearest] = (x, y) + redraw_points() + + def on_key(event): + k = event.key or "" + if k in ("escape", "q"): + state["action"] = "quit" + plt.close(fig) + elif k == "r": + state["points"].clear() + redraw_points() + elif k == "d": + state["action"] = "skip" + plt.close(fig) + elif k == "u": + state["action"] = "unusable" + plt.close(fig) + elif k == "enter": + if len(state["points"]) == 3: + state["action"] = "save" + plt.close(fig) + elif k == ".": + reload_frame(state["frame_idx"] + 25) + elif k == ",": + reload_frame(state["frame_idx"] - 25) + elif k == "]": + reload_frame(state["frame_idx"] + big_step) + elif k == "[": + reload_frame(state["frame_idx"] - big_step) + elif k == "#": + if n_frames > 0: + reload_frame(n_frames // 2) + + fig.canvas.mpl_connect("button_press_event", on_click) + fig.canvas.mpl_connect("key_press_event", on_key) + update_title() + plt.show() # blocks until the figure is closed + + if state["action"] == "save": + return { + "action": "save", + "frame_idx": state["frame_idx"], + "points": state["points"], + } + if state["action"] == "unusable": + return {"action": "unusable", "frame_idx": state["frame_idx"]} + if state["action"] in ("skip", "quit"): + return {"action": state["action"]} + # Window closed via the WM "X" button — treat as quit so the loop stops + return {"action": "quit"} + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--redo", action="store_true", + help="re-pick videos that already have JSON sidecars", + ) + parser.add_argument( + "--frame", type=int, default=125, + help="default frame index to display (default 125 ≈ 5 s @ 25 fps)", + ) + parser.add_argument( + "--limit", type=int, default=None, + help="only process the first N videos", + ) + args = parser.parse_args() + + if not INVENTORY_CSV.exists(): + sys.exit( + f"Inventory not found at {INVENTORY_CSV}. " + "Run build_video_inventory.py first." + ) + + inv = pd.read_csv(INVENTORY_CSV) + todo = inv[inv["in_xlsx"] & ~inv["already_tracked"]].copy() + todo = todo.sort_values( + ["session_date", "machine_name", "session_time"] + ).reset_index(drop=True) + + TARGETS_DIR.mkdir(parents=True, exist_ok=True) + + def sidecar_for(mp4_path: str) -> Path: + return TARGETS_DIR / (Path(mp4_path).stem + ".json") + + if not args.redo: + todo = todo[ + ~todo["mp4_path"].apply(lambda p: sidecar_for(p).exists()) + ].reset_index(drop=True) + + if args.limit: + todo = todo.head(args.limit) + + n = len(todo) + if n == 0: + print("Nothing to pick. All eligible videos already have target JSONs.") + return + + print( + f"Picking targets for {n} videos. " + "Window keys: ENTER=save r=reset d=skip u=unusable q=quit " + ".,[]=step frame | pan/zoom via toolbar" + ) + saved = skipped = unusable = 0 + for i, row in todo.iterrows(): + mp4 = Path(row["mp4_path"]) + prefix = f"[{i + 1}/{n}] {row['machine_name']} {row['session_datetime']}" + print(f"\n{prefix}") + + # If --redo and a JSON sidecar exists, pre-load its points (only for + # regular saves — unusable sidecars are left as-is and shown empty). + initial_points = None + existing = sidecar_for(row["mp4_path"]) + if args.redo and existing.exists(): + try: + prev = json.loads(existing.read_text()) + if not prev.get("unusable") and prev.get("reference_points"): + initial_points = [tuple(p) for p in prev["reference_points"]] + print(f" pre-loaded {len(initial_points)} previous point(s)") + except Exception as e: + print(f" ! could not read previous sidecar: {e}") + + result = pick_one(mp4, args.frame, prefix, initial_points=initial_points) + if result is None or result.get("action") == "quit": + print(" quitting picker.") + break + if result["action"] == "skip": + skipped += 1 + print(" skipped (no JSON written, will be re-asked next run).") + continue + if result["action"] == "unusable": + try: + reason = input(" reason for marking unusable (Enter to skip): ").strip() + except EOFError: + reason = "" + payload = { + "video_path": str(mp4), + "unusable": True, + "reason": reason, + "marked_at": dt.datetime.now().isoformat(timespec="seconds"), + } + out_path = sidecar_for(row["mp4_path"]) + out_path.write_text(json.dumps(payload, indent=2)) + unusable += 1 + print(f" marked unusable → {out_path.name}") + continue + if result["action"] == "save": + payload = { + "video_path": str(mp4), + "frame_index": int(result["frame_idx"]), + "reference_points": [list(map(int, p)) for p in result["points"]], + "order": ["top", "corner", "left"], + "picked_at": dt.datetime.now().isoformat(timespec="seconds"), + } + out_path = sidecar_for(row["mp4_path"]) + out_path.write_text(json.dumps(payload, indent=2)) + saved += 1 + print(f" saved → {out_path.name}") + + remaining = n - saved - skipped - unusable + print( + f"\nDone. saved={saved} unusable={unusable} " + f"skipped(this run)={skipped} remaining={remaining}" + ) + + +if __name__ == "__main__": + main() diff --git a/scripts/track_videos.py b/scripts/track_videos.py new file mode 100644 index 0000000..d9bd197 --- /dev/null +++ b/scripts/track_videos.py @@ -0,0 +1,218 @@ +"""Headless offline tracker. + +Reads target JSONs produced by `pick_targets.py`, builds the 6 ROIs of the +HD mating arena from the L-shape reference points, runs ethoscope's +`MultiFlyTracker` against the merged.mp4 file via `MovieVirtualCamera`, and +writes a SQLite DB to `data/tracked/_tracking.db`. + +Idempotent: skips videos whose tracking DB already exists (unless --redo). + +Usage: + python track_videos.py # process all videos with target JSON + python track_videos.py --redo # re-track even if DB exists + python track_videos.py --jobs 4 # run up to 4 videos in parallel + python track_videos.py --max-duration 1800 # cap each video at 30 min (sec) +""" + +from __future__ import annotations + +import argparse +import json +import logging +import os +import sys +import traceback +from concurrent.futures import ProcessPoolExecutor, as_completed +from pathlib import Path + +import numpy as np + +# Import ethoscope from the local source tree (no pip install). +ETHOSCOPE_SRC = Path("/home/gg/Code/ethoscope_project/ethoscope/src/ethoscope") +sys.path.insert(0, str(ETHOSCOPE_SRC)) + +from config import TARGETS_DIR, TRACKING_OUTPUT_DIR # noqa: E402 +from tracking_geometry import HD_FG_DATA, compute_roi_polygons # noqa: E402 + + +def build_rois_from_targets(reference_points): + """Wrap the shared geometry into ethoscope `ROI` objects.""" + from ethoscope.core.roi import ROI + + polys = compute_roi_polygons(reference_points) + return [ROI(poly.reshape((1, 4, 2)), idx=i + 1) for i, poly in enumerate(polys)] + + +def track_one(json_path: Path, output_dir: Path, max_duration: float | None, + redo: bool) -> tuple[str, str]: + """Track a single video. Returns (status, message). Run in subprocess. + + Statuses: "ok", "skip", "error". + """ + # Re-import inside subprocess so each worker has its own ethoscope state. + import sys as _sys + _sys.path.insert(0, str(ETHOSCOPE_SRC)) + import cv2 + from ethoscope.core.monitor import Monitor + from ethoscope.hardware.input.cameras import MovieVirtualCamera + from ethoscope.io.sqlite import SQLiteResultWriter + from ethoscope.trackers.multi_fly_tracker import MultiFlyTracker + + class BGRMovieCamera(MovieVirtualCamera): + """MovieVirtualCamera variant that keeps BGR frames. + + MultiFlyTracker calls cv2.cvtColor(img, COLOR_BGR2GRAY) without checking + whether img is already grayscale, so we must feed it 3-channel input. + """ + def _next_image(self): + ret, frame = self.capture.read() + if not ret or frame is None: + return None + return frame # BGR, untouched + + payload = json.loads(json_path.read_text()) + if payload.get("unusable"): + reason = payload.get("reason") or "no reason given" + return "skip", f"marked unusable: {reason}" + video_path = Path(payload["video_path"]) + if not video_path.exists(): + return "error", f"video missing: {video_path}" + + out_db = output_dir / f"{video_path.stem}_tracking.db" + if out_db.exists() and not redo: + return "skip", f"DB exists: {out_db.name}" + if out_db.exists(): + out_db.unlink() + + rois = build_rois_from_targets(payload["reference_points"]) + + cam_kwargs = {"use_wall_clock": False} + if max_duration is not None: + cam_kwargs["max_duration"] = max_duration + cam = BGRMovieCamera(str(video_path), **cam_kwargs) + + metadata = { + "machine_id": payload.get("machine_uuid", "unknown"), + "machine_name": payload.get("machine_name", "unknown"), + "date_time": int(payload.get("session_epoch", 0)), + "frame_width": cam.width, + "frame_height": cam.height, + "version": "offline-tracker-1", + "experimental_info": "{}", + "selected_options": json.dumps({ + "tracker": "MultiFlyTracker", + "template": "HD_Mating_Arena_6_ROIS", + "fg_data": HD_FG_DATA, + "maxN": 2, + }), + "hardware_info": "{}", + "reference_points": str([list(map(int, p)) for p in payload["reference_points"]]), + "backup_filename": out_db.name, + "result_writer_type": "SQLite3", + "sqlite_source_path": str(out_db), + } + + tracker_data = { + "maxN": 2, + "visualise": False, + "fg_data": HD_FG_DATA, + "adaptive_threshold": True, + "min_fg_threshold": 10, + "max_fg_threshold": 50, + } + + db_credentials = {"name": str(out_db)} + rw = SQLiteResultWriter( + db_credentials, rois, metadata=metadata, + make_dam_like_table=False, take_frame_shots=False, erase_old_db=True, + ) + + monit = Monitor( + cam, MultiFlyTracker, rois, + reference_points=payload["reference_points"], + data=tracker_data, + ) + + try: + with rw as result_writer: + monit.run(result_writer=result_writer, drawer=None, verbose=False) + except Exception: + return "error", traceback.format_exc(limit=5) + finally: + try: + cam._close() + except Exception: + pass + + if not out_db.exists(): + return "error", "tracking finished but DB was not created" + return "ok", str(out_db) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--redo", action="store_true", help="re-track even if DB exists") + parser.add_argument("--jobs", type=int, default=1, help="parallel workers") + parser.add_argument( + "--max-duration", type=float, default=None, + help="cap each video at this many seconds (default: full video)", + ) + parser.add_argument("--limit", type=int, default=None, help="process only first N") + parser.add_argument("--video", type=str, default=None, + help="track a single video (mp4 path); requires its target JSON") + args = parser.parse_args() + + TRACKING_OUTPUT_DIR.mkdir(parents=True, exist_ok=True) + + if args.video: + stem = Path(args.video).stem + json_path = TARGETS_DIR / f"{stem}.json" + if not json_path.exists(): + sys.exit(f"No target JSON for {args.video}: expected {json_path}") + jsons = [json_path] + else: + jsons = sorted(TARGETS_DIR.glob("*.json")) + + if args.limit: + jsons = jsons[: args.limit] + + if not jsons: + print("No target JSONs found. Run pick_targets.py first.") + return + + print(f"Tracking {len(jsons)} videos (jobs={args.jobs}, redo={args.redo}).") + n_ok = n_skip = n_err = 0 + + if args.jobs <= 1: + for jp in jsons: + print(f" → {jp.name}", flush=True) + status, msg = track_one(jp, TRACKING_OUTPUT_DIR, args.max_duration, args.redo) + print(f" {status}: {msg.splitlines()[-1] if msg else ''}", flush=True) + n_ok += status == "ok" + n_skip += status == "skip" + n_err += status == "error" + else: + with ProcessPoolExecutor(max_workers=args.jobs) as ex: + futs = { + ex.submit(track_one, jp, TRACKING_OUTPUT_DIR, args.max_duration, args.redo): jp + for jp in jsons + } + for fut in as_completed(futs): + jp = futs[fut] + try: + status, msg = fut.result() + except Exception as e: + status, msg = "error", f"future raised: {e}" + print(f" {jp.name}: {status} — {msg.splitlines()[-1] if msg else ''}", + flush=True) + n_ok += status == "ok" + n_skip += status == "skip" + n_err += status == "error" + + print(f"\nDone. ok={n_ok} skipped={n_skip} errors={n_err}") + sys.exit(0 if n_err == 0 else 1) + + +if __name__ == "__main__": + logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s") + main() diff --git a/scripts/tracking_geometry.py b/scripts/tracking_geometry.py new file mode 100644 index 0000000..1f98918 --- /dev/null +++ b/scripts/tracking_geometry.py @@ -0,0 +1,71 @@ +"""Shared HD-mating-arena ROI geometry, used by both pick_targets.py +(for live overlay) and track_videos.py (for actual tracking). + +Pure numpy + cv2; no ethoscope dependency. +""" + +from __future__ import annotations + +import itertools + +import cv2 +import numpy as np + +# Layout from +# ethoscope/.../roi_builders/roi_templates/builtin/HD_Mating_Arena_6_ROIS.json +HD_MATING_ARENA = { + "n_rows": 2, + "n_cols": 3, + "top_margin": -0.21, + "bottom_margin": -0.13, + "left_margin": 0.05, + "right_margin": 0.05, + "horizontal_fill": 0.85, + "vertical_fill": 1.3, +} + +HD_FG_DATA = { + "sample_size": 400, + "normal_limits": [800, 2000], + "tolerance": 0.8, +} + + +def compute_roi_polygons(reference_points, layout=HD_MATING_ARENA): + """Map 3 L-shape reference points to 6 ROI polygons, in the order ROI 1..6. + + Reference points must be ordered: + [TOP, CORNER, LEFT] + matching ethoscope's dst_points = [(0, -1), (0, 0), (-1, 0)]. + + Returns: + list[np.ndarray] # 6 arrays, each shape (4, 2), int32, in image coords + """ + ref = np.asarray(reference_points, dtype=np.float32) + if ref.shape != (3, 2): + raise ValueError(f"reference_points must be 3x2, got shape {ref.shape}") + + dst_points = np.array([(0, -1), (0, 0), (-1, 0)], dtype=np.float32) + wrap_mat = cv2.getAffineTransform(dst_points, ref) + + n_col = layout["n_cols"] + n_row = layout["n_rows"] + tm, bm = layout["top_margin"], layout["bottom_margin"] + lm, rm = layout["left_margin"], layout["right_margin"] + hf, vf = layout["horizontal_fill"], layout["vertical_fill"] + + y_positions = (np.arange(n_row) * 2.0 + 1) * (1 - tm - bm) / (2 * n_row) + tm + x_positions = (np.arange(n_col) * 2.0 + 1) * (1 - lm - rm) / (2 * n_col) + lm + centres = [np.array([x, y]) for x, y in itertools.product(x_positions, y_positions)] + sign_mat = np.array([[-1, -1], [+1, -1], [+1, +1], [-1, +1]]) + xy_size = np.array([hf / float(n_col), vf / float(n_row)]) / 2.0 + rectangles = [sign_mat * xy_size + c for c in centres] + + shift = np.dot(wrap_mat, [1, 1, 0]) - ref[1] + + polys = [] + for r in rectangles: + r3 = np.append(r, np.zeros((4, 1)), axis=1) + mapped = np.dot(wrap_mat, r3.T).T - shift + polys.append(mapped.astype(np.int32)) + return polys diff --git a/tasks/todo.md b/tasks/todo.md index f5e8b3f..f86bd65 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -51,6 +51,68 @@ See `docs/bimodal_hypothesis.md` for detailed methodology. - [ ] Consider converting pixel distances to physical units (need calibration) - [ ] The second notebook (`flies_analysis.ipynb`) re-runs from DB extraction - consider deprecating +## Phase: Offline Tracking of 2024 Video Backlog (added 2026-04-27) + +### Recap + +Tracked so far (5 sessions, all from 2025-07-15, machines 076/145/268). The DBs in +`data/raw/` use tracker `ConstrainedMultiFlyTracker` and template +`HD_Mating_Arena_6_ROIS.json` (2 flies × 6 ROIs per video). + +The metadata file `../all_video_info_merged.xlsx` indexes a different set of +experiments: 7 dates from 2024-09-17 → 2024-10-21, 16 ethoscope machines, +63 unique (date, machine) sessions = 484 ROI-rows. **None of the already-tracked +sessions are in this xlsx — these are fresh recordings to track.** + +Inventory: see `data/metadata/video_inventory.csv` (built by +`scripts/build_video_inventory.py`). +- 1163 video sessions on disk under `/mnt/ethoscope_data/videos/` +- 63/63 xlsx (date, machine) sessions have video on disk +- 129 video instances need tracking (some (date, machine) have 2-4 recordings/day) + +### Plan + +The HD-mating-arena videos have no auto-detectable targets — the user must +manually click 3 reference points (L-shape: top, corner, left) per video. Once +all targets are picked, tracking can run in the background. + +- [x] **Step 1 — Inventory**: `scripts/build_video_inventory.py` → + `data/metadata/video_inventory.csv`. 63 (date,machine) sessions match + the xlsx, all videos found, 129 video instances need tracking. +- [x] **Step 2 — Manual target picker**: `scripts/pick_targets.py`. Loops over + videos with `in_xlsx & ~already_tracked & no JSON yet`; per video, shows + a representative frame, captures 3 clicks (top, corner, left), saves + `data/targets/.json`. Skips videos already done. +- [x] **Step 3 — Background tracker**: `scripts/track_videos.py`. Reads target + JSONs, builds 6 ROIs from the HD-mating-arena geometry, runs + `MovieVirtualCamera` + `MultiFlyTracker` + `SQLiteResultWriter`, writes + `data/tracked/_tracking.db`. Idempotent. Smoke-tested + end-to-end: 90s of video → ~3000 rows/ROI, areas in 800-2000 band. +- [x] **Step 4 — Tracking deps**: `requirements-tracking.txt`. + +### Still TODO +- [ ] User to run `pick_targets.py` (interactive — needs DISPLAY) on the 129 + pending videos. +- [ ] Run `track_videos.py --jobs 4` against the resulting JSONs. +- [ ] (Optional) `auto_detect_targets.py` exists as a fallback for videos that + DO have visible targets (saves clicks). Confirmed not useful on the + 2025-07-15 batch — these arenas don't have black target dots — but worth + trying on 2024 batches before falling back to manual. +- [ ] Decide what to do with the 4 (date, machine) sessions that have 3-4 + recordings/day instead of 2 (e.g. ETHOSCOPE_086 on 2024-09-17 has 4). + One of them is at lower resolution (1280x960) — likely an aborted take. + +### Open questions / risks + +- Some (date, machine) combos have 3-4 recordings (e.g. ETHOSCOPE_086 on + 2024-09-17). Need to figure out which is the real "test" video vs aborted + takes — possibly use video duration or filename pattern. +- One mismatched-resolution file: `1280x960@25fps-20q` instead of + `1920x1088@25fps-28q` — flag for inspection. +- The original `ConstrainedMultiFlyTracker` is no longer in the ethoscope repo; + `MultiFlyTracker` is its likely successor. Validate output schema matches + what the existing analysis pipeline expects (`load_roi_data.py`, etc.). + ## Discovered During Work (Add new items here as they come up during analysis)