lab/cupido
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add offline tracking pipeline for video backlog

Browse source 
The 2024 video set in all_video_info_merged.xlsx covers 63 (date, machine)
sessions — 129 video instances — that have no auto-detectable targets, so
ROI placement requires manual reference-point selection. This commit adds
the three-stage pipeline that lets a user click for an hour, then walk
away while the tracker grinds overnight:

  1. build_video_inventory.py — scan /mnt/ethoscope_data/videos/ and join
     against the xlsx, producing data/metadata/video_inventory.csv

  2. pick_targets.py — interactive matplotlib/Tk picker. User clicks
     TOP/CORNER/LEFT (the L-shape ethoscope expects); after the third
     click the 6 ROI rectangles are drawn on top of the frame so geometry
     can be verified before saving. Also supports marking a video
     'unusable' (FOV wrong) so it's permanently skipped, frame stepping
     by ±1s/±5%/midpoint, point editing in --redo mode, and a crosshair
     cursor that survives matplotlib's per-motion cursor reset.

  3. track_videos.py — headless batch tracker. Reads the JSON sidecars,
     builds 6 ROIs from the HD-mating-arena geometry, runs MultiFlyTracker
     against the merged.mp4 via MovieVirtualCamera, writes SQLite DBs to
     data/tracked/. Idempotent (skips done DBs), parallel via --jobs,
     subclasses MovieVirtualCamera so frames stay BGR (MultiFlyTracker
     calls cvtColor(BGR2GRAY) without checking channel count).

Plus auto_detect_targets.py (fallback that runs ethoscope's auto-detector
in case any videos do have visible target dots), monitor_tracking.py
(progress + ETA from data/tracked/ ground truth, --watch for live view),
and tracking_geometry.py (single source of truth for the affine math
shared by picker and tracker).

requirements-tracking.txt pins the extra deps (opencv-python, openpyxl,
gitpython, netifaces, mysql-connector-python) — these are only needed
for the tracking pipeline, not the existing analysis notebooks.

Verified end-to-end on one of the user-picked videos: ~4000 rows/ROI in
a 120s slice, fly bounding boxes in the expected 800-2000 px² band.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Giorgio Gilestro 2026-04-27 17:25:26 +01:00
parent e7e4db264d
commit e4da7691d5
11 changed files with 1296 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

9
.gitignore vendored
View file

@ -2,6 +2,15 @@
data/raw/*.db
data/processed/*.csv
# Offline-tracking outputs (reproducible from videos + target JSONs)
data/tracked/*.db
data/tracked/*.db-wal
data/tracked/*.db-shm
data/tracked/*.db-journal
data/targets/*.json
data/metadata/video_inventory.csv
data/logs/*.log
# Generated figures (reproducible from scripts)
figures/*.png

26
README.md
View file

@ -46,6 +46,32 @@ The key insight: not all "trained" flies may have actually learned. The trained
**Read `docs/bimodal_hypothesis.md` for the detailed analysis plan and code sketches.**
## Offline Tracking Pipeline (added Apr 2026)
For tracking new videos that have **no auto-detectable targets**, the pipeline
is split in two stages so you can sit at the screen and click for an hour, then
let the tracker grind through overnight.
```bash
# extra deps (ethoscope src must be at /home/gg/Code/ethoscope_project/...)
pip install -r requirements-tracking.txt
# 1) build the inventory (xlsx ↔ /mnt/ethoscope_data/videos/)
python scripts/build_video_inventory.py
# 2) interactive: click TOP, CORNER, LEFT on each video (one frame per video)
python scripts/pick_targets.py # process all not-yet-picked
python scripts/pick_targets.py --redo # re-pick already-picked videos
# keys: r=reset n=skip f=jump frame q/ESC=quit ENTER=save
# 3) batch tracking (idempotent, can run in background)
python scripts/track_videos.py --jobs 4 # parallel
# output → data/tracked/*_tracking.db (SQLite, same schema as data/raw/)
```
See `tasks/todo.md` "Offline Tracking" section for the full plan, and
`data/metadata/video_inventory.csv` for the list of videos to process.
## Folder Structure
```

11
requirements-tracking.txt Normal file
View file

@ -0,0 +1,11 @@
# Extra dependencies needed only for the offline-tracking pipeline
# (build_video_inventory.py, pick_targets.py, auto_detect_targets.py,
# track_videos.py). Not needed for the existing analysis notebooks.
#
# install with: pip install -r requirements-tracking.txt
opencv-python>=4.8
openpyxl>=3.1
gitpython>=3.1
netifaces>=0.11
mysql-connector-python>=8.0
pyserial>=3.5

119
scripts/auto_detect_targets.py Normal file
View file

@ -0,0 +1,119 @@
"""Try auto-detection of L-shape targets on each video and save JSON sidecars.
Useful for:
- videos that DO have visible black-circle targets (saves manual clicks);
- as a smoke test of the whole pipeline before running the picker.
Failure is silent — videos that fail auto-detection are simply not written
to disk, leaving them for the manual `pick_targets.py` tool.
Output JSON has the same shape as the manual picker's so `track_videos.py`
can consume either.
"""
from __future__ import annotations
import argparse
import datetime as dt
import json
import logging
import sys
from pathlib import Path
import cv2
import numpy as np
import pandas as pd
# ethoscope source tree
sys.path.insert(0, "/home/gg/Code/ethoscope_project/ethoscope/src/ethoscope")
from config import INVENTORY_CSV, TARGETS_DIR # noqa: E402
from ethoscope.roi_builders.target_roi_builder import TargetGridROIBuilder # noqa: E402
def detect_one(video_path: Path, frame_idx: int) -> tuple[list[list[int]], int] | None:
"""Run ethoscope target detection on one frame; return (points, frame_idx) or None."""
cap = cv2.VideoCapture(str(video_path))
if not cap.isOpened():
return None
n = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
if n > 0 and frame_idx >= n:
frame_idx = max(0, n - 1)
cap.set(cv2.CAP_PROP_POS_FRAMES, frame_idx)
ok, frame = cap.read()
cap.release()
if not ok or frame is None:
return None
# The detector expects a single-channel image (grey) like ethoscope cameras produce.
if frame.ndim == 3:
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
else:
gray = frame
# We don't actually need a fully-configured grid here — _find_target_coordinates
# alone gives us the 3 reference points.
builder = TargetGridROIBuilder(n_rows=2, n_cols=3)
try:
ref = builder._find_target_coordinates(gray)
except Exception as e:
logging.debug(f"detection failed for {video_path.name}: {e}")
return None
if ref is None:
return None
return [[int(p[0]), int(p[1])] for p in ref], frame_idx
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--frame", type=int, default=125)
parser.add_argument("--limit", type=int, default=None)
parser.add_argument("--video", type=str, default=None,
help="run on a single video path (skips inventory)")
parser.add_argument("--overwrite", action="store_true",
help="overwrite existing JSON sidecars")
args = parser.parse_args()
TARGETS_DIR.mkdir(parents=True, exist_ok=True)
if args.video:
videos = [Path(args.video)]
else:
if not INVENTORY_CSV.exists():
sys.exit("Inventory missing — run build_video_inventory.py first.")
inv = pd.read_csv(INVENTORY_CSV)
todo = inv[inv["in_xlsx"] & ~inv["already_tracked"]]
videos = [Path(p) for p in todo["mp4_path"].tolist()]
if args.limit:
videos = videos[: args.limit]
n_ok = n_fail = n_skip = 0
for v in videos:
out = TARGETS_DIR / f"{v.stem}.json"
if out.exists() and not args.overwrite:
n_skip += 1
continue
result = detect_one(v, args.frame)
if result is None:
n_fail += 1
print(f" fail: {v.name}")
continue
points, used_frame = result
out.write_text(json.dumps({
"video_path": str(v),
"frame_index": int(used_frame),
"reference_points": points,
"order": ["top", "corner", "left"],
"picked_at": dt.datetime.now().isoformat(timespec="seconds"),
"method": "auto",
}, indent=2))
n_ok += 1
print(f" ok: {v.name} → {points}")
print(f"\nDone. ok={n_ok} fail={n_fail} skipped(existing)={n_skip}")
if __name__ == "__main__":
logging.basicConfig(level=logging.WARNING, format="%(levelname)s %(message)s")
main()

150
scripts/build_video_inventory.py Normal file
View file

@ -0,0 +1,150 @@
"""Build an inventory of videos available on disk and join with the metadata xlsx.
Scans /mnt/ethoscope_data/videos/<uuid>/<machine_name>/<date_time>/*.mp4
and produces a CSV mapping each (date, machine_name) row in
all_video_info_merged.xlsx to the corresponding merged.mp4 path on disk.
Output: data/metadata/video_inventory.csv with columns:
machine_uuid, machine_name, session_date, session_time, mp4_path,
in_xlsx (bool), already_tracked (bool)
"""
from __future__ import annotations
import re
from pathlib import Path
import pandas as pd
from config import DATA_RAW, INVENTORY_CSV, VIDEO_INFO_XLSX, VIDEOS_ROOT
SESSION_RE = re.compile(r"^(\d{4}-\d{2}-\d{2})_(\d{2}-\d{2}-\d{2})$")
def scan_videos(videos_root: Path) -> pd.DataFrame:
"""Walk videos_root and return one row per merged.mp4 found.
Args:
videos_root: Root directory containing <uuid>/<machine_name>/<date_time>/.
Returns:
DataFrame with columns: machine_uuid, machine_name, session_date,
session_time, session_datetime, mp4_path.
"""
rows = []
for uuid_dir in sorted(videos_root.iterdir()):
if not uuid_dir.is_dir():
continue
for machine_dir in uuid_dir.iterdir():
if not machine_dir.is_dir() or not machine_dir.name.startswith("ETHOSCOPE_"):
continue
for session_dir in machine_dir.iterdir():
if not session_dir.is_dir():
continue
m = SESSION_RE.match(session_dir.name)
if not m:
continue
date_str, time_str = m.group(1), m.group(2)
# Prefer *_merged.mp4 if present
merged = sorted(session_dir.glob("*_merged.mp4"))
if not merged:
merged = sorted(session_dir.glob("*.mp4"))
if not merged:
continue
rows.append(
{
"machine_uuid": uuid_dir.name,
"machine_name": machine_dir.name,
"session_date": date_str,
"session_time": time_str,
"session_datetime": f"{date_str}_{time_str}",
"mp4_path": str(merged[0]),
}
)
return pd.DataFrame(rows)
def already_tracked_set(data_raw: Path) -> set[tuple[str, str]]:
"""Return the set of (date, time) sessions for which a tracking DB exists.
DBs are named like:
2025-07-15_16-03-10_<uuid>__1920x1088@25fps-28q_merged_tracking.db
"""
out = set()
for db in data_raw.glob("*_tracking.db"):
m = re.match(r"^(\d{4}-\d{2}-\d{2})_(\d{2}-\d{2}-\d{2})_", db.name)
if m:
out.add((m.group(1), m.group(2)))
return out
def main() -> None:
print(f"Scanning {VIDEOS_ROOT} ...")
videos_df = scan_videos(VIDEOS_ROOT)
print(f" found {len(videos_df)} video sessions on disk")
print(f"Loading metadata xlsx: {VIDEO_INFO_XLSX}")
meta = pd.read_excel(VIDEO_INFO_XLSX)
meta["session_date"] = meta["date"].dt.strftime("%Y-%m-%d")
# The xlsx has one row per (date, machine, ROI) — collapse to unique sessions
meta_sessions = (
meta[["session_date", "machine_name"]].drop_duplicates().reset_index(drop=True)
)
print(f" xlsx contains {len(meta_sessions)} unique (date, machine) sessions")
# Mark which video sessions are referenced by the xlsx
xlsx_keys = set(zip(meta_sessions["session_date"], meta_sessions["machine_name"]))
videos_df["in_xlsx"] = videos_df.apply(
lambda r: (r["session_date"], r["machine_name"]) in xlsx_keys, axis=1
)
# Mark which already have tracking DBs in data/raw/
tracked = already_tracked_set(DATA_RAW)
videos_df["already_tracked"] = videos_df.apply(
lambda r: (r["session_date"], r["session_time"]) in tracked, axis=1
)
INVENTORY_CSV.parent.mkdir(parents=True, exist_ok=True)
videos_df.sort_values(["session_date", "machine_name", "session_time"]).to_csv(
INVENTORY_CSV, index=False
)
# Coverage report
in_xlsx = videos_df["in_xlsx"]
needed = videos_df[in_xlsx & ~videos_df["already_tracked"]]
n_xlsx_sessions = len(meta_sessions)
n_with_video = videos_df[in_xlsx].drop_duplicates(
["session_date", "machine_name"]
).shape[0]
# xlsx sessions that have no video on disk
found_keys = set(
zip(
videos_df.loc[in_xlsx, "session_date"],
videos_df.loc[in_xlsx, "machine_name"],
)
)
missing = sorted(xlsx_keys - found_keys)
print()
print("=" * 70)
print(f"Wrote inventory: {INVENTORY_CSV}")
print(f" total video sessions on disk: {len(videos_df)}")
print(f" xlsx unique sessions: {n_xlsx_sessions}")
print(f" xlsx sessions with video: {n_with_video}")
print(f" xlsx sessions missing video: {len(missing)}")
print(f" already tracked (DB exists): {videos_df['already_tracked'].sum()}")
print(f" TO TRACK (in_xlsx & ~tracked, video instances): {len(needed)}")
if missing:
print()
print("xlsx sessions with NO matching video on disk:")
for d, m in missing[:20]:
print(f" {d} {m}")
if len(missing) > 20:
print(f" ... and {len(missing) - 20} more")
if __name__ == "__main__":
main()

8
scripts/config.py
View file

@ -7,3 +7,11 @@ DATA_RAW = PROJECT_ROOT / "data" / "raw"
DATA_METADATA = PROJECT_ROOT / "data" / "metadata"
DATA_PROCESSED = PROJECT_ROOT / "data" / "processed"
FIGURES = PROJECT_ROOT / "figures"
# Offline-tracking pipeline paths
VIDEOS_ROOT = Path("/mnt/ethoscope_data/videos")
VIDEO_INFO_XLSX = PROJECT_ROOT.parent / "all_video_info_merged.xlsx"
INVENTORY_CSV = DATA_METADATA / "video_inventory.csv"
TARGETS_DIR = PROJECT_ROOT / "data" / "targets"
TRACKING_OUTPUT_DIR = PROJECT_ROOT / "data" / "tracked"
LOGS_DIR = PROJECT_ROOT / "data" / "logs"

155
scripts/monitor_tracking.py Normal file
View file

@ -0,0 +1,155 @@
"""Live progress + ETA for the offline tracker batch.
Counts ground-truth (DBs on disk) rather than parsing log lines, so it works
whether the batch is running fresh or was resumed after a crash. Errors are
parsed out of any *.log files in data/logs/.
Usage:
python monitor_tracking.py # one snapshot, exit
python monitor_tracking.py --watch # refresh every 10 s
python monitor_tracking.py --watch 30 # refresh every 30 s
"""
from __future__ import annotations
import argparse
import json
import re
import time
from datetime import datetime, timedelta
from pathlib import Path
from config import LOGS_DIR, TARGETS_DIR, TRACKING_OUTPUT_DIR
def count_target_jsons() -> tuple[int, int, list[str]]:
"""Return (n_pickable, n_unusable, unusable_video_stems)."""
pickable = 0
unusable_stems: list[str] = []
for j in TARGETS_DIR.glob("*.json"):
try:
d = json.loads(j.read_text())
except Exception:
continue
if d.get("unusable"):
unusable_stems.append(j.stem)
elif d.get("reference_points"):
pickable += 1
return pickable, len(unusable_stems), unusable_stems
def count_tracked_dbs() -> tuple[int, datetime | None, str | None]:
"""Return (n_dbs, mtime_of_newest, name_of_newest)."""
dbs = list(TRACKING_OUTPUT_DIR.glob("*_tracking.db"))
if not dbs:
return 0, None, None
newest = max(dbs, key=lambda p: p.stat().st_mtime)
return len(dbs), datetime.fromtimestamp(newest.stat().st_mtime), newest.stem
def parse_recent_errors(log_dir: Path, tail_lines: int = 5000) -> list[str]:
"""Scan the most recent *.log file for lines reporting errors."""
if not log_dir.exists():
return []
logs = sorted(log_dir.glob("*.log"), key=lambda p: p.stat().st_mtime)
if not logs:
return []
latest = logs[-1]
try:
with latest.open() as f:
tail = f.readlines()[-tail_lines:]
except Exception:
return []
out = []
for line in tail:
if re.search(r":\s*error\b", line) or " error: " in line.lower():
out.append(line.rstrip())
return out
def db_completion_history() -> list[float]:
"""Return mtimes of all tracking DBs, sorted ascending. Used for rate."""
return sorted(p.stat().st_mtime for p in TRACKING_OUTPUT_DIR.glob("*_tracking.db"))
def fmt_duration(seconds: float) -> str:
if seconds < 60:
return f"{int(seconds)} s"
if seconds < 3600:
return f"{int(seconds // 60)} min"
h = int(seconds // 3600)
m = int((seconds % 3600) // 60)
return f"{h} h {m} min"
def snapshot() -> str:
pickable, unusable, _ = count_target_jsons()
tracked, last_mtime, last_name = count_tracked_dbs()
history = db_completion_history()
errors = parse_recent_errors(LOGS_DIR)
lines = [f"tracking progress @ {datetime.now():%Y-%m-%d %H:%M:%S}"]
lines.append(f" pickable JSONs: {pickable}")
lines.append(f" unusable JSONs: {unusable} (skipped by tracker)")
pct = (tracked / pickable * 100) if pickable else 0
lines.append(
f" DBs on disk: {tracked} / {pickable} ({pct:.0f}%)"
)
lines.append(f" errors in log: {len(errors)}")
# Rate from the last 10 completions, when available.
if len(history) >= 2:
window = history[-min(10, len(history)) :]
span = window[-1] - window[0]
if span > 0:
rate_per_hour = (len(window) - 1) / span * 3600
lines.append(f" rate (last {len(window) - 1}): {rate_per_hour:.1f} videos/hour")
remaining = max(0, pickable - tracked)
if rate_per_hour > 0 and remaining > 0:
eta_sec = remaining * 3600 / rate_per_hour
eta_at = datetime.now() + timedelta(seconds=eta_sec)
lines.append(
f" ETA remaining: {fmt_duration(eta_sec)} "
f"(done by {eta_at:%H:%M %a})"
)
if last_mtime is not None and last_name is not None:
ago = (datetime.now() - last_mtime).total_seconds()
lines.append(
f" most recent DB: {last_name[:60]}... ({fmt_duration(ago)} ago)"
)
if errors:
lines.append("")
lines.append(f" recent errors ({min(5, len(errors))} of {len(errors)}):")
for e in errors[-5:]:
lines.append(f" {e[:120]}")
return "\n".join(lines)
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--watch", nargs="?", type=int, const=10, default=None,
help="refresh every N seconds (default 10 if flag given without value)",
)
args = parser.parse_args()
if args.watch is None:
print(snapshot())
return
try:
while True:
# Clear screen and reprint
print("\033[2J\033[H", end="")
print(snapshot())
print(f"\n(refreshing every {args.watch}s — Ctrl-C to exit)")
time.sleep(args.watch)
except KeyboardInterrupt:
print()
if __name__ == "__main__":
main()

467
scripts/pick_targets.py Normal file
View file

@ -0,0 +1,467 @@
"""Interactive target picker for offline tracking (matplotlib/Tk GUI).
Loops through videos that need tracking and lets the user click 3 reference
points per video in L-shape order:
1) TOP target (above the corner)
2) CORNER target (the right-angle vertex)
3) LEFT target (to the left of the corner)
These three points are the same reference layout used by ethoscope's
`TargetGridROIBuilder`: dst_points = [(0, -1), (0, 0), (-1, 0)] in unit
coordinates. Saving them as a JSON sidecar lets the offline tracker build the
6-ROI HD mating arena grid without needing auto-target detection.
Output JSON sidecar: data/targets/<video_basename>.json
{
"video_path": "/mnt/.../*.mp4",
"frame_index": <int>,
"reference_points": [[x0, y0], [x1, y1], [x2, y2]],
"order": ["top", "corner", "left"],
"picked_at": "<isoformat>"
}
Keys (in the picker window):
LEFT-CLICK add a point (top → corner → left)
r reset clicks for current video
d skip this video for THIS run only (no JSON written)
u mark this video unusable (FOV wrong etc.); skipped forever
. / , advance / rewind by 25 frames (≈ 1 s @ 25 fps)
] / [ advance / rewind by 5% of the video (~3 min in a 1 h video)
# jump to the middle of the video
enter save the 3 points and move on
q / ESC quit picker
After the 3rd click, the 6 ROI rectangles are drawn over the frame so you
can sanity-check the geometry before pressing ENTER.
With --redo, if a JSON sidecar exists, its points are pre-loaded so you can
nudge them rather than restart from scratch.
Why matplotlib instead of cv2.imshow:
OpenCV's bundled GUI uses Qt, which needs XKeyboard + a fonts directory and
is fragile over SSH X11-forwarding. matplotlib's TkAgg backend uses pure
Tk/X11 and works out of the box on any DISPLAY (and gives free pan/zoom
via the toolbar — useful for clicking small targets precisely).
"""
from __future__ import annotations
import argparse
import datetime as dt
import json
import os
import sys
from pathlib import Path
# Force TkAgg BEFORE importing matplotlib. We override even if MPLBACKEND is
# already set, because the script is unusable with a non-interactive backend.
os.environ["MPLBACKEND"] = "TkAgg"
import cv2 # noqa: E402
import matplotlib # noqa: E402
import matplotlib.pyplot as plt # noqa: E402
import numpy as np # noqa: E402
import pandas as pd # noqa: E402
# matplotlib.backend_bases exposes the cursor identifiers under different
# names depending on version: `Cursors` enum on 3.5+, lowercase `cursors`
# instance on older releases. Both have the same integer attributes.
try:
from matplotlib.backend_bases import Cursors as _Cursors # 3.5+
except ImportError:
try:
from matplotlib.backend_bases import cursors as _Cursors # older
except ImportError:
_Cursors = None
# Verify we ended up on an interactive backend; bail loud (with a concrete
# explanation) if not. matplotlib silently falls back to 'agg' when its
# requested backend can't load, which is hard to debug without help.
_backend = matplotlib.get_backend()
if _backend.lower() in ("agg", "headless", "template", "pdf", "svg", "ps"):
diag = []
try:
import tkinter as _tk
try:
_tk.Tk().destroy()
diag.append("tkinter import + Tk() instantiation: OK")
except Exception as e:
diag.append(f"tkinter imported but Tk() failed: {e!r}")
except Exception as e:
diag.append(f"tkinter import FAILED: {e!r}")
diag.append(" → on Manjaro/Arch, run: sudo pacman -S tk")
print(
f"ERROR: matplotlib loaded the non-interactive backend {_backend!r}.\n"
f" Expected 'TkAgg'. Diagnostic info:\n"
f" DISPLAY = {os.environ.get('DISPLAY')!r}\n"
f" MPLBACKEND = {os.environ.get('MPLBACKEND')!r}\n"
f" matplotlib ver = {matplotlib.__version__}\n"
+ "\n".join(f" {d}" for d in diag),
file=sys.stderr,
)
sys.exit(2)
from config import INVENTORY_CSV, TARGETS_DIR # noqa: E402
from tracking_geometry import compute_roi_polygons # noqa: E402
# Strip default matplotlib keybindings that would conflict with ours.
for k in ("keymap.home", "keymap.save", "keymap.quit", "keymap.fullscreen",
"keymap.pan", "keymap.zoom", "keymap.back", "keymap.forward"):
try:
plt.rcParams[k] = []
except KeyError:
pass
CLICK_LABELS = ("TOP", "CORNER", "LEFT")
CLICK_COLORS = ("red", "lime", "deepskyblue")
def grab_frame(
video_path: Path, frame_idx: int
) -> tuple[np.ndarray, int, int] | None:
"""Return (RGB frame, actual_frame_idx, n_frames) from the video, or None.
Clamps frame_idx to [0, n_frames-1] so callers can step blindly.
"""
cap = cv2.VideoCapture(str(video_path))
if not cap.isOpened():
return None
n = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
if n > 0:
frame_idx = max(0, min(frame_idx, n - 1))
cap.set(cv2.CAP_PROP_POS_FRAMES, frame_idx)
ok, frame = cap.read()
cap.release()
if not ok or frame is None:
return None
return cv2.cvtColor(frame, cv2.COLOR_BGR2RGB), frame_idx, n
def pick_one(
video_path: Path,
frame_idx: int,
status_prefix: str,
initial_points: list[tuple[float, float]] | None = None,
) -> dict | None:
"""Show the picker UI for a single video; return the result dict or None."""
grabbed = grab_frame(video_path, frame_idx)
if grabbed is None:
print(f" ! cannot read {video_path}")
return None
frame, frame_idx, n_frames = grabbed
# Big-step size for ] / [ : 5% of total length, ~3 min in a 1h video.
big_step = max(1, int(round(0.05 * n_frames))) if n_frames > 0 else 250
fig, ax = plt.subplots(figsize=(14, 8))
try:
fig.canvas.manager.set_window_title("pick targets")
except Exception:
pass
# Use a crosshair cursor over the axes so it's obvious where the click
# will land. matplotlib's toolbar resets the cursor to POINTER (arrow) on
# every mouse-move when no tool is active, so we intercept set_cursor:
# whenever it asks for POINTER, we substitute SELECT_REGION (crosshair).
# Tool modes (zoom/pan) keep their native cursors.
if _Cursors is not None:
_orig_set_cursor = fig.canvas.set_cursor
def _set_cursor_with_crosshair(cursor):
if cursor == _Cursors.POINTER:
cursor = _Cursors.SELECT_REGION
return _orig_set_cursor(cursor)
fig.canvas.set_cursor = _set_cursor_with_crosshair
try:
fig.canvas.set_cursor(_Cursors.SELECT_REGION)
except Exception:
pass
else:
# Last-ditch: just set the Tk widget's cursor once and hope the
# toolbar doesn't immediately overwrite it.
try:
fig.canvas.get_tk_widget().config(cursor="tcross")
except Exception:
pass
img_artist = ax.imshow(frame)
ax.set_axis_off()
fig.tight_layout()
state = {
"points": list(initial_points) if initial_points else [],
"action": None, # 'save' | 'skip' | 'quit' | 'unusable'
"frame": frame,
"frame_idx": frame_idx,
"drawn": [], # artists drawn on top of the image
}
def update_title():
nb = len(state["points"])
nxt = (
f"click {CLICK_LABELS[nb]}"
if nb < 3
else "ENTER=save | r=reset d=skip u=unusable q=quit | . , [ ] # = step frame"
)
ax.set_title(
f'{status_prefix} frame {state["frame_idx"]} | {nxt}',
fontsize=10,
)
def redraw_points():
for a in state["drawn"]:
try:
a.remove()
except Exception:
pass
state["drawn"].clear()
for i, (x, y) in enumerate(state["points"]):
color = CLICK_COLORS[i]
label = CLICK_LABELS[i]
(cross,) = ax.plot(x, y, marker="+", color=color, markersize=22, mew=2)
(ring,) = ax.plot(
x, y, marker="o", color=color, markersize=22,
fillstyle="none", mew=2,
)
txt = ax.text(
x + 14, y - 14, label,
color=color, fontsize=10, weight="bold",
)
state["drawn"].extend([cross, ring, txt])
if len(state["points"]) >= 2:
(line1,) = ax.plot(
[state["points"][0][0], state["points"][1][0]],
[state["points"][0][1], state["points"][1][1]],
color="white", linewidth=0.7, alpha=0.6,
)
state["drawn"].append(line1)
if len(state["points"]) == 3:
(line2,) = ax.plot(
[state["points"][1][0], state["points"][2][0]],
[state["points"][1][1], state["points"][2][1]],
color="white", linewidth=0.7, alpha=0.6,
)
state["drawn"].append(line2)
# ROI overlay — draw the 6 computed rectangles on top of the frame
try:
polys = compute_roi_polygons(state["points"])
except Exception as e:
polys = []
print(f" (ROI preview failed: {e})")
for j, poly in enumerate(polys):
# Close the polygon by repeating the first point
xs = list(poly[:, 0]) + [poly[0, 0]]
ys = list(poly[:, 1]) + [poly[0, 1]]
(line,) = ax.plot(
xs, ys, color="yellow", linewidth=1.5, alpha=0.9,
)
state["drawn"].append(line)
cx = float(np.mean(poly[:, 0]))
cy = float(np.mean(poly[:, 1]))
lbl = ax.text(
cx, cy, str(j + 1),
color="yellow", fontsize=14, weight="bold",
ha="center", va="center",
)
state["drawn"].append(lbl)
update_title()
fig.canvas.draw_idle()
def reload_frame(new_idx: int):
grabbed = grab_frame(video_path, new_idx)
if grabbed is None:
return
new_frame, new_idx, _ = grabbed
state["frame"] = new_frame
state["frame_idx"] = new_idx
img_artist.set_data(new_frame)
# Keep clicked targets + ROI overlay in place across frame-stepping —
# press 'r' to clear them explicitly.
redraw_points()
def on_click(event):
if event.inaxes is not ax:
return
if event.button != 1: # left click only
return
if event.xdata is None or event.ydata is None:
return
# Skip clicks fired while the toolbar's pan/zoom is active.
toolbar = getattr(fig.canvas, "toolbar", None)
if toolbar is not None and getattr(toolbar, "mode", ""):
return
x, y = float(event.xdata), float(event.ydata)
if len(state["points"]) < 3:
state["points"].append((x, y))
else:
# 3 points already there — replace the nearest one. Lets the user
# nudge pre-loaded targets in --redo mode, or correct a bad click.
dists = [(x - px) ** 2 + (y - py) ** 2 for px, py in state["points"]]
i_nearest = min(range(3), key=dists.__getitem__)
state["points"][i_nearest] = (x, y)
redraw_points()
def on_key(event):
k = event.key or ""
if k in ("escape", "q"):
state["action"] = "quit"
plt.close(fig)
elif k == "r":
state["points"].clear()
redraw_points()
elif k == "d":
state["action"] = "skip"
plt.close(fig)
elif k == "u":
state["action"] = "unusable"
plt.close(fig)
elif k == "enter":
if len(state["points"]) == 3:
state["action"] = "save"
plt.close(fig)
elif k == ".":
reload_frame(state["frame_idx"] + 25)
elif k == ",":
reload_frame(state["frame_idx"] - 25)
elif k == "]":
reload_frame(state["frame_idx"] + big_step)
elif k == "[":
reload_frame(state["frame_idx"] - big_step)
elif k == "#":
if n_frames > 0:
reload_frame(n_frames // 2)
fig.canvas.mpl_connect("button_press_event", on_click)
fig.canvas.mpl_connect("key_press_event", on_key)
update_title()
plt.show() # blocks until the figure is closed
if state["action"] == "save":
return {
"action": "save",
"frame_idx": state["frame_idx"],
"points": state["points"],
}
if state["action"] == "unusable":
return {"action": "unusable", "frame_idx": state["frame_idx"]}
if state["action"] in ("skip", "quit"):
return {"action": state["action"]}
# Window closed via the WM "X" button — treat as quit so the loop stops
return {"action": "quit"}
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--redo", action="store_true",
help="re-pick videos that already have JSON sidecars",
)
parser.add_argument(
"--frame", type=int, default=125,
help="default frame index to display (default 125 ≈ 5 s @ 25 fps)",
)
parser.add_argument(
"--limit", type=int, default=None,
help="only process the first N videos",
)
args = parser.parse_args()
if not INVENTORY_CSV.exists():
sys.exit(
f"Inventory not found at {INVENTORY_CSV}. "
"Run build_video_inventory.py first."
)
inv = pd.read_csv(INVENTORY_CSV)
todo = inv[inv["in_xlsx"] & ~inv["already_tracked"]].copy()
todo = todo.sort_values(
["session_date", "machine_name", "session_time"]
).reset_index(drop=True)
TARGETS_DIR.mkdir(parents=True, exist_ok=True)
def sidecar_for(mp4_path: str) -> Path:
return TARGETS_DIR / (Path(mp4_path).stem + ".json")
if not args.redo:
todo = todo[
~todo["mp4_path"].apply(lambda p: sidecar_for(p).exists())
].reset_index(drop=True)
if args.limit:
todo = todo.head(args.limit)
n = len(todo)
if n == 0:
print("Nothing to pick. All eligible videos already have target JSONs.")
return
print(
f"Picking targets for {n} videos. "
"Window keys: ENTER=save r=reset d=skip u=unusable q=quit "
".,[]=step frame | pan/zoom via toolbar"
)
saved = skipped = unusable = 0
for i, row in todo.iterrows():
mp4 = Path(row["mp4_path"])
prefix = f"[{i + 1}/{n}] {row['machine_name']} {row['session_datetime']}"
print(f"\n{prefix}")
# If --redo and a JSON sidecar exists, pre-load its points (only for
# regular saves — unusable sidecars are left as-is and shown empty).
initial_points = None
existing = sidecar_for(row["mp4_path"])
if args.redo and existing.exists():
try:
prev = json.loads(existing.read_text())
if not prev.get("unusable") and prev.get("reference_points"):
initial_points = [tuple(p) for p in prev["reference_points"]]
print(f" pre-loaded {len(initial_points)} previous point(s)")
except Exception as e:
print(f" ! could not read previous sidecar: {e}")
result = pick_one(mp4, args.frame, prefix, initial_points=initial_points)
if result is None or result.get("action") == "quit":
print(" quitting picker.")
break
if result["action"] == "skip":
skipped += 1
print(" skipped (no JSON written, will be re-asked next run).")
continue
if result["action"] == "unusable":
try:
reason = input(" reason for marking unusable (Enter to skip): ").strip()
except EOFError:
reason = ""
payload = {
"video_path": str(mp4),
"unusable": True,
"reason": reason,
"marked_at": dt.datetime.now().isoformat(timespec="seconds"),
}
out_path = sidecar_for(row["mp4_path"])
out_path.write_text(json.dumps(payload, indent=2))
unusable += 1
print(f" marked unusable → {out_path.name}")
continue
if result["action"] == "save":
payload = {
"video_path": str(mp4),
"frame_index": int(result["frame_idx"]),
"reference_points": [list(map(int, p)) for p in result["points"]],
"order": ["top", "corner", "left"],
"picked_at": dt.datetime.now().isoformat(timespec="seconds"),
}
out_path = sidecar_for(row["mp4_path"])
out_path.write_text(json.dumps(payload, indent=2))
saved += 1
print(f" saved → {out_path.name}")
remaining = n - saved - skipped - unusable
print(
f"\nDone. saved={saved} unusable={unusable} "
f"skipped(this run)={skipped} remaining={remaining}"
)
if __name__ == "__main__":
main()

218
scripts/track_videos.py Normal file
View file

@ -0,0 +1,218 @@
"""Headless offline tracker.
Reads target JSONs produced by `pick_targets.py`, builds the 6 ROIs of the
HD mating arena from the L-shape reference points, runs ethoscope's
`MultiFlyTracker` against the merged.mp4 file via `MovieVirtualCamera`, and
writes a SQLite DB to `data/tracked/<video_basename>_tracking.db`.
Idempotent: skips videos whose tracking DB already exists (unless --redo).
Usage:
python track_videos.py # process all videos with target JSON
python track_videos.py --redo # re-track even if DB exists
python track_videos.py --jobs 4 # run up to 4 videos in parallel
python track_videos.py --max-duration 1800 # cap each video at 30 min (sec)
"""
from __future__ import annotations
import argparse
import json
import logging
import os
import sys
import traceback
from concurrent.futures import ProcessPoolExecutor, as_completed
from pathlib import Path
import numpy as np
# Import ethoscope from the local source tree (no pip install).
ETHOSCOPE_SRC = Path("/home/gg/Code/ethoscope_project/ethoscope/src/ethoscope")
sys.path.insert(0, str(ETHOSCOPE_SRC))
from config import TARGETS_DIR, TRACKING_OUTPUT_DIR # noqa: E402
from tracking_geometry import HD_FG_DATA, compute_roi_polygons # noqa: E402
def build_rois_from_targets(reference_points):
"""Wrap the shared geometry into ethoscope `ROI` objects."""
from ethoscope.core.roi import ROI
polys = compute_roi_polygons(reference_points)
return [ROI(poly.reshape((1, 4, 2)), idx=i + 1) for i, poly in enumerate(polys)]
def track_one(json_path: Path, output_dir: Path, max_duration: float | None,
redo: bool) -> tuple[str, str]:
"""Track a single video. Returns (status, message). Run in subprocess.
Statuses: "ok", "skip", "error".
"""
# Re-import inside subprocess so each worker has its own ethoscope state.
import sys as _sys
_sys.path.insert(0, str(ETHOSCOPE_SRC))
import cv2
from ethoscope.core.monitor import Monitor
from ethoscope.hardware.input.cameras import MovieVirtualCamera
from ethoscope.io.sqlite import SQLiteResultWriter
from ethoscope.trackers.multi_fly_tracker import MultiFlyTracker
class BGRMovieCamera(MovieVirtualCamera):
"""MovieVirtualCamera variant that keeps BGR frames.
MultiFlyTracker calls cv2.cvtColor(img, COLOR_BGR2GRAY) without checking
whether img is already grayscale, so we must feed it 3-channel input.
"""
def _next_image(self):
ret, frame = self.capture.read()
if not ret or frame is None:
return None
return frame # BGR, untouched
payload = json.loads(json_path.read_text())
if payload.get("unusable"):
reason = payload.get("reason") or "no reason given"
return "skip", f"marked unusable: {reason}"
video_path = Path(payload["video_path"])
if not video_path.exists():
return "error", f"video missing: {video_path}"
out_db = output_dir / f"{video_path.stem}_tracking.db"
if out_db.exists() and not redo:
return "skip", f"DB exists: {out_db.name}"
if out_db.exists():
out_db.unlink()
rois = build_rois_from_targets(payload["reference_points"])
cam_kwargs = {"use_wall_clock": False}
if max_duration is not None:
cam_kwargs["max_duration"] = max_duration
cam = BGRMovieCamera(str(video_path), **cam_kwargs)
metadata = {
"machine_id": payload.get("machine_uuid", "unknown"),
"machine_name": payload.get("machine_name", "unknown"),
"date_time": int(payload.get("session_epoch", 0)),
"frame_width": cam.width,
"frame_height": cam.height,
"version": "offline-tracker-1",
"experimental_info": "{}",
"selected_options": json.dumps({
"tracker": "MultiFlyTracker",
"template": "HD_Mating_Arena_6_ROIS",
"fg_data": HD_FG_DATA,
"maxN": 2,
}),
"hardware_info": "{}",
"reference_points": str([list(map(int, p)) for p in payload["reference_points"]]),
"backup_filename": out_db.name,
"result_writer_type": "SQLite3",
"sqlite_source_path": str(out_db),
}
tracker_data = {
"maxN": 2,
"visualise": False,
"fg_data": HD_FG_DATA,
"adaptive_threshold": True,
"min_fg_threshold": 10,
"max_fg_threshold": 50,
}
db_credentials = {"name": str(out_db)}
rw = SQLiteResultWriter(
db_credentials, rois, metadata=metadata,
make_dam_like_table=False, take_frame_shots=False, erase_old_db=True,
)
monit = Monitor(
cam, MultiFlyTracker, rois,
reference_points=payload["reference_points"],
data=tracker_data,
)
try:
with rw as result_writer:
monit.run(result_writer=result_writer, drawer=None, verbose=False)
except Exception:
return "error", traceback.format_exc(limit=5)
finally:
try:
cam._close()
except Exception:
pass
if not out_db.exists():
return "error", "tracking finished but DB was not created"
return "ok", str(out_db)
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--redo", action="store_true", help="re-track even if DB exists")
parser.add_argument("--jobs", type=int, default=1, help="parallel workers")
parser.add_argument(
"--max-duration", type=float, default=None,
help="cap each video at this many seconds (default: full video)",
)
parser.add_argument("--limit", type=int, default=None, help="process only first N")
parser.add_argument("--video", type=str, default=None,
help="track a single video (mp4 path); requires its target JSON")
args = parser.parse_args()
TRACKING_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
if args.video:
stem = Path(args.video).stem
json_path = TARGETS_DIR / f"{stem}.json"
if not json_path.exists():
sys.exit(f"No target JSON for {args.video}: expected {json_path}")
jsons = [json_path]
else:
jsons = sorted(TARGETS_DIR.glob("*.json"))
if args.limit:
jsons = jsons[: args.limit]
if not jsons:
print("No target JSONs found. Run pick_targets.py first.")
return
print(f"Tracking {len(jsons)} videos (jobs={args.jobs}, redo={args.redo}).")
n_ok = n_skip = n_err = 0
if args.jobs <= 1:
for jp in jsons:
print(f" → {jp.name}", flush=True)
status, msg = track_one(jp, TRACKING_OUTPUT_DIR, args.max_duration, args.redo)
print(f" {status}: {msg.splitlines()[-1] if msg else ''}", flush=True)
n_ok += status == "ok"
n_skip += status == "skip"
n_err += status == "error"
else:
with ProcessPoolExecutor(max_workers=args.jobs) as ex:
futs = {
ex.submit(track_one, jp, TRACKING_OUTPUT_DIR, args.max_duration, args.redo): jp
for jp in jsons
}
for fut in as_completed(futs):
jp = futs[fut]
try:
status, msg = fut.result()
except Exception as e:
status, msg = "error", f"future raised: {e}"
print(f" {jp.name}: {status} — {msg.splitlines()[-1] if msg else ''}",
flush=True)
n_ok += status == "ok"
n_skip += status == "skip"
n_err += status == "error"
print(f"\nDone. ok={n_ok} skipped={n_skip} errors={n_err}")
sys.exit(0 if n_err == 0 else 1)
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
main()

71
scripts/tracking_geometry.py Normal file
View file

@ -0,0 +1,71 @@
"""Shared HD-mating-arena ROI geometry, used by both pick_targets.py
(for live overlay) and track_videos.py (for actual tracking).
Pure numpy + cv2; no ethoscope dependency.
"""
from __future__ import annotations
import itertools
import cv2
import numpy as np
# Layout from
# ethoscope/.../roi_builders/roi_templates/builtin/HD_Mating_Arena_6_ROIS.json
HD_MATING_ARENA = {
"n_rows": 2,
"n_cols": 3,
"top_margin": -0.21,
"bottom_margin": -0.13,
"left_margin": 0.05,
"right_margin": 0.05,
"horizontal_fill": 0.85,
"vertical_fill": 1.3,
}
HD_FG_DATA = {
"sample_size": 400,
"normal_limits": [800, 2000],
"tolerance": 0.8,
}
def compute_roi_polygons(reference_points, layout=HD_MATING_ARENA):
"""Map 3 L-shape reference points to 6 ROI polygons, in the order ROI 1..6.
Reference points must be ordered:
[TOP, CORNER, LEFT]
matching ethoscope's dst_points = [(0, -1), (0, 0), (-1, 0)].
Returns:
list[np.ndarray] # 6 arrays, each shape (4, 2), int32, in image coords
"""
ref = np.asarray(reference_points, dtype=np.float32)
if ref.shape != (3, 2):
raise ValueError(f"reference_points must be 3x2, got shape {ref.shape}")
dst_points = np.array([(0, -1), (0, 0), (-1, 0)], dtype=np.float32)
wrap_mat = cv2.getAffineTransform(dst_points, ref)
n_col = layout["n_cols"]
n_row = layout["n_rows"]
tm, bm = layout["top_margin"], layout["bottom_margin"]
lm, rm = layout["left_margin"], layout["right_margin"]
hf, vf = layout["horizontal_fill"], layout["vertical_fill"]
y_positions = (np.arange(n_row) * 2.0 + 1) * (1 - tm - bm) / (2 * n_row) + tm
x_positions = (np.arange(n_col) * 2.0 + 1) * (1 - lm - rm) / (2 * n_col) + lm
centres = [np.array([x, y]) for x, y in itertools.product(x_positions, y_positions)]
sign_mat = np.array([[-1, -1], [+1, -1], [+1, +1], [-1, +1]])
xy_size = np.array([hf / float(n_col), vf / float(n_row)]) / 2.0
rectangles = [sign_mat * xy_size + c for c in centres]
shift = np.dot(wrap_mat, [1, 1, 0]) - ref[1]
polys = []
for r in rectangles:
r3 = np.append(r, np.zeros((4, 1)), axis=1)
mapped = np.dot(wrap_mat, r3.T).T - shift
polys.append(mapped.astype(np.int32))
return polys

62
tasks/todo.md
View file

@ -51,6 +51,68 @@ See `docs/bimodal_hypothesis.md` for detailed methodology.
- [ ] Consider converting pixel distances to physical units (need calibration)
- [ ] The second notebook (`flies_analysis.ipynb`) re-runs from DB extraction - consider deprecating
## Phase: Offline Tracking of 2024 Video Backlog (added 2026-04-27)
### Recap
Tracked so far (5 sessions, all from 2025-07-15, machines 076/145/268). The DBs in
`data/raw/` use tracker `ConstrainedMultiFlyTracker` and template
`HD_Mating_Arena_6_ROIS.json` (2 flies × 6 ROIs per video).
The metadata file `../all_video_info_merged.xlsx` indexes a different set of
experiments: 7 dates from 2024-09-17 → 2024-10-21, 16 ethoscope machines,
63 unique (date, machine) sessions = 484 ROI-rows. **None of the already-tracked
sessions are in this xlsx — these are fresh recordings to track.**
Inventory: see `data/metadata/video_inventory.csv` (built by
`scripts/build_video_inventory.py`).
- 1163 video sessions on disk under `/mnt/ethoscope_data/videos/`
- 63/63 xlsx (date, machine) sessions have video on disk
- 129 video instances need tracking (some (date, machine) have 2-4 recordings/day)
### Plan
The HD-mating-arena videos have no auto-detectable targets — the user must
manually click 3 reference points (L-shape: top, corner, left) per video. Once
all targets are picked, tracking can run in the background.
- [x] **Step 1 — Inventory**: `scripts/build_video_inventory.py` →
`data/metadata/video_inventory.csv`. 63 (date,machine) sessions match
the xlsx, all videos found, 129 video instances need tracking.
- [x] **Step 2 — Manual target picker**: `scripts/pick_targets.py`. Loops over
videos with `in_xlsx & ~already_tracked & no JSON yet`; per video, shows
a representative frame, captures 3 clicks (top, corner, left), saves
`data/targets/<video_basename>.json`. Skips videos already done.
- [x] **Step 3 — Background tracker**: `scripts/track_videos.py`. Reads target
JSONs, builds 6 ROIs from the HD-mating-arena geometry, runs
`MovieVirtualCamera` + `MultiFlyTracker` + `SQLiteResultWriter`, writes
`data/tracked/<basename>_tracking.db`. Idempotent. Smoke-tested
end-to-end: 90s of video → ~3000 rows/ROI, areas in 800-2000 band.
- [x] **Step 4 — Tracking deps**: `requirements-tracking.txt`.
### Still TODO
- [ ] User to run `pick_targets.py` (interactive — needs DISPLAY) on the 129
pending videos.
- [ ] Run `track_videos.py --jobs 4` against the resulting JSONs.
- [ ] (Optional) `auto_detect_targets.py` exists as a fallback for videos that
DO have visible targets (saves clicks). Confirmed not useful on the
2025-07-15 batch — these arenas don't have black target dots — but worth
trying on 2024 batches before falling back to manual.
- [ ] Decide what to do with the 4 (date, machine) sessions that have 3-4
recordings/day instead of 2 (e.g. ETHOSCOPE_086 on 2024-09-17 has 4).
One of them is at lower resolution (1280x960) — likely an aborted take.
### Open questions / risks
- Some (date, machine) combos have 3-4 recordings (e.g. ETHOSCOPE_086 on
2024-09-17). Need to figure out which is the real "test" video vs aborted
takes — possibly use video duration or filename pattern.
- One mismatched-resolution file: `1280x960@25fps-20q` instead of
`1920x1088@25fps-28q` — flag for inspection.
- The original `ConstrainedMultiFlyTracker` is no longer in the ethoscope repo;
`MultiFlyTracker` is its likely successor. Validate output schema matches
what the existing analysis pipeline expects (`load_roi_data.py`, etc.).
## Discovered During Work
(Add new items here as they come up during analysis)