Giorgio Gilestro
f60a9d0530
- Tracked DBs now live at /mnt/data/projects/cupido/tracked/ (out of
ownCloud to avoid sync conflicts and bandwidth churn). config.py
TRACKING_OUTPUT_DIR points there; the docker-compose for ethoscope-lab
mounts it world-readable for JupyterHub users.
- New scripts/export_video_db_index.py joins all_video_info_merged.xlsx
with the video inventory and the on-disk DBs, producing a TSV that has
one row per fly/ROI plus training/testing video and DB paths. Handles
approximate xlsx times, cross-day training/testing, the 12 AM/PM
ambiguity, and date typos.
- scripts/load_roi_data.py rewritten as a TSV-driven loader returning a
single DataFrame with session and metadata columns. calculate_distances
and the two flies_analysis notebooks migrated to use it; downstream
trained/naive splits remain available via simple equality filters.
- Metadata vocabulary canonicalized: {naïve, niave, untrained, test} all
resolve to {trained, naive}. Normalization happens at the TSV-export
boundary (idempotent); the xlsx and the 2025-07-15 legacy CSV were
edited in place to remove the worst variants.
- scripts/monitor_tracking.py rate calculation fixed: with N parallel
workers, completions arrive in bursts; the old formula divided by burst
width and reported nonsense rates. Now uses a 6 h window denominator.
- scripts/track_videos.py: BGRMovieCamera retries cv2.read on transient
NFS hiccups and a post-tracking completeness gate (≥ 90 % of expected
duration via MAX(t) across all 6 ROIs) deletes silent partial DBs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2 KiB
2 KiB
Processed Data
CSVs derived from the tracking DBs (/mnt/data/projects/cupido/tracked/)
and the merged TSV (../../all_video_info_merged.tsv). All files are
gitignored and regenerable.
Files and Regeneration
| File | Description | Generated By |
|---|---|---|
distances.csv |
Per-frame inter-fly distances for every (date, machine, ROI, session). Includes metadata columns to filter trained vs naïve, training phase, species, etc. | scripts/calculate_distances.py |
*_distances_aligned.csv |
(legacy, 2025-07-15 only) distances aligned to barrier opening | notebooks/flies_analysis*.ipynb |
*_tracked.csv |
(legacy) identity-tracked fly positions | notebooks/flies_analysis_simple.ipynb |
*_max_velocity.csv |
(legacy) max velocity over 10 s windows | notebooks/flies_analysis_simple.ipynb |
Loading the data
import sys
sys.path.insert(0, "../scripts")
from load_roi_data import load_roi_data
data = load_roi_data() # full batch as one DataFrame
# Or filter the metadata first:
import pandas as pd
tsv = pd.read_csv("../../all_video_info_merged.tsv", sep="\t")
data = load_roi_data(tsv[tsv.species.str.contains("Melanogaster")])
The returned DataFrame has columns:
id, t, x, y, w, h, phi, is_inferred, has_interacted, session, ROI, date, machine_name, species, male, training_date_time, testing_date_time, training_length_hr, consolidation_length_hr, memory, age.
session is "training" or "testing"; male is "trained" or
"naive" (canonical — variants like "naïve" and "niave" are normalized
at the TSV-export step).
Column Reference (distances.csv)
date,machine_name,ROI,session: identifies one fly trajectoryt: time in ms within that sessiondistance: Euclidean distance between the two flies in pixelsn_flies: number of fly detections at this frame (1 or 2)area_fly1,area_fly2: bounding-box areas (w * h) in pixels²male:trainedornaive(carried from the xlsx; normalized)species,memory,age: experimental metadata