- Cell 6: raise a clear ValueError if no loaded machine has a barrier-
opening entry, listing what's loaded vs what's annotated. Previously
alignment quietly produced empty DataFrames and we crashed five cells
later with a cryptic KeyError on 'distance'.
- Cell 10: validate the cached CSVs (presence + expected columns +
non-empty) before using them; fall through to recomputation if not.
Skip writing the cache when results are empty so future runs don't
pick up a 1-byte placeholder.
- Cell 3: derive a 'group' column from 'male' so downstream helpers
that reference fly['group'] still work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Personal copy of all_video_info_merged.tsv now lives at
~/cupido/data/metadata/all_video_info_merged.tsv (gitignored) instead
of ~/cupido_metadata.tsv. That sits next to the other small metadata
CSVs (barrier_opening, etc.) — the natural home for it. Updated all
five notebooks and processed/README accordingly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The shared TSV at /mnt/data/projects/cupido/ is read-only inside the
container, so users who want to customize the `include` column (or any
metadata) need a personal copy. Notebooks now check for
~/cupido_metadata.tsv first and fall back to the shared master if it
doesn't exist. Each user keeps their own edits without stepping on
anyone else's analysis.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Deleted the 5 stale pre-pipeline tracking DBs and the data/raw/ directory.
Dropped DATA_RAW from config.py; build_video_inventory now scans
TRACKING_OUTPUT_DIR for already-tracked sessions. Notebooks no longer
import DATA_RAW. README, PLANNING and todo updated to reflect that the
repo holds only code + small curated metadata, never bulky DBs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- export_video_db_index.py now writes a boolean `include` column
(default True). Flip it to False to drop a noisy/unusable row from
analysis without deleting it.
- load_roi_data filters on `include` automatically (back-compat:
missing column = load everything).
- flies_analysis_simple.ipynb section headers now explain *why* each
step exists (barrier alignment, body-area baseline, merged-blob
heuristic, Hungarian identity tracking) rather than just naming
the step.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Define METADATA_TSV and TRACKED_DBS up front in cell 1, assert they
exist before doing anything else, and pass the loaded metadata to
load_roi_data() explicitly. Surfaces path problems immediately with a
readable message instead of failing deep inside the loader.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Notebooks now use Path.home() / "cupido" for the repo root (works for
any user inside the JupyterLab container), and the offline-tracking
scripts read the ethoscope source-tree location from the new
ETHOSCOPE_SRC config constant — defaulting to ~/Code/ethoscope_project/...
and overridable via the ETHOSCOPE_SRC environment variable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the cryptic Path("..").resolve() walk-up with explicit DATA_DIR
and REPO_ROOT constants, then import the rest of the path constants
(DATA_RAW, DATA_METADATA, DATA_PROCESSED, FIGURES) directly from
scripts/config.py — single source of truth, easier to read for students.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Consolidates everything bulky (tracking DBs, targets, metadata
spreadsheet) under a single DATA_VOLUME root outside the ownCloud-synced
repo. Notebooks now use a visible DATA_DIR = Path(...) idiom rather than
walking up the filesystem with PROJECT_ROOT.parent — easier for students
with no Python background to follow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Four guided notebooks under notebooks/getting_started/ aimed at someone
new to Python and data science. The series progresses: project orientation
→ Python/pandas crash course → exploring one tracking DB → first
trained-vs-naive comparison using load_roi_data + Mann-Whitney U.
Each notebook leans heavily on markdown explanations, includes exercises
with empty cells, and links out to canonical references (JupyterLab,
official Python tutorial, pandas 10-min guide, Wikipedia for stats
concepts).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Tracked DBs now live at /mnt/data/projects/cupido/tracked/ (out of
ownCloud to avoid sync conflicts and bandwidth churn). config.py
TRACKING_OUTPUT_DIR points there; the docker-compose for ethoscope-lab
mounts it world-readable for JupyterHub users.
- New scripts/export_video_db_index.py joins all_video_info_merged.xlsx
with the video inventory and the on-disk DBs, producing a TSV that has
one row per fly/ROI plus training/testing video and DB paths. Handles
approximate xlsx times, cross-day training/testing, the 12 AM/PM
ambiguity, and date typos.
- scripts/load_roi_data.py rewritten as a TSV-driven loader returning a
single DataFrame with session and metadata columns. calculate_distances
and the two flies_analysis notebooks migrated to use it; downstream
trained/naive splits remain available via simple equality filters.
- Metadata vocabulary canonicalized: {naïve, niave, untrained, test} all
resolve to {trained, naive}. Normalization happens at the TSV-export
boundary (idempotent); the xlsx and the 2025-07-15 legacy CSV were
edited in place to remove the worst variants.
- scripts/monitor_tracking.py rate calculation fixed: with N parallel
workers, completions arrive in bursts; the old formula divided by burst
width and reported nonsense rates. Now uses a 6 h window denominator.
- scripts/track_videos.py: BGRMovieCamera retries cv2.read on transient
NFS hiccups and a post-tracking completeness gate (≥ 90 % of expected
duration via MAX(t) across all 6 ROIs) deletes silent partial DBs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reorganized flat 41-file directory into structured layout with:
- scripts/ for Python analysis code with shared config.py
- notebooks/ for Jupyter analysis notebooks
- data/ split into raw/, metadata/, processed/
- docs/ with analysis summary, experimental design, and bimodal hypothesis tutorial
- tasks/ with todo checklist and lessons learned
- Comprehensive README, PLANNING.md, and .gitignore
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
