lab/cupido
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
cupido/data/processed
History
Giorgio Gilestro 9f3ee24a23 Add per-row include flag to TSV; expand flies_analysis_simple narrative 
- export_video_db_index.py now writes a boolean `include` column
  (default True). Flip it to False to drop a noisy/unusable row from
  analysis without deleting it.
- load_roi_data filters on `include` automatically (back-compat:
  missing column = load everything).
- flies_analysis_simple.ipynb section headers now explain *why* each
  step exists (barrier alignment, body-area baseline, merged-blob
  heuristic, Hungarian identity tracking) rather than just naming
  the step.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-01 09:09:59 +01:00
..
README.md Add per-row include flag to TSV; expand flies_analysis_simple narrative 2026-05-01 09:09:59 +01:00

README.md

Processed Data

CSVs derived from the tracking DBs (/mnt/data/projects/cupido/tracked/) and the merged TSV (/mnt/data/projects/cupido/all_video_info_merged.tsv). All files are gitignored and regenerable.

Files and Regeneration

File Description Generated By
distances.csv Per-frame inter-fly distances for every (date, machine, ROI, session). Includes metadata columns to filter trained vs naïve, training phase, species, etc. scripts/calculate_distances.py
*_distances_aligned.csv (legacy, 2025-07-15 only) distances aligned to barrier opening notebooks/flies_analysis*.ipynb
*_tracked.csv (legacy) identity-tracked fly positions notebooks/flies_analysis_simple.ipynb
*_max_velocity.csv (legacy) max velocity over 10 s windows notebooks/flies_analysis_simple.ipynb

Loading the data

import sys
sys.path.insert(0, "../scripts")
from load_roi_data import load_roi_data

data = load_roi_data()              # full batch as one DataFrame
# Or filter the metadata first:
import pandas as pd
tsv = pd.read_csv("/mnt/data/projects/cupido/all_video_info_merged.tsv", sep="\t")
data = load_roi_data(tsv[tsv.species.str.contains("Melanogaster")])

The returned DataFrame has columns: id, t, x, y, w, h, phi, is_inferred, has_interacted, session, ROI, date, machine_name, species, male, training_date_time, testing_date_time, training_length_hr, consolidation_length_hr, memory, age.

session is "training" or "testing"; male is "trained" or "naive" (canonical — variants like "naïve" and "niave" are normalized at the TSV-export step).

The TSV also has a per-row boolean include column (default True). Flip it to False to drop a noisy / unusable fly+session from analysis without deleting the row. load_roi_data honors this flag automatically.

Column Reference (distances.csv)

  • date, machine_name, ROI, session: identifies one fly trajectory
  • t: time in ms within that session
  • distance: Euclidean distance between the two flies in pixels
  • n_flies: number of fly detections at this frame (1 or 2)
  • area_fly1, area_fly2: bounding-box areas (w * h) in pixels²
  • male: trained or naive (carried from the xlsx; normalized)
  • species, memory, age: experimental metadata