cupido/data/processed/README.md

Processed Data

CSVs derived from the tracking DBs (/mnt/data/projects/cupido/tracked/) and the merged TSV (/mnt/data/projects/cupido/all_video_info_merged.tsv). All files are gitignored and regenerable.

Files and Regeneration

File	Description	Generated By
distances.csv	Per-frame inter-fly distances for every (date, machine, ROI, session). Includes metadata columns to filter trained vs naïve, training phase, species, etc.	scripts/calculate_distances.py
*_distances_aligned.csv	(legacy, 2025-07-15 only) distances aligned to barrier opening	notebooks/flies_analysis*.ipynb
*_tracked.csv	(legacy) identity-tracked fly positions	notebooks/flies_analysis_simple.ipynb
*_max_velocity.csv	(legacy) max velocity over 10 s windows	notebooks/flies_analysis_simple.ipynb

Loading the data

import sys
sys.path.insert(0, "../scripts")
from load_roi_data import load_roi_data

data = load_roi_data()              # full batch as one DataFrame
# Or filter the metadata first:
import pandas as pd
tsv = pd.read_csv("/mnt/data/projects/cupido/all_video_info_merged.tsv", sep="\t")
data = load_roi_data(tsv[tsv.species.str.contains("Melanogaster")])

The returned DataFrame has columns: id, t, x, y, w, h, phi, is_inferred, has_interacted, session, ROI, date, machine_name, species, male, training_date_time, testing_date_time, training_length_hr, consolidation_length_hr, memory, age.

session is "training" or "testing"; male is "trained" or "naive" (canonical — variants like "naïve" and "niave" are normalized at the TSV-export step).

The TSV also has a per-row boolean include column (default True). Flip it to False to drop a noisy / unusable fly+session from analysis without deleting the row. load_roi_data honors this flag automatically.

The shared TSV at /mnt/data/projects/cupido/all_video_info_merged.tsv is read-only (the data volume is mounted :ro in the container) so each user keeps their own edits in a personal copy inside the repo's data/metadata/ folder. That path is gitignored. Notebooks pick up the personal copy automatically if it exists; otherwise they fall back to the shared master. To start your personal copy, run once in a terminal:

cp /mnt/data/projects/cupido/all_video_info_merged.tsv ~/cupido/data/metadata/

Column Reference (distances.csv)

• date, machine_name, ROI, session: identifies one fly trajectory
• t: time in ms within that session
• distance: Euclidean distance between the two flies in pixels
• n_flies: number of fly detections at this frame (1 or 2)
• area_fly1, area_fly2: bounding-box areas (w * h) in pixels²
• male: trained or naive (carried from the xlsx; normalized)
• species, memory, age: experimental metadata