Giorgio Gilestro
ac3b8c13f0
Personal copy of all_video_info_merged.tsv now lives at ~/cupido/data/metadata/all_video_info_merged.tsv (gitignored) instead of ~/cupido_metadata.tsv. That sits next to the other small metadata CSVs (barrier_opening, etc.) — the natural home for it. Updated all five notebooks and processed/README accordingly. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
63 lines
2.8 KiB
Markdown
63 lines
2.8 KiB
Markdown
# Processed Data
|
|
|
|
CSVs derived from the tracking DBs (`/mnt/data/projects/cupido/tracked/`)
|
|
and the merged TSV (`/mnt/data/projects/cupido/all_video_info_merged.tsv`).
|
|
All files are gitignored and regenerable.
|
|
|
|
## Files and Regeneration
|
|
|
|
| File | Description | Generated By |
|
|
|------|-------------|--------------|
|
|
| `distances.csv` | Per-frame inter-fly distances for every (date, machine, ROI, session). Includes metadata columns to filter trained vs naïve, training phase, species, etc. | `scripts/calculate_distances.py` |
|
|
| `*_distances_aligned.csv` | (legacy, 2025-07-15 only) distances aligned to barrier opening | `notebooks/flies_analysis*.ipynb` |
|
|
| `*_tracked.csv` | (legacy) identity-tracked fly positions | `notebooks/flies_analysis_simple.ipynb` |
|
|
| `*_max_velocity.csv` | (legacy) max velocity over 10 s windows | `notebooks/flies_analysis_simple.ipynb` |
|
|
|
|
## Loading the data
|
|
|
|
```python
|
|
import sys
|
|
sys.path.insert(0, "../scripts")
|
|
from load_roi_data import load_roi_data
|
|
|
|
data = load_roi_data() # full batch as one DataFrame
|
|
# Or filter the metadata first:
|
|
import pandas as pd
|
|
tsv = pd.read_csv("/mnt/data/projects/cupido/all_video_info_merged.tsv", sep="\t")
|
|
data = load_roi_data(tsv[tsv.species.str.contains("Melanogaster")])
|
|
```
|
|
|
|
The returned DataFrame has columns:
|
|
`id, t, x, y, w, h, phi, is_inferred, has_interacted, session, ROI, date,
|
|
machine_name, species, male, training_date_time, testing_date_time,
|
|
training_length_hr, consolidation_length_hr, memory, age`.
|
|
|
|
`session` is `"training"` or `"testing"`; `male` is `"trained"` or
|
|
`"naive"` (canonical — variants like `"naïve"` and `"niave"` are normalized
|
|
at the TSV-export step).
|
|
|
|
The TSV also has a per-row boolean `include` column (default `True`).
|
|
Flip it to `False` to drop a noisy / unusable fly+session from analysis
|
|
without deleting the row. `load_roi_data` honors this flag automatically.
|
|
|
|
The shared TSV at `/mnt/data/projects/cupido/all_video_info_merged.tsv`
|
|
is **read-only** (the data volume is mounted `:ro` in the container) so
|
|
each user keeps their own edits in a personal copy inside the repo's
|
|
`data/metadata/` folder. That path is gitignored. Notebooks pick up
|
|
the personal copy automatically if it exists; otherwise they fall back
|
|
to the shared master. To start your personal copy, run once in a
|
|
terminal:
|
|
|
|
```bash
|
|
cp /mnt/data/projects/cupido/all_video_info_merged.tsv ~/cupido/data/metadata/
|
|
```
|
|
|
|
## Column Reference (`distances.csv`)
|
|
|
|
- `date`, `machine_name`, `ROI`, `session`: identifies one fly trajectory
|
|
- `t`: time in ms within that session
|
|
- `distance`: Euclidean distance between the two flies in pixels
|
|
- `n_flies`: number of fly detections at this frame (1 or 2)
|
|
- `area_fly1`, `area_fly2`: bounding-box areas (`w * h`) in pixels²
|
|
- `male`: `trained` or `naive` (carried from the xlsx; normalized)
|
|
- `species`, `memory`, `age`: experimental metadata
|