lab/cupido
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Unify analysis pipeline around the TSV; move tracked DBs out of cloud sync

Browse source 
- Tracked DBs now live at /mnt/data/projects/cupido/tracked/ (out of
  ownCloud to avoid sync conflicts and bandwidth churn). config.py
  TRACKING_OUTPUT_DIR points there; the docker-compose for ethoscope-lab
  mounts it world-readable for JupyterHub users.
- New scripts/export_video_db_index.py joins all_video_info_merged.xlsx
  with the video inventory and the on-disk DBs, producing a TSV that has
  one row per fly/ROI plus training/testing video and DB paths. Handles
  approximate xlsx times, cross-day training/testing, the 12 AM/PM
  ambiguity, and date typos.
- scripts/load_roi_data.py rewritten as a TSV-driven loader returning a
  single DataFrame with session and metadata columns. calculate_distances
  and the two flies_analysis notebooks migrated to use it; downstream
  trained/naive splits remain available via simple equality filters.
- Metadata vocabulary canonicalized: {naïve, niave, untrained, test} all
  resolve to {trained, naive}. Normalization happens at the TSV-export
  boundary (idempotent); the xlsx and the 2025-07-15 legacy CSV were
  edited in place to remove the worst variants.
- scripts/monitor_tracking.py rate calculation fixed: with N parallel
  workers, completions arrive in bursts; the old formula divided by burst
  width and reported nonsense rates. Now uses a 6 h window denominator.
- scripts/track_videos.py: BGRMovieCamera retries cv2.read on transient
  NFS hiccups and a post-tracking completeness gate (≥ 90 % of expected
  duration via MAX(t) across all 6 ROIs) deletes silent partial DBs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Giorgio Gilestro 2026-04-30 15:20:14 +01:00
parent e4da7691d5
commit f60a9d0530
13 changed files with 569 additions and 237 deletions
Download patch file Download diff file Expand all files Collapse all files

58
data/processed/README.md
View file

@ -1,39 +1,47 @@
# Processed Data
Large CSV files generated from the analysis pipeline. All files are gitignored (~370MB total) and can be regenerated.
CSVs derived from the tracking DBs (`/mnt/data/projects/cupido/tracked/`)
and the merged TSV (`../../all_video_info_merged.tsv`). All files are
gitignored and regenerable.
## Files and Regeneration
| File | Description | Generated By |
|------|-------------|--------------|
| `trained_roi_data.csv` | Raw tracking data for trained ROIs | `scripts/load_roi_data.py` or notebook step 1 |
| `untrained_roi_data.csv` | Raw tracking data for untrained ROIs | `scripts/load_roi_data.py` or notebook step 1 |
| `trained_distances.csv` | Pairwise distances (unaligned) | `scripts/calculate_distances.py` |
| `untrained_distances.csv` | Pairwise distances (unaligned) | `scripts/calculate_distances.py` |
| `trained_distances_aligned.csv` | Distances aligned to barrier opening | Notebook step 4 |
| `untrained_distances_aligned.csv` | Distances aligned to barrier opening | Notebook step 4 |
| `trained_tracked.csv` | Identity-tracked fly positions | Notebook step 7 |
| `untrained_tracked.csv` | Identity-tracked fly positions | Notebook step 7 |
| `trained_max_velocity.csv` | Max velocity over 10s windows | Notebook step 7 |
| `untrained_max_velocity.csv` | Max velocity over 10s windows | Notebook step 7 |
| `distances.csv` | Per-frame inter-fly distances for every (date, machine, ROI, session). Includes metadata columns to filter trained vs naïve, training phase, species, etc. | `scripts/calculate_distances.py` |
| `*_distances_aligned.csv` | (legacy, 2025-07-15 only) distances aligned to barrier opening | `notebooks/flies_analysis*.ipynb` |
| `*_tracked.csv` | (legacy) identity-tracked fly positions | `notebooks/flies_analysis_simple.ipynb` |
| `*_max_velocity.csv` | (legacy) max velocity over 10 s windows | `notebooks/flies_analysis_simple.ipynb` |
## To Regenerate All Data
## Loading the data
Run the full notebook `notebooks/flies_analysis_simple.ipynb` with:
```python
recalculate_distances = True
recalculate_tracking = True
import sys
sys.path.insert(0, "../scripts")
from load_roi_data import load_roi_data
data = load_roi_data() # full batch as one DataFrame
# Or filter the metadata first:
import pandas as pd
tsv = pd.read_csv("../../all_video_info_merged.tsv", sep="\t")
data = load_roi_data(tsv[tsv.species.str.contains("Melanogaster")])
```
**Warning**: Identity tracking and velocity calculations take significant time (~30+ minutes).
The returned DataFrame has columns:
`id, t, x, y, w, h, phi, is_inferred, has_interacted, session, ROI, date,
machine_name, species, male, training_date_time, testing_date_time,
training_length_hr, consolidation_length_hr, memory, age`.
## Column Reference
`session` is `"training"` or `"testing"`; `male` is `"trained"` or
`"naive"` (canonical — variants like `"naïve"` and `"niave"` are normalized
at the TSV-export step).
### Distance CSVs (`*_distances_aligned.csv`)
- `machine_name`: Ethoscope machine ID (string)
- `ROI`: ROI number (1-6)
- `aligned_time`: Time in ms relative to barrier opening (0 = opening)
- `distance`: Euclidean distance between flies in pixels
- `n_flies`: Number of flies detected at this time point
- `area_fly1`, `area_fly2`: Bounding box areas (w*h) in pixels^2
- `group`: "trained" or "untrained"
## Column Reference (`distances.csv`)
- `date`, `machine_name`, `ROI`, `session`: identifies one fly trajectory
- `t`: time in ms within that session
- `distance`: Euclidean distance between the two flies in pixels
- `n_flies`: number of fly detections at this frame (1 or 2)
- `area_fly1`, `area_fly2`: bounding-box areas (`w * h`) in pixels²
- `male`: `trained` or `naive` (carried from the xlsx; normalized)
- `species`, `memory`, `age`: experimental metadata