Giorgio Gilestro
231c7a437f
Notebooks now use Path.home() / "cupido" for the repo root (works for any user inside the JupyterLab container), and the offline-tracking scripts read the ethoscope source-tree location from the new ETHOSCOPE_SRC config constant — defaulting to ~/Code/ethoscope_project/... and overridable via the ETHOSCOPE_SRC environment variable. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
137 lines
5.4 KiB
Markdown
137 lines
5.4 KiB
Markdown
# Cupido: Drosophila Social Interaction Tracking
|
|
|
|
Behavioral analysis of trained vs untrained *Drosophila melanogaster* in a barrier-opening social interaction assay. Part of the Cupido project studying learned social behaviors.
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Clone the repository
|
|
git clone ssh://git@git.lab.gilest.ro:222/lab/cupido.git
|
|
cd cupido
|
|
|
|
# Create virtual environment
|
|
python -m venv .venv
|
|
source .venv/bin/activate
|
|
pip install -r requirements.txt
|
|
|
|
# Get the data files (not in git - ask lab for copies)
|
|
# Place .db files in data/raw/
|
|
# Place large .csv files in data/processed/
|
|
|
|
# Run the main analysis notebook
|
|
jupyter notebook notebooks/flies_analysis_simple.ipynb
|
|
```
|
|
|
|
## Project Overview
|
|
|
|
### The Experiment
|
|
|
|
Pairs of flies are placed in chambers (ROIs) separated by a physical barrier. After a configurable delay, the barrier is removed, allowing flies to interact. We track the distance between flies over time to compare social approach behavior between trained (socially experienced) and untrained (naive) groups.
|
|
|
|
- **3 ethoscope machines**, 5 recording sessions, 6 ROIs each = 30 ROIs with data
|
|
- **18 trained ROIs, 18 untrained ROIs** (6 from Machine 139 have no tracking data)
|
|
- See `docs/experimental_design.md` for full details
|
|
|
|
### Current Findings
|
|
|
|
Aggregate analysis shows statistically significant but **tiny** differences:
|
|
- Post-opening distance: Cohen's d = 0.09 (96% distribution overlap)
|
|
- Max velocity (50-200s): Cohen's d = 0.14
|
|
|
|
These effect sizes are inflated by pseudoreplication (230K data points from 18 independent ROIs per group).
|
|
|
|
### Next Direction: Bimodal Hypothesis
|
|
|
|
The key insight: not all "trained" flies may have actually learned. The trained group likely contains **true learners** (showing distinct behavior) and **non-learners** (indistinguishable from untrained). Testing this requires per-ROI analysis and bimodality testing.
|
|
|
|
**Read `docs/bimodal_hypothesis.md` for the detailed analysis plan and code sketches.**
|
|
|
|
## Offline Tracking Pipeline (added Apr 2026)
|
|
|
|
For tracking new videos that have **no auto-detectable targets**, the pipeline
|
|
is split in two stages so you can sit at the screen and click for an hour, then
|
|
let the tracker grind through overnight.
|
|
|
|
```bash
|
|
# extra deps (set ETHOSCOPE_SRC env var if your ethoscope clone isn't at ~/Code/ethoscope_project/...)
|
|
pip install -r requirements-tracking.txt
|
|
|
|
# 1) build the inventory (xlsx ↔ /mnt/ethoscope_data/videos/)
|
|
python scripts/build_video_inventory.py
|
|
|
|
# 2) interactive: click TOP, CORNER, LEFT on each video (one frame per video)
|
|
python scripts/pick_targets.py # process all not-yet-picked
|
|
python scripts/pick_targets.py --redo # re-pick already-picked videos
|
|
# keys: r=reset n=skip f=jump frame q/ESC=quit ENTER=save
|
|
|
|
# 3) batch tracking (idempotent, can run in background)
|
|
python scripts/track_videos.py --jobs 4 # parallel
|
|
# output → /mnt/data/projects/cupido/tracked/*_tracking.db (SQLite, same schema as data/raw/)
|
|
```
|
|
|
|
See `tasks/todo.md` "Offline Tracking" section for the full plan, and
|
|
`data/metadata/video_inventory.csv` for the list of videos to process.
|
|
|
|
## Folder Structure
|
|
|
|
```
|
|
tracking/
|
|
├── README.md # This file
|
|
├── PLANNING.md # Architecture & conventions
|
|
├── requirements.txt # Python dependencies
|
|
├── data/
|
|
│ ├── raw/ # SQLite tracking databases (gitignored)
|
|
│ ├── metadata/ # Experiment metadata CSVs
|
|
│ └── processed/ # Generated analysis CSVs (gitignored)
|
|
├── scripts/ # Python analysis scripts
|
|
│ ├── config.py # Shared path constants
|
|
│ ├── load_roi_data.py # Extract data from DBs
|
|
│ ├── calculate_distances.py
|
|
│ ├── analyze_distances.py
|
|
│ ├── statistical_tests.py
|
|
│ ├── ml_classification.py
|
|
│ └── plot_*.py # Plotting scripts
|
|
├── notebooks/ # Jupyter notebooks
|
|
│ ├── flies_analysis_simple.ipynb # Main analysis (use this one)
|
|
│ └── flies_analysis.ipynb # Full pipeline from DB extraction
|
|
├── figures/ # Generated plots (gitignored)
|
|
├── docs/ # Scientific documentation
|
|
│ ├── analysis_summary.md
|
|
│ ├── bimodal_hypothesis.md
|
|
│ └── experimental_design.md
|
|
└── tasks/
|
|
├── todo.md # Task checklist
|
|
└── lessons.md # Pitfalls & patterns
|
|
```
|
|
|
|
## Data Pipeline
|
|
|
|
```
|
|
SQLite DBs (data/raw/)
|
|
│
|
|
▼ load_roi_data.py / notebook step 1
|
|
ROI CSVs (data/processed/*_roi_data.csv)
|
|
│
|
|
▼ notebook steps 2-4
|
|
Aligned Distance CSVs (data/processed/*_distances_aligned.csv)
|
|
│
|
|
├──▶ Plots (figures/)
|
|
├──▶ Statistical tests
|
|
└──▶ Identity tracking → Velocity analysis
|
|
```
|
|
|
|
## Key Files
|
|
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `notebooks/flies_analysis_simple.ipynb` | **Start here** - main analysis notebook |
|
|
| `docs/bimodal_hypothesis.md` | **Read next** - the new analysis direction |
|
|
| `data/metadata/2025_07_15_metadata_fixed.csv` | ROI-to-group mapping |
|
|
| `data/metadata/2025_07_15_barrier_opening.csv` | Barrier opening times per machine |
|
|
| `scripts/config.py` | Shared path constants for all scripts |
|
|
|
|
## Requirements
|
|
|
|
- Python 3.10+
|
|
- See `requirements.txt` for packages (numpy, pandas, matplotlib, seaborn, scipy, scikit-learn, jupyter)
|
|
- Large data files (~370MB CSVs + ~33MB DBs) must be obtained separately
|