Five existing migrations used op.alter_column / op.create_unique_constraint /
op.drop_constraint / op.create_foreign_key directly on the users + quotes +
quotes_daily tables. SQLite has no native support for those operations and
requires Alembic's batch_alter_table copy-and-rename workaround.
This wasn't noticed until now because the test suite uses
Base.metadata.create_all to materialise schema, not the migration chain
itself; and prod is MariaDB. But running `alembic upgrade head` against
a fresh SQLite database (developer onboarding, CI smoke tests, the
test container's own bootstrap) would fail at 0005.
Fixes:
- alembic/env.py: set render_as_batch=True when the dialect is SQLite.
This auto-wraps any future autogenerated migration but doesn't
retroactively rewrite existing op.* calls.
- 0005 (widen quotes.symbol), 0013 (referrals), 0018 (polar webhook),
0019 (stripe), 0023 (users.lang index + qd_symbol widen) explicitly
wrap their problematic ops in `with op.batch_alter_table(...) as bop`.
Now `alembic upgrade head` + `alembic downgrade base` round-trip cleanly
on a fresh SQLite database. MariaDB prod behaviour unchanged.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three recently-added tables (strategic_log_translations,
indicator_summary_translations, csv_format_templates) drifted from
the codebase's existing naming convention:
- llm_model -> model
- llm_cost_usd -> cost_usd
- content_md -> content (on the two translation tables; csv_format
doesn't have a content field)
Also added prompt_tokens and completion_tokens to the three tables;
they were silently dropped at write time despite LogResult exposing
them.
All writer call sites (ai_log_job, indicator_summary_job,
llm_csv_parser) and reader call sites (api.py localized helpers)
updated to match. Tests realigned.
Migration 0025 uses batch_alter_table for SQLite compatibility.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two related schema fixes from the code review:
- users.lang gets a single-column index. The ai_log_job and
email_digest_job both SELECT DISTINCT on this column every cycle;
even at low cardinality an index is the right shape.
- quotes_daily.symbol widened to VARCHAR(128) to match quotes.symbol
(widened back in 0005). Long Eurostat/ONS symbols would silently
truncate during rollup otherwise.
Models updated to match (User.lang gains index=True, QuoteDaily.symbol
goes to String(128)).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bundles three related pieces that came out of the operator's first
end-to-end test of the paid flow:
1. Manage subscription button on /settings (paid users with a real
Stripe sub — i.e. not credit-granted access). POSTs to the existing
/api/stripe/portal endpoint; Stripe-hosted customer portal handles
card updates, cancellation, monthly↔annual switch, invoice history.
Replaces the stale "Paid features unlock with Paddle (D.3) or
invite credits" hint for free users with a live link to /pricing.
2. Per-cadence cooling-off treatment:
- **Annual £70**: 14-day free trial via
subscription_data.trial_period_days=14. No money moves during
the trial, so the CCR 2013 14-day refund question doesn't arise
(nothing paid = nothing to refund). Card is still required at
checkout so Stripe can charge on day 15.
- **Monthly £7**: bills immediately. A 14-day trial there would
give away ~50% of cycle one. Instead, /pricing now carries a
required tick-box above the Subscribe buttons (subscribe stays
disabled until checked) — by ticking, the user expressly
consents to begin performance immediately and acknowledges that
this extinguishes their statutory 14-day right under Reg 36
CCR 2013. Consent collected on our own page (not via Stripe's
account-wide consent_collection.terms_of_service) so each
product can keep its own Terms URL as we add more.
3. T&C §6 clause 1 split into 1a (annual / trial substitute) +
1b (monthly / Reg 36 waiver via on-page tick-box). Clause 2
(post-cooling-off cancellation) unchanged.
Settings page shows "Free trial — N days remaining" while the
sub is in `trialing` status, falling back to "Paid subscription
active." once it transitions to active. Countdown is computed
server-side from User.stripe_trial_end_at (new column, migration
0020) populated by the subscription.created/updated webhook from
the Stripe trial_end timestamp; cleared on the trialing→active
transition and on revoke.
Drive-by: fixed a structlog kwarg-name collision on
`log.warning(..., event=event_type, ...)` in both polar_webhook.py
and stripe_billing.py — `event` is structlog's positional event
name and "got multiple values" crashed the user-not-found log
path. Renamed to `event_type=` everywhere it appeared. Caught by
the new trialing-stores-trial-end test.
Tests
- 4 new in test_stripe_billing.py covering monthly (no trial, no
consent_collection), annual (trial, no consent), trialing stores
trial_end, trialing→active clears trial_end.
- 1 existing test renamed + reworked for the consent split.
- Full suite: 224 passed, 5 skipped.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Stripe is the merchant-on-record for read.markets after Polar/Paddle
both declined the financial-media category. This commit lands the
full subscription flow: an "Upgrade" button on /pricing now opens a
real Stripe-hosted Checkout, completes the subscription, and the
webhook flips user.tier to "paid" idempotently.
Endpoints
- POST /api/stripe/checkout (require_auth) — creates a hosted
Checkout Session in subscription mode, passes user.id as
client_reference_id + email as customer_email, returns the URL
for the page-side JS to redirect to. Reuses an existing
stripe_customer_id to avoid duplicate Stripe customers on repeat
checkouts. allow_promotion_codes=True so the referral-credit
redemption can attach a coupon at checkout once that flow ships.
- POST /api/stripe/portal (require_auth) — mints a Stripe Customer
Portal session. Used by /settings; returns 404 until the user has
a stripe_customer_id (i.e. completed at least one checkout).
- POST /api/stripe/webhook — signature-verified via
stripe.Webhook.construct_event. Idempotent via UNIQUE on
stripe_events.event_id. Event dispatch:
checkout.session.completed → grant paid, store IDs
customer.subscription.created → grant paid (active/trialing)
customer.subscription.updated → grant paid (active/trialing)
customer.subscription.deleted → drop to free, clear sub id
invoice.paid / failed → audit only
charge.refunded → audit only
Stripe-SDK objects don't expose dict.get(); we use the SDK for
signature verification then re-parse the JSON body for handler
dispatch — cleaner than reaching into StripeObject internals.
Schema (migration 0019)
- users.stripe_customer_id, users.stripe_subscription_id (nullable
String(64), UNIQUE on customer_id).
- stripe_events table mirroring polar_events: event_id (unique),
event_type, received_at, processed_at, error, raw payload
(truncated to 16 KiB).
Settings (.env)
- STRIPE_API_KEY (rk_test_… for dev, rk_live_… for GA)
- STRIPE_WEBHOOK_SECRET (whsec_… from the dashboard endpoint)
- STRIPE_PRICE_MONTHLY (price_xxx for £7/month)
- STRIPE_PRICE_ANNUAL (price_xxx for £70/year)
Pricing page
- Free tier CTA unchanged.
- Paid CTA branches three ways: paid → "Manage subscription" to
/settings; logged-in free → two buttons (£7/mo, £70/yr) that POST
to /api/stripe/checkout and redirect; anonymous → /login?next=/pricing.
- Inline JS intercepts the button click, calls the checkout
endpoint, redirects on success, surfaces errors via alert(). No
Stripe.js dep — we use the hosted-checkout URL directly.
Polar handler stays in place for berengar.io / flyroom.net which
still ship through Polar. polar_* and stripe_* columns coexist
independently on the User row.
Tests
- 9 in tests/test_stripe_billing.py covering: bad signature → 401,
missing signature → 400, checkout.session.completed flips tier +
stores IDs, subscription.updated active grants paid,
subscription.deleted drops to free with customer id preserved,
replayed event id is no-op (one row in stripe_events),
unknown event acked 200, checkout endpoint mocks the SDK and
returns the hosted URL, checkout requires login.
- Full suite: 221 passed, 5 skipped.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Standalone router for inbound Polar (merchant-of-record) deliveries.
No bearer-token dep — authenticity comes from the Standard Webhooks
HMAC instead. Wired up so it's safe to deploy dark: empty
POLAR_WEBHOOK_SECRET makes the endpoint return 503 (loud) rather than
accept unsigned events.
Behaviour
- Standard Webhooks signature verification: HMAC-SHA256 over
`{webhook-id}.{webhook-timestamp}.{body}`, base64 secret prefixed
whsec_, ±5min replay window, constant-time compare against any of
the space-separated v1 tokens.
- Idempotency via UNIQUE on polar_events.event_id — a replayed
webhook-id short-circuits to 200 "duplicate" without re-running.
- Event dispatch table covers the 10 events we subscribed to:
subscription.{created,active,updated,uncanceled} -> tier=paid +
persist polar_customer_id / polar_subscription_id.
subscription.revoked -> tier=free (customer id kept so a resub
matches the same User row).
canceled / past_due / order.* / refund.created -> audit only.
- Unknown event types are acked 200 + recorded; we don't want to 4xx
on something Polar adds in the future and trigger their retry loop.
Schema (migration 0018)
- users.polar_customer_id, users.polar_subscription_id (both nullable
String(64)); UNIQUE on polar_customer_id so two users can't claim
the same Polar identity.
- polar_events table: event_id (unique), event_type, received_at,
processed_at, error, raw payload (truncated to 16 KiB).
Tests
- 7 in tests/test_polar_webhook.py: bad signature -> 401, stale
timestamp -> 401, missing headers -> 400, subscription.active flips
tier to paid + stores IDs, subscription.revoked drops to free while
keeping customer link, replayed webhook-id is no-op, unknown event
is acked.
- Full suite: 212 passed, 5 skipped.
Operator next steps before saving the webhook in Polar
1. Pull this branch to prod and apply migration 0018.
2. Save the webhook in Polar pointing at
https://read.markets/api/polar/webhook — Polar will accept the
save even though our endpoint still 503s (no secret yet).
3. Copy the secret Polar reveals into the prod .env as
POLAR_WEBHOOK_SECRET=whsec_... and restart the app.
4. Trigger a test event from Polar's dashboard to confirm 200 OK.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The composite ix_email_sends_user_kind_sent (user_id, kind, sent_at)
already satisfies leftmost-prefix WHERE user_id=? lookups on both
MariaDB and SQLite, so the standalone per-column index was dead
weight. Drop it before the migration ever lands in prod.
Adds an 8-byte HKDF fingerprint of the current pepper to portfolio_sync
rows. On fetch, a mismatch surfaces as 410 Gone (distinct from genuine
GCM corruption → 500), and the UI silently cleans up the dead row and
shows a soft "please re-import" notice instead of a confusing PIN
re-prompt. Legacy rows (pepper_fp NULL) are probed optimistically and
backfilled on success.
Also fixes a latent bug in unwrap(): AESGCM.decrypt args were swapped
(ct, nonce instead of nonce, ct), so restore-from-cloud always failed
even when the pepper was correct.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Lays the billing-prep spine before Paddle lands in D.3.
D.1 — referrals
- users.referral_code: unique 8-char URL-safe code (alphabet excludes the
ambiguous 0/O/1/I/L). Generated lazily on first /settings hit so existing
accounts pick one up without a backfill migration.
- users.referred_by_user_id + new referrals audit table (referrer,
referred, created_at, converted_at, credited_at). converted_at /
credited_at stay null until D.3 fills them via the Paddle webhook.
- POST /login accepts ?ref=<code>; the code rides on the signed
pending-verify cookie so it survives the GET → POST → /verify hop.
- /settings page: email, tier badge, referral code chip + invite link
with one-click copy, pending/converted/active-credits stats grid.
Settings nav link added to the top bar.
Reward shape: when the referred user makes their first paid Paddle
subscription, both they and the referrer get 50% off for 3 months.
(D.3 wires the actual credit application via the Paddle webhook.)
D.2 — paid-access gate
- users.credit_until: timestamp until which a free-tier account has
paid-tier access. Null = no credit. Populated by admin CLI now and the
D.3 webhook later.
- app.services.access exposes paid_status(user) → PaidStatus dataclass
(active / source / expires_at / days_remaining), is_paid_active() with
admin-bearer-token bypass, and a require_paid FastAPI dependency that
raises 402 Payment Required for free-tier callers.
- POST /api/analyze (portfolio AI commentary) gated behind require_paid.
- Settings page surfaces credit window when active ("free · credit · N
day(s) remaining (expires YYYY-MM-DD)") and the upgrade hint when not.
- Admin CLI: python -m app.cli {grant-credit,revoke-credit,show-status}.
grant-credit is idempotent — extends from max(now, current expiry) so
re-running the command never erodes an existing grant.
Migrations 0013 (referrals) and 0014 (credit_until). Tests cover the
paid-status truth table, code generation + normalisation, CLI argument
parsing, and the pending-cookie ref roundtrip (29 new tests).
- Move news_job from hourly to 3x/hour (cron 10,30,50), with a CadencePolicy
gate that throttles to active hours (07-21 UTC weekdays at 20 min), off-hours
(3 h), weekends (6 h). Keeps the daytime feed fresh without spamming RSS
sources overnight.
- Tag each headline on ingestion via DeepSeek (BATCH_SIZE=25, max_tokens=4000,
json.JSONDecoder().raw_decode + per-row regex recovery for resilient parsing).
Vocabulary: 16 tags including new EU / USA / AI / Conflict. NULL tags are
picked up automatically on the next news_job run, so back-tagging is implicit
rather than a separate migration step.
- Tag UI: pill bar above the feed with off → include → exclude cycle on click;
shift-click jumps straight to exclude. State persists in localStorage and is
injected into /api/news requests via htmx:configRequest. Per-row chips sit to
the right of the headline (new 5-column grid: age | source | title | tags |
UTC) so vertical density stays high.
- Strategic log header bug: model was hallucinating "(Updated 21:30 UTC)" in
future tense. Bumped PROMPT_VERSION 6→7, added explicit ban on time-of-day
clauses, and supply the actual current UTC time in the user prompt so the
model has no need to invent one.
Migration 0012 adds headlines.tags (JSON, nullable). Tests cover vocabulary
integrity, validation/normalisation, and the JSON-recovery parser (17 tests).
Server no longer holds portfolios. Holdings live in the browser
(localStorage); the server publishes an anonymous ticker_universe and a
gzipped /api/universe payload identical for every authenticated user, so
access patterns can't betray which tickers a user holds. AI commentary
is generated ephemerally from the browser-supplied pie and the cost
ledger row records no positions. Migrations 0009-0011 added the
universe table and dropped positions / portfolio_snapshots /
portfolios.
Authentication is now e-mail OTP only. Migration 0010 dropped
password_hash and email_verified (every active session is by
construction proof of email control). The /signup endpoint is gone;
signup and login share a single email-entry page. Email rendering is
HTML+plain-text multipart with a shared brand palette (app/branding.py)
asserted in sync with the CSS by a drift-detection test.
LLM provider defaults to DeepSeek-direct (cheaper, api.deepseek.com)
with OpenRouter as automatic fallback if DeepSeek fails. ai_log_job and
indicator_summary_job now iterate the two tones (NOVICE, INTERMEDIATE)
per cycle so the dashboard's tone toggle is instant; PROMPT_VERSION
bumped to 6 with an educational anti-TA / anti-gambling stance baked
into _CORE. NOVICE mode renders a curated glossary inline (CBOE VIX,
yield curve, HY OAS, etc.) with JS-positioned tooltips that survive
viewport edges and sticky bars. Model name and tokens hidden from the
user UI; still recorded in StrategicLog.model and AICall for admin.
Layout adds a sticky top nav, a sticky bottom markets bar (one chip per
exchange with status LED + headline index + 1d change), and
Phase H feedback reporting is queued in tasks/todo.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the static bearer-token gate with a real auth boundary. The
existing CASSANDRA_TOKEN path is retained as an admin / scripting escape
hatch — kept compatible by aliasing require_token to require_auth.
- New users table (migration 0007): email, argon2 password_hash, tier,
email_verified (declared but not enforced until phase E), settings_json
for the tone/analysis/anchor knobs we'll wire in phase D.
- app/services/auth_service.py: argon2-cffi password hashing with timing-
attack-resistant authenticate() (always runs a hash verify even on
unknown-email to deny a username-enumeration oracle).
- app/auth.py rewritten: require_auth returns a CurrentUser with either
is_admin=True (bearer path) or a User object (session path). Failing
requests get 303 → /login for HTML, 401 for API. Sessions signed with
itsdangerous against CASSANDRA_SESSION_SECRET; 14-day TTL.
- app/routers/auth.py: /login, /signup, /logout. Login form preserves the
?next=… param for redirect-after-login. Signup respects a new
CASSANDRA_SIGNUP_ENABLED flag.
- Standalone /login + /signup templates (no app chrome). base.html grows
a user chip + logout link in the header (reads request.state.current_user).
Phase A's main known limitations are documented in the plan: email
verification is declared but not enforced; session revocation is
best-effort (cookie-only, not DB-backed). Both land in phase E.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First two slices of the multi-user roadmap (Phase B). Validates the
core onboarding mechanic against the user's real T212 export before
paying any auth/tenancy tax.
CSV parser (app/services/csv_import.py):
- Header-name matched (survives T212 reordering columns between
exports), tolerant of UTF-8 BOM, dash/N/A/empty markers, thousand-
separator commas, blank rows, zero-quantity stubs, missing Total row.
- Returns ParsedPie(name, positions, invested, value, result) with
derived avg_price + current_price per share in account currency.
- 14 tests covering happy path on the real CSV + 13 edge cases.
InstrumentMap (migration 0006 + app/services/instrument_map.py):
- Catalogue table mapping T212 ticker → Yahoo ticker, populated by
sync_from_t212() against the dev's read-only API key. Manual rows
(manual=True) are protected from auto-overwrite.
- Pure t212_ticker_to_yahoo() handles both suffix forms: single
trailing exchange letter (l/a/p/d/m/s/...) and country code (US,
DE, FR, IT, CA, ...). All 13 of the user's holdings + 15 case-
coverage tests pass.
- Live sync against T212 ingests 17,050 instruments (~2.2% unmappable
on exotic exchanges; can extend the suffix map later).
- resolve_slice() picks the right listing per shortName using a
UK-friendly currency preference (GBX > GBP > EUR > USD). Resolved
correctly for all 13 of the user's positions, including TTE on
Paris vs the NYSE dual-listing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three new data sources hooked into the existing SOURCES registry. All
open APIs, no keys:
- EUROSTAT: prefix EUROSTAT:dataset?dim=val&... — current EU bond
yields (Bund/OAT/BTP/EZ) and Eurozone economic indicators that
FRED's OECD-mirror series stopped updating in 2022-2023.
- ONS: prefix ONS:topic/cdid/dataset — current UK CPI, unemployment,
GDP, industrial production. Replaces the 5+ month-stale FRED
LRHUTTTTGBM156S mirror.
New indicator groups in default.toml feed the strategic/fundamental
lens we converged on: valuation (CAPE/Buffett anchors), bubble_watch
(SKEW/VVIX/RSP vs SPY/HYG vs TLT/IPO/crypto), economy (multi-region,
ALL current-or-stale-flagged), bonds (UK/EU/US/JPN sovereign yields).
Indicator panel now opens with an AI "read" interpretation per group
(generated hourly at :07 UTC alongside an aggregate cross-group read
shown in the dashboard header). The aggregate is grounded by a markets
strip — NYSE/LSE/Frankfurt/Tokyo/HK/Shanghai with open/closed LEDs and
next-open countdown, computed locally from each exchange's tz.
Other UX bits: indicator-row tooltips populated from TOML notes;
rows whose last observation is >90 days old get a 'stale' chip;
ghost symbols (in DB but no longer in TOML) filtered out of the
panel; Eurostat/ONS symbols display as short codes rather than the
full API path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Containerised macro-strategy dashboard: 4-panel web UI (indicators,
portfolio, flash news, AI strategic log), MariaDB store, hourly
ingestion jobs, OpenRouter-backed AI analysis.
Ports the four prototype scripts in the parent dir (market_pulse,
flash_news, trading212, strategic_log) into async services backed by a
persistent DB and served via FastAPI + Jinja2 + HTMX. APScheduler runs
as a separate compose service for crash-safety and easier restarts.
Portfolio composition + position names come live from Trading 212;
news per-ticker headlines reuse those names. Tone (NOVICE/INTERMEDIATE/
PRO) and analysis style (DRY/SPECULATIVE) are env-configurable and
stored on each log row so historical entries show what produced them.
Default model is deepseek/deepseek-v4-flash (overridable via env).
Light/dark theme toggle, sans-serif for prose surfaces, monospace for
data. Bearer-token auth, OpenRouter monthly cost cap, RSS feeds auto-
disabled on consecutive failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
