compliance: flag-gate AI portfolio + cloud sync + Stripe; de-risk prompts; harden reviewer
Implements docs/read-markets-compliance-changes.md as flag-gated changes (no deletions) so paused features stay in the tree for future re-enable. All four flags default False so a fresh deploy is compliance-safe. - New env flags: PORTFOLIO_AI_ENABLED, PORTFOLIO_SYNC_ENABLED, TICKER_UNIVERSE_AGGREGATE_ENABLED, SUBSCRIPTIONS_ENABLED. - Gates: /api/analyze, /api/portfolio/sync*, /api/stripe/*, /pricing, ticker_universe writes, portfolio_analysis.analyse(). is_paid_active() returns True for any auth'd user when subscriptions are paused. - Prompts (PROMPT_VERSION 10): universal _COMPLIANCE_RIDER prepended to every system prompt; watch list removed; price-target / close-above-below / trigger / forward-state-as-description rules added; SPECULATIVE pivoted to regime-only scenarios; daily + weekly digests tightened. - Reviewer: deterministic regex/lexicon pre-check fail-closed under the Haiku call; portfolio rider gated by PORTFOLIO_AI_ENABLED; base prompt sharpened for forward-state and MAR forward-opinion patterns; ReviewerVerdict audit table; generate_with_review retry helper. - Migration 0026: purge portfolio_sync + ticker_universe; create reviewer_verdicts. - Copy: MAR cite fixed to Art 3(1)(35) + Art 20 + Del Reg 2016/958; portfolio reframed as browser-only viewer in disclaimer / privacy / terms / about / pricing / landing (en + it). TODO(legal) marker for lawyer sign-off on disclaimer. - Tests: 13 lexicon + 6 reviewer compliance regressions; conftest enables all flags so existing 402 tests still cover their code paths. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
parent
ee8384f1ba
commit
47dce1a1a4
38 changed files with 1188 additions and 279 deletions
|
|
@ -1,17 +1,20 @@
|
|||
"""Second-pass reviewer agent for AI-generated reads.
|
||||
"""Two-layer reviewer for AI-generated reads.
|
||||
|
||||
The per-group and aggregate indicator summaries are generated in JSON
|
||||
mode and the publishable text comes out of a single "read" field, but a
|
||||
misbehaving model can still slip chain-of-thought INSIDE the field
|
||||
("Let's see…", "X? Actually Y?", multi-question parentheticals). This
|
||||
module makes a small second LLM call that judges the candidate read as
|
||||
clean / unclean. Cost is ~$0.0001 per check; latency ~1-2 s in the
|
||||
hourly job. No user-facing latency.
|
||||
Architecture (both layers fail-closed):
|
||||
1. Deterministic lexicon/regex pre-check (cheap, zero-variance) — see
|
||||
``app.services.output_review_lexicon``. Hard-catches the obvious
|
||||
advice / forecast / level-trigger patterns.
|
||||
2. LLM nuance check (Haiku via OpenRouter) — catches the subtle cases
|
||||
the lexicon misses (chain-of-thought leakage, forward-state-as-
|
||||
description, MAR forward price opinions on named instruments).
|
||||
|
||||
Either layer can reject. A reject drops the candidate; the previously
|
||||
cached good version stays visible. Every verdict (pass and fail) is
|
||||
persisted to ``reviewer_verdicts`` — the regulator-facing audit trail.
|
||||
|
||||
The reviewer is deliberately a tiny, JSON-shaped classifier — same
|
||||
JSON-mode mechanism as the generator, so the verdict can't be lost in
|
||||
prose. If parsing fails or the call errors, the row is rejected
|
||||
(fail-safe: the previously cached good summary stays visible).
|
||||
prose. Cost is ~$0.0001 per LLM check; latency ~1-2 s in the hourly job.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
|
@ -19,10 +22,13 @@ import json
|
|||
from dataclasses import dataclass
|
||||
|
||||
import httpx
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config import get_settings
|
||||
from app.logging import get_logger
|
||||
from app.models import ReviewerVerdict
|
||||
from app.services.openrouter import call_llm
|
||||
from app.services.output_review_lexicon import check as lexicon_check
|
||||
|
||||
log = get_logger("output_review")
|
||||
|
||||
|
|
@ -93,6 +99,28 @@ Mark UNCLEAN if the text contains ANY of:
|
|||
"valuations are stretched", "real yields are restrictive", "rates
|
||||
and credit disagree". The test: does the text describe a STATE, or
|
||||
does it suggest an ACTION? States are fine; actions are not.
|
||||
- FORWARD PREDICTION DRESSED AS PRESENT-STATE DESCRIPTION. A claim
|
||||
about direction smuggled into state-shaped language is still a
|
||||
forecast, and an action-focused rule misses it.
|
||||
* Allowed (pure state): "valuations are stretched", "real yields
|
||||
are restrictive", "positioning is crowded".
|
||||
* Reject (state + direction): "valuations are stretched and unlikely
|
||||
to hold", "the path of least resistance is lower", "risk is
|
||||
skewed to the downside", "the setup favours further weakness".
|
||||
If the sentence implies which way prices go next, flag it.
|
||||
- MAR INVESTMENT-RECOMMENDATION — a forward-looking opinion on the
|
||||
price or value of a specific NAMED INSTRUMENT, even without an
|
||||
explicit numeric target and even without a "you should" verb.
|
||||
* Reject: "Brent is likely to consolidate near $90–93", "the
|
||||
base case is a move to X for gold", "TSLA appears poised to
|
||||
drift lower", "EUR/USD looks set to test new highs".
|
||||
* Allowed: regime-level forward observation that does NOT name a
|
||||
specific instrument's price trajectory ("the policy mix is still
|
||||
tightening", "real yields are likely to stay restrictive while
|
||||
the labour market holds up").
|
||||
This rule applies to single tickers, single commodities, and single
|
||||
FX pairs equally. A cross-asset *regime* claim is fine; a price
|
||||
claim on a *named* instrument is not.
|
||||
- Anything else other than the finished, publishable commentary.
|
||||
|
||||
Return ONLY a JSON object with this exact shape:
|
||||
|
|
@ -102,8 +130,11 @@ No preamble, no markdown fences, no other fields.
|
|||
|
||||
|
||||
# Surface-specific rider appended to the system prompt when the caller
|
||||
# passes a known `surface` to review_read(). Lets us relax or tighten
|
||||
# rules per editorial context without rewriting the whole prompt.
|
||||
# passes a known `surface` to review_read(). The portfolio rider — the only
|
||||
# entry here — only fires when PORTFOLIO_AI_ENABLED is on AND the caller
|
||||
# explicitly tags surface="portfolio". With the flag off (default) the
|
||||
# strict base rules govern every surface; future re-enable requires both
|
||||
# flipping the flag and a deliberate code review of this loosened block.
|
||||
_SURFACE_RIDERS = {
|
||||
"portfolio": """\
|
||||
|
||||
|
|
@ -115,8 +146,6 @@ as financial advice. The following ARE fine:
|
|||
exposure", "currency risk is unhedged", "FX exposure", "elevated
|
||||
risk", "stretched valuations", "concentration is manageable", "low
|
||||
diversification".
|
||||
- Stating what would invalidate the posture: "this view fails if
|
||||
rates retrace", "the thesis depends on X holding".
|
||||
- Impersonal observation about a position's behaviour or sensitivity:
|
||||
"the position warrants monitoring", "carries vulnerability to a
|
||||
policy shock", "is sensitive to rate moves".
|
||||
|
|
@ -129,6 +158,9 @@ aimed at the reader:
|
|||
- Specific allocation prescriptions: "go 20% bonds", "overweight
|
||||
tech", "underweight defensives".
|
||||
- Price-target predictions: "will reach $X by year-end".
|
||||
- Forward conditional judgements on a held position ("the thesis
|
||||
fails if rates retrace", "this view depends on X holding"): these
|
||||
read as "you should be watching for X" and are out of scope.
|
||||
""",
|
||||
}
|
||||
|
||||
|
|
@ -138,30 +170,101 @@ class Verdict:
|
|||
clean: bool
|
||||
reason: str
|
||||
cost_usd: float | None # cost of the review call itself, for the ledger
|
||||
layer: str = "llm" # "deterministic" | "llm" | "error"
|
||||
|
||||
|
||||
# Truncation cap for the audit log's candidate_text column. Generous enough
|
||||
# to keep useful context, bounded to keep pathological inputs from blowing
|
||||
# the row size.
|
||||
_AUDIT_CANDIDATE_MAX = 16_000
|
||||
|
||||
|
||||
async def _record_verdict(
|
||||
session: AsyncSession | None,
|
||||
*,
|
||||
surface: str | None,
|
||||
candidate: str,
|
||||
verdict: Verdict,
|
||||
model: str | None,
|
||||
) -> None:
|
||||
"""Best-effort persist the verdict to ``reviewer_verdicts``. Errors here
|
||||
must NEVER mask the caller's verdict — wrap and swallow."""
|
||||
if session is None:
|
||||
return
|
||||
try:
|
||||
row = ReviewerVerdict(
|
||||
surface=surface,
|
||||
candidate_text=candidate[:_AUDIT_CANDIDATE_MAX],
|
||||
clean=verdict.clean,
|
||||
reason=verdict.reason[:240] if verdict.reason else None,
|
||||
layer=verdict.layer,
|
||||
model=model,
|
||||
)
|
||||
session.add(row)
|
||||
await session.flush()
|
||||
except Exception as e:
|
||||
log.warning("review.audit_write_failed", error=str(e)[:200])
|
||||
|
||||
|
||||
async def review_read(
|
||||
client: httpx.AsyncClient,
|
||||
candidate: str,
|
||||
surface: str | None = None,
|
||||
*,
|
||||
session: AsyncSession | None = None,
|
||||
) -> Verdict:
|
||||
"""Ask the LLM whether `candidate` is a publishable read.
|
||||
"""Run the two-layer reviewer on `candidate`.
|
||||
|
||||
Returns Verdict(clean, reason, cost). Any error — provider failure,
|
||||
JSON parse failure, missing field, wrong type — yields a CONSERVATIVE
|
||||
verdict (clean=False) so the caller drops the candidate. The
|
||||
previously cached good summary stays visible on the dashboard.
|
||||
Layer 1: deterministic lexicon/regex pre-check. On hit, returns
|
||||
``clean=False, reason="lexicon:<rule>"``, layer="deterministic" — no
|
||||
LLM call is made.
|
||||
|
||||
Layer 2: LLM nuance check (Haiku via OpenRouter). On JSON-mode
|
||||
success, returns whatever the model decided. Any error — provider
|
||||
failure, JSON parse failure, missing field, wrong type — yields a
|
||||
CONSERVATIVE verdict (clean=False) so the caller drops the candidate.
|
||||
The previously cached good summary stays visible on the dashboard.
|
||||
|
||||
`surface` selects a surface-specific rider — see _SURFACE_RIDERS. The
|
||||
only entry, "portfolio", is additionally gated behind
|
||||
PORTFOLIO_AI_ENABLED so it cannot loosen review when the feature is
|
||||
paused.
|
||||
|
||||
When ``session`` is provided, every verdict (pass and fail, every
|
||||
layer) is persisted to ``reviewer_verdicts`` for the audit trail.
|
||||
Persistence errors are swallowed."""
|
||||
settings = get_settings()
|
||||
|
||||
`surface` selects a surface-specific rider that's appended to the
|
||||
base system prompt — see _SURFACE_RIDERS. Currently only the
|
||||
"portfolio" surface uses one (descriptive risk language is the
|
||||
whole point there and shouldn't be flagged as advice). Unknown
|
||||
or None surfaces fall back to the generic rules."""
|
||||
if not candidate or not candidate.strip():
|
||||
return Verdict(clean=False, reason="empty candidate", cost_usd=0.0)
|
||||
verdict = Verdict(clean=False, reason="empty candidate", cost_usd=0.0,
|
||||
layer="deterministic")
|
||||
await _record_verdict(session, surface=surface, candidate=candidate or "",
|
||||
verdict=verdict, model=None)
|
||||
return verdict
|
||||
|
||||
# Layer 1 — deterministic lexicon pre-check. Cheap, zero-variance.
|
||||
hit = lexicon_check(candidate)
|
||||
if hit is not None:
|
||||
verdict = Verdict(
|
||||
clean=False,
|
||||
reason=f"lexicon:{hit.rule}: {hit.snippet}",
|
||||
cost_usd=0.0,
|
||||
layer="deterministic",
|
||||
)
|
||||
log.info("review.deterministic_reject",
|
||||
rule=hit.rule, snippet=hit.snippet, surface=surface)
|
||||
await _record_verdict(session, surface=surface, candidate=candidate,
|
||||
verdict=verdict, model=None)
|
||||
return verdict
|
||||
|
||||
# Layer 2 — LLM nuance check. The portfolio rider is doubly gated: by
|
||||
# PORTFOLIO_AI_ENABLED here, and by the only caller's own flag-gate.
|
||||
system_prompt = _SYSTEM_PROMPT
|
||||
if surface and surface in _SURFACE_RIDERS:
|
||||
if (
|
||||
surface
|
||||
and surface in _SURFACE_RIDERS
|
||||
and (surface != "portfolio" or settings.PORTFOLIO_AI_ENABLED)
|
||||
):
|
||||
system_prompt = system_prompt + _SURFACE_RIDERS[surface]
|
||||
|
||||
messages = [
|
||||
|
|
@ -171,7 +274,6 @@ async def review_read(
|
|||
# contain prompt-like prose.
|
||||
{"role": "user", "content": f"Candidate read:\n```\n{candidate}\n```"},
|
||||
]
|
||||
settings = get_settings()
|
||||
reviewer_model = getattr(settings, "REVIEWER_MODEL", None) or DEFAULT_REVIEWER_MODEL
|
||||
try:
|
||||
result = await call_llm(
|
||||
|
|
@ -190,8 +292,11 @@ async def review_read(
|
|||
)
|
||||
except Exception as e:
|
||||
log.warning("review.call_failed", error=str(e)[:200])
|
||||
return Verdict(clean=False, reason=f"reviewer error: {str(e)[:80]}",
|
||||
cost_usd=None)
|
||||
verdict = Verdict(clean=False, reason=f"reviewer error: {str(e)[:80]}",
|
||||
cost_usd=None, layer="error")
|
||||
await _record_verdict(session, surface=surface, candidate=candidate,
|
||||
verdict=verdict, model=reviewer_model)
|
||||
return verdict
|
||||
|
||||
# Haiku (and several other models) occasionally wrap their JSON
|
||||
# output in a markdown code fence even with response_format set —
|
||||
|
|
@ -211,12 +316,101 @@ async def review_read(
|
|||
parsed = json.loads(raw)
|
||||
except json.JSONDecodeError:
|
||||
log.warning("review.parse_failed", preview=result.content[:200])
|
||||
return Verdict(clean=False, reason="reviewer returned non-JSON",
|
||||
cost_usd=result.cost_usd)
|
||||
verdict = Verdict(clean=False, reason="reviewer returned non-JSON",
|
||||
cost_usd=result.cost_usd, layer="error")
|
||||
await _record_verdict(session, surface=surface, candidate=candidate,
|
||||
verdict=verdict, model=reviewer_model)
|
||||
return verdict
|
||||
|
||||
clean = parsed.get("clean")
|
||||
reason = parsed.get("reason") or ""
|
||||
if not isinstance(clean, bool):
|
||||
return Verdict(clean=False, reason="reviewer omitted bool 'clean'",
|
||||
cost_usd=result.cost_usd)
|
||||
return Verdict(clean=clean, reason=str(reason)[:200], cost_usd=result.cost_usd)
|
||||
verdict = Verdict(clean=False, reason="reviewer omitted bool 'clean'",
|
||||
cost_usd=result.cost_usd, layer="error")
|
||||
await _record_verdict(session, surface=surface, candidate=candidate,
|
||||
verdict=verdict, model=reviewer_model)
|
||||
return verdict
|
||||
|
||||
verdict = Verdict(clean=clean, reason=str(reason)[:200],
|
||||
cost_usd=result.cost_usd, layer="llm")
|
||||
await _record_verdict(session, surface=surface, candidate=candidate,
|
||||
verdict=verdict, model=reviewer_model)
|
||||
return verdict
|
||||
|
||||
|
||||
# --- Regenerate-on-fail helper -----------------------------------------------
|
||||
|
||||
|
||||
# Maximum regenerate attempts before falling back to stale-cache. Two retries
|
||||
# is the sweet spot: cheap enough on cost ($0.0003-$0.001 extra per rejected
|
||||
# draft), enough to absorb the occasional model drift, and capped so a
|
||||
# genuinely bad prompt template doesn't loop forever.
|
||||
MAX_REGENERATE_ATTEMPTS = 2
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ReviewedGeneration:
|
||||
"""Result of generate_with_review. ``content`` is the accepted text (or
|
||||
None if all attempts were rejected — caller falls back to stale cache).
|
||||
``verdict`` is the final reviewer verdict (which may be a reject when
|
||||
content is None). ``attempts`` counts how many generator calls ran."""
|
||||
content: str | None
|
||||
verdict: Verdict
|
||||
attempts: int
|
||||
|
||||
|
||||
async def generate_with_review(
|
||||
*,
|
||||
client: httpx.AsyncClient,
|
||||
generate,
|
||||
surface: str | None = None,
|
||||
session: AsyncSession | None = None,
|
||||
max_attempts: int = MAX_REGENERATE_ATTEMPTS + 1, # 1 initial + N retries
|
||||
) -> ReviewedGeneration:
|
||||
"""Run a generator, review it, and on reject feed the reviewer's reason
|
||||
back into the generator for up to ``max_attempts - 1`` retries before
|
||||
falling back. Returns ``ReviewedGeneration(content=None, ...)`` when
|
||||
every attempt was rejected — caller decides how to fall back (typically
|
||||
keep the stale cached version).
|
||||
|
||||
``generate`` is an async callable ``(reject_reason: str | None) -> str``
|
||||
that produces a candidate. The reject_reason is None on the first call
|
||||
and the prior verdict's reason on subsequent calls — the generator is
|
||||
expected to thread it into its system prompt as guidance.
|
||||
|
||||
Every attempt's verdict is audited via the session — see review_read."""
|
||||
last_verdict: Verdict | None = None
|
||||
for attempt in range(1, max_attempts + 1):
|
||||
reject_reason = last_verdict.reason if last_verdict else None
|
||||
try:
|
||||
candidate = await generate(reject_reason)
|
||||
except Exception as e:
|
||||
log.warning("review.generator_failed",
|
||||
attempt=attempt, error=str(e)[:200])
|
||||
return ReviewedGeneration(
|
||||
content=None,
|
||||
verdict=Verdict(clean=False,
|
||||
reason=f"generator error: {str(e)[:80]}",
|
||||
cost_usd=None, layer="error"),
|
||||
attempts=attempt,
|
||||
)
|
||||
|
||||
verdict = await review_read(client, candidate, surface, session=session)
|
||||
if verdict.clean:
|
||||
return ReviewedGeneration(
|
||||
content=candidate, verdict=verdict, attempts=attempt,
|
||||
)
|
||||
|
||||
log.info("review.regenerate_attempt",
|
||||
attempt=attempt, max=max_attempts,
|
||||
reason=verdict.reason[:80] if verdict.reason else None,
|
||||
layer=verdict.layer)
|
||||
last_verdict = verdict
|
||||
|
||||
# Exhausted — every attempt was rejected.
|
||||
return ReviewedGeneration(
|
||||
content=None,
|
||||
verdict=last_verdict or Verdict(clean=False, reason="no attempts",
|
||||
cost_usd=None, layer="error"),
|
||||
attempts=max_attempts,
|
||||
)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue