feedback: thumb up/down on logs + reviewer self-score 0-10

Two unrelated features bundled because they ship together and share
migration 0028.

Strategic-log feedback (thumb up/down):
- New strategic_log_feedback table with UNIQUE(log_id, user_id) so each
  user has one vote per log, flippable in place (up -> down -> clear).
  UI shows aggregate counts only.
- app/services/log_feedback.py: set_vote, get_counts, sign/verify
  feedback tokens (same itsdangerous pattern as auth.sign_pending,
  30-day TTL for email links).
- POST /api/log/{id}/feedback: web vote, auth required, returns counts
  + the requesting user's own vote.
- GET /feedback?token=...&vote=...: email-link target, no auth, signed
  token encodes (user, log, vote), renders feedback_thanks.html.
- partials/log.html: thumbs row below content, JS-driven swap via the
  POST endpoint. Dashboard latest-log card and /log page both render
  this partial via htmx, so the buttons appear in all three surfaces.
- digest emails: a "How was today's read?" row above the unsub footer,
  with signed-token URLs against the latest StrategicLog at send time.
  Plain-text fallback included.

Reviewer self-score (0-10):
- _SYSTEM_PROMPT asks for an integer score with anchors (10 exemplary,
  5 borderline, 0 unfit). Verdict gains score: int | None.
- Deterministic-layer hits get score=0 (hard rule, no nuance);
  error rows get None; LLM rows get the model's score clamped 0..10.
- ReviewerVerdict.score, StrategicLog.reviewer_score, and
  IndicatorSummary.reviewer_score all new SMALLINT NULL columns.
- ai_log_job + indicator_summary_job persist verdict.score onto their
  content rows when committing the row alongside content.

Tests:
- tests/test_strategic_log_feedback.py: vote, flip, clear, aggregate
  across users, invalid vote, token round-trip + tamper + garbage +
  'clear' not signable for email path.
- tests/test_output_review.py: score parsing, clamping (>10, <0),
  missing/non-numeric -> None, deterministic-layer score=0.

Full suite: 427 passed (was 412), 5 skipped, no regressions.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Giorgio Gilestro 2026-05-29 21:28:03 +02:00
parent f3ac65f8f7
commit 8946dee2e0
14 changed files with 962 additions and 14 deletions

View file

@ -234,6 +234,7 @@ async def run() -> None:
prompt_tokens=result.prompt_tokens,
completion_tokens=result.completion_tokens,
cost_usd=full_cost,
reviewer_score=verdict.score,
)
session.add(slog)
session.add(AICall(

View file

@ -198,17 +198,32 @@ def _pick_variant(
async def _send_one(user: User, kind: str, content_html: str, date_str: str,
session) -> None:
session, *, latest_log_id: int | None = None) -> None:
settings_url = f"{branding.SITE_URL}/settings"
unsubscribe_url = (
f"{branding.SITE_URL}/email/unsubscribe"
f"?token={sign_unsubscribe_token(user.id)}"
)
# Build signed feedback URLs against the latest strategic log at send
# time. The token encodes (user, log, vote) so the recipient can
# click without being logged in; the receiving /feedback endpoint
# verifies the signature and applies the vote.
feedback_up_url = feedback_down_url = None
if latest_log_id is not None:
from app.services.log_feedback import sign_feedback_token
up_tok = sign_feedback_token(user.id, latest_log_id, "up")
down_tok = sign_feedback_token(user.id, latest_log_id, "down")
feedback_up_url = f"{branding.SITE_URL}/feedback?token={up_tok}&vote=up"
feedback_down_url = f"{branding.SITE_URL}/feedback?token={down_tok}&vote=down"
subject, text_body, html_body = render_digest_email(
kind=kind, date_str=date_str,
content_html=content_html,
unsubscribe_url=unsubscribe_url,
settings_url=settings_url,
feedback_up_url=feedback_up_url,
feedback_down_url=feedback_down_url,
)
try:
await send_email(to=user.email, subject=subject,
@ -288,6 +303,18 @@ async def run() -> None:
client, variants, active_non_en,
)
# Resolve the latest strategic log once per job — used as the
# target of the email's thumb up/down feedback links. None if
# nothing has been generated yet (shouldn't happen at this point
# in the flow, but defensible).
from sqlalchemy import desc, select
from app.models import StrategicLog
latest_log_id = (await session.execute(
select(StrategicLog.id)
.order_by(desc(StrategicLog.generated_at))
.limit(1)
)).scalar_one_or_none()
written = 0
for u in fresh:
tone = (u.digest_tone or "INTERMEDIATE").upper()
@ -296,7 +323,8 @@ async def run() -> None:
tone=tone,
lang=(u.lang or "en"),
)
await _send_one(u, kind, content, date_str, session)
await _send_one(u, kind, content, date_str, session,
latest_log_id=latest_log_id)
await asyncio.sleep(0.1)
written += 1

View file

@ -215,6 +215,7 @@ async def _generate_one(
# Include the reviewer's cost in the row's recorded spend so the
# monthly budget tracking covers the full pipeline cost.
cost_usd=(result.cost_usd or 0.0) + (verdict.cost_usd or 0.0),
reviewer_score=verdict.score,
)
session.add(summary)
session.add(AICall(
@ -342,6 +343,7 @@ async def run() -> None:
prompt_tokens=result.prompt_tokens,
completion_tokens=result.completion_tokens,
cost_usd=full_cost,
reviewer_score=verdict.score,
)
session.add(agg_summary)
session.add(AICall(