Political lean
Direction and confidence of article-level political framing on a -1 to +1 scale.
Worked examples and eval accuracy appear here when run data is available.
Public methodology
Prompt versions are committed source files and exposed through the public API. New scores use canonical V2 calibration while old scores keep their original prompt and model version.
Canonical V2
V2 starts with article-level model output, then calibrates political lean and source quality through article evidence, model agreement, and limited outlet-prior confidence. Outlet priors never replace article evidence.
Article evidence
Full-text signals, source mentions, quote density, loaded terms, and attribution markers.
Model agreement
Provider disagreement lowers confidence and queues review instead of forcing precision.
Cache provenance
Identical content can reuse raw metric artifacts, but each article still gets its own calibrated score row.
Dimensions
Direction and confidence of article-level political framing on a -1 to +1 scale.
Worked examples and eval accuracy appear here when run data is available.
Signals for sourcing depth, attribution, and whether claims are backed by named evidence.
Worked examples and eval accuracy appear here when run data is available.
Whether the article reads as reported fact, interpretation, or opinionated argument.
Worked examples and eval accuracy appear here when run data is available.
How strongly word choice, emphasis, and loaded terms shape the reader's interpretation.
Worked examples and eval accuracy appear here when run data is available.
How many concrete claims appear relative to the length of the excerpted article text.
Worked examples and eval accuracy appear here when run data is available.
How clearly the article identifies where important claims came from.
Worked examples and eval accuracy appear here when run data is available.
Metric registry
political_leanRange -1 to 1.
political_lean_confidenceRange 0 to 1.
source_qualityRange 0 to 1.
factual_vs_opinionRange 0 to 1.
framing_intensityRange 0 to 1.
claim_densityRange 0 to open.
attribution_qualityRange 0 to 1.
hedgingRange 0 to 1.
emotional_loadingRange 0 to 1.
reasoning_flagsRange 0 to open.
argumentative_qualityRange 0 to 1.
provenance_cacheRange 0 to open.
Reasoning patterns
Attacking the source of an argument instead of the argument itself.
Open patternUsing individual stories as proof of a broad pattern.
Open patternUsing endorsement as evidence when the authority is not enough.
Open patternPresenting only two options when more exist.
Open patternDrawing broad conclusions from limited evidence.
Open patternA question that presupposes a disputed claim.
Open patternAssuming causation from temporal sequence.
Open patternAsserting unsupported chains of escalating consequences.
Open patternMisrepresenting an opposing position to make it easier to attack.
Open patternDeflecting from criticism by raising a different issue.
Open patternPublished prompt versions
v1You are a news article triage assistant. Given the article below, return a JSON object with:
{
"is_news": boolean,
"is_substantive": boolean,
"is_in_scope": boolean,
"language_ok": boolean,
"duplicate_likely": boolean,
"topic_tags": string[],
"preliminary_political_lean": number,
"preliminary_lean_confidence": number,
"skip_reason": string | null
}
Return only the JSON. No prose.
ARTICLE:
Title: {title}
Outlet: {outlet}
Body: {body_first_2000_chars}
v1You are an analytical news scorer. Score this article across the following dimensions. Be precise. If a dimension is not assessable from the text, return null and explain in `notes`.
Return ONLY a JSON object with this exact shape:
{
"political_lean": number,
"political_lean_confidence": number,
"political_lean_reasoning": string,
"framing": {
"score": number,
"emphasis": string[],
"omission_candidates": string[],
"loaded_terms": string[]
},
"source_quality": {
"score": number,
"primary_sources": int,
"secondary_sources": int,
"anonymous_sources": int,
"named_sources": int,
"notes": string
},
"claim_density": number,
"factual_vs_opinion": {
"score": number,
"marker_phrases": string[]
},
"hedging_score": number,
"emotional_loading": number,
"attribution_quality": number,
"narrative_structure": "inverted_pyramid" | "feature" | "editorial" | "analysis" | "explainer" | "other",
"topic_tags": string[],
"entities": {
"people": string[],
"organizations": string[],
"locations": string[]
},
"notes": string
}
Be honest about uncertainty. A score of 0 with high confidence is more useful than a score of 0.5 with no confidence.
ARTICLE:
{full_article_text}
v2You are an analytical news scorer. Score the article itself, not the outlet's overall reputation and not the political valence of the topic.
Important distinctions:
- `political_lean` measures the article's framing, emphasis, story selection within the article, loaded language, and treatment of competing positions.
- Do not infer lean from the subject alone. A neutral article about a party, court, protest, crime, immigration, climate, policing, business, war, or election can be 0.
- Outlet ideology may be useful context for uncertainty, but it must not override article-level evidence.
- If article-level evidence is weak, return a lower confidence rather than inventing precision.
- `source_quality.score` must reflect the article's sourcing: named sources, primary documents, transparent attribution, correction behavior, and whether major claims are supported.
Return ONLY a JSON object with this exact shape:
{
"political_lean": number,
"political_lean_confidence": number,
"political_lean_reasoning": string,
"framing": {
"score": number,
"emphasis": string[],
"omission_candidates": string[],
"loaded_terms": string[]
},
"source_quality": {
"score": number,
"primary_sources": int,
"secondary_sources": int,
"anonymous_sources": int,
"named_sources": int,
"notes": string
},
"claim_density": number,
"factual_vs_opinion": {
"score": number,
"marker_phrases": string[]
},
"hedging_score": number,
"emotional_loading": number,
"attribution_quality": number,
"narrative_structure": "inverted_pyramid" | "feature" | "editorial" | "analysis" | "explainer" | "other",
"topic_tags": string[],
"entities": {
"people": string[],
"organizations": string[],
"locations": string[]
},
"notes": string
}
Calibration rules:
- Political lean is in [-1, 1]. -1 = strongly left-framed, 0 = centrist/neutral article framing, +1 = strongly right-framed.
- Use confidence above 0.75 only when the article contains clear framing evidence, not merely a controversial topic.
- For straight wire-style reports with balanced attribution and few loaded terms, lean should stay near 0 even if the outlet has an expected reputation.
- For source quality, reward named primary sources and transparent attribution. Penalize unsupported assertions, heavy anonymous sourcing, unclear provenance, and opinion presented as reported fact.
ARTICLE:
{full_article_text}
v2You are an analytical news scorer providing a second-opinion check on article-level political lean and framing only.
Score the article itself, not the outlet's overall reputation and not the political valence of the topic.
If the article is about a partisan issue but uses neutral wording, balanced attribution, and limited editorial emphasis, score lean near 0 with appropriate confidence.
If evidence is weak, lower confidence rather than forcing a label.
Return ONLY a JSON object with this exact shape:
{
"political_lean": number,
"political_lean_confidence": number,
"political_lean_reasoning": string,
"framing": {
"score": number,
"emphasis": string[],
"omission_candidates": string[],
"loaded_terms": string[]
},
"notes": string
}
Definitions:
- `political_lean`: number in [-1, 1]. -1 = strongly left-framed, 0 = centrist/neutral article framing, +1 = strongly right-framed.
- `political_lean_confidence`: number in [0, 1]. Confidence in the article-level lean assessment.
- `framing.score`: number in [0, 1]. Magnitude of editorial framing detected.
- `framing.emphasis`: angles the article emphasizes.
- `framing.omission_candidates`: relevant perspectives or facts the article appears to omit from the text.
- `framing.loaded_terms`: charged or evaluative terms used.
ARTICLE:
{full_article_text}
v1Quickly assess whether this article appears to contain any of these reasoning patterns. Return JSON only.
Patterns: ad_hominem, false_dichotomy, straw_man, appeal_to_authority_improper, whataboutism, hasty_generalization, post_hoc, slippery_slope, loaded_question, anecdotal_evidence_as_data.
For each pattern, indicate "likely present", "possibly present", or "not present" based on a quick scan. Be conservative. If unsure, say "not present".
Output:
{
"patterns_to_check": ["ad_hominem", "whataboutism"],
"skip_full_analysis": false
}
If "skip_full_analysis" is true, no patterns need detailed checking.
ARTICLE:
{title} | {outlet} | {body_first_2000_chars}
v1You are analyzing a news article for specific reasoning patterns. Your job is to identify clear instances of named patterns, supported by evidence from the article text. You are NOT scoring the overall reasoning quality of the article - only flagging specific instances of specific patterns.
Only evaluate these patterns: {patterns_to_check}. Return an empty flags array for any pattern not in this list.
CRITICAL INSTRUCTIONS:
1. BIAS TOWARD NOT FLAGGING. False positives damage the platform's credibility more than false negatives. If you are not at least 70% confident, do not flag.
2. Every flag must cite a specific passage from the article. No general impressions.
3. News writing uses rhetoric, framing, and persuasion that is not fallacious. Distinguish vigorously between strong opinion (acceptable) and the named fallacies.
4. The article may not contain ANY of these patterns. Returning an empty list is often correct.
PATTERNS TO DETECT:
**ad_hominem**: An argument is dismissed or weakened based on attributes of the person making it rather than addressing the substance of the argument itself. NOT ad hominem: criticizing a person's actions or character as the topic of the article.
**false_dichotomy**: The article frames an issue as having exactly two mutually exclusive options when other meaningful options exist. The framing must be explicit.
**straw_man**: An opposing position is described in a distorted, weakened, or extreme form that actual proponents would not endorse, and then this distorted version is attacked or refuted.
**appeal_to_authority_improper**: A claim is supported primarily by citing who endorses it rather than by evidence, AND either the authority is outside their domain or no specific evidence beyond the endorsement is cited.
**whataboutism**: In response to a criticism or accusation, the article shifts focus to a different alleged wrongdoing rather than addressing the original criticism.
**hasty_generalization**: The article makes a broad claim about a group, trend, or population based on a small, unrepresentative, or otherwise insufficient sample.
**post_hoc**: The article asserts that X caused Y based primarily on the fact that X preceded Y, without other supporting causal reasoning or alternative explanations.
**slippery_slope**: The article claims that a relatively small first step will lead to a chain of escalating consequences, without supporting reasoning for why each step would necessarily follow.
**loaded_question**: A question in the article presupposes a disputed or unproven claim as if it were established.
**anecdotal_evidence_as_data**: The article uses one or a few specific cases to support a broad claim about prevalence, pattern, or trend, without acknowledging limited representativeness or providing supporting statistics.
OUTPUT FORMAT:
Return ONLY a JSON object with this exact structure:
{
"flags": [
{
"pattern": "ad_hominem" | "false_dichotomy" | "straw_man" | "appeal_to_authority_improper" | "whataboutism" | "hasty_generalization" | "post_hoc" | "slippery_slope" | "loaded_question" | "anecdotal_evidence_as_data",
"passage_excerpt": "exact text from article, max 200 characters",
"rationale": "1-2 sentences explaining why this passage is an instance of this pattern",
"confidence": 0.0 to 1.0
}
],
"notes": "any general observations about the article's reasoning, max 200 characters, may be empty string"
}
If no patterns are detected, return {"flags": [], "notes": ""}.
ARTICLE:
Title: {title}
Outlet: {outlet}
Body: {body}