Sources and Methodology

Data collection

I collect benchmark results manually from primary public sources. That usually means official model-provider blogs, papers, technical reports, release posts, model cards, and benchmark leaderboards. Benchmark platforms include sources such as LiveBench, Arena, Aider, SWE-Bench, Terminal-Bench, MathArena, and similar public result pages.

RankedAGI does not own the underlying benchmark data. The site organizes public results for comparison, research, and discovery.

Benchmark inclusion criteria

Benchmarks are included when they help answer practical model-comparison questions and have publicly checkable results. I prioritize current evaluations for coding, reasoning, math, general preference, multimodal understanding, and agentic tasks.

Older benchmarks may be hidden or de-emphasized when they become saturated, superseded, hard to compare across releases, or less useful for ranking current frontier models.

Composite score methodology

RankedAGI composite scores combine selected benchmarks that target a related capability area, such as coding or agentic work. Scores are normalized before mixing so benchmarks with different raw formats can contribute to one comparable percentage-style score.

The benchmark mix and weights are chosen to reflect practical model performance, source quality, recency, benchmark coverage, and how well the benchmark tends to translate to real-world results. A model that is very strong in coding but weaker in reasoning will score well in coding-focused composites, while broader composites balance that strength against reasoning, agentic, math, general, and multimodal evidence when those signals are available.

Where public benchmark coverage is sparse, RankedAGI may use estimated or synthetic evidence to reduce gaps between models. Those estimates are supporting signals, not replacements for real benchmark results, and are weighted with less confidence than direct public scores. Missing benchmark data should not be read as a zero score.

Limitations: benchmark availability differs by model, some scores come from provider-reported runs, and public leaderboards can change over time. Composite scores are directional summaries, not claims that one public formula captures every use case.

signal_weight = (source_relevance / 50) ^ 1.25
effective_weight = signal_weight * trust * sampling_reliability

family_score = sum(source_score * effective_weight) / sum(effective_weight)
family_evidence = min(sum(effective_weight), strongest_source_weight * 1.2)

observed_score = sum(family_score * family_evidence)
evidence = sum(family_evidence)
final_score = (prior_score * prior_weight + observed_score) / (prior_weight + evidence)

Thinking model variants

For older model families with explicit thinking and non-thinking variants, RankedAGI records separate model entries when public benchmark data distinguishes those modes. If a benchmark source publishes a thinking-mode score, that score is recorded for the relevant thinking variant; if it publishes a non-reasoning or standard-mode score, that is recorded separately when the model naming makes the distinction clear.

RankedAGI records the public benchmark result as reported by the source. Reasoning traces are not scored separately unless the benchmark or source itself publishes a separate trace-level score.

Human preference benchmark uncertainty

Arena-style human preference benchmarks are useful signals, but they are not treated as a dominant penalty or top-priority input in composite scores. Their contribution is weighted by practical relevance and by how well the benchmark appears to translate to real-world model quality.

RankedAGI checks public Arena-style leaderboards weekly and updates changed scores when the source updates. Confidence intervals and sample sizes are not independently recalculated unless the source publishes them in a way that can be represented consistently on the site.

Model update frequency

New public model data is generally updated within 24 hours of release, often within 5-6 hours when the release and benchmark sources are clear. Each model carries a last-updated value so freshness is visible on the site.

Model versioning

Models are listed as public names change over time. Duplicate-looking names can exist when a provider releases a new dated build, preview, thinking variant, or API-visible version with different benchmark behavior.

A suffix such as -old means the row represents an earlier model entry kept for comparison or historical continuity, not the preferred current listing. The older pattern was to move a replaced same-name model to an -old slug and keep the newer release at the main slug.

The newer standard is to add a date or version marker to the slug when it avoids ambiguity. Weight or size is added for open-model families that release multiple sizes under the same model name. If a model has only one relevant size and there is no naming ambiguity, the size is usually omitted from the slug.

Context window sourcing

Context-window values are provider- or developer-reported unless otherwise marked. They may represent API limits, product limits, or documented model-card limits depending on what the source publishes.

Custom comparison behavior

Users can compare models by filtering the model table and showing or hiding benchmark columns. These controls change the view in the browser; they do not change the underlying public data.

Corrections and revisions

When source data changes or errors are found, RankedAGI updates the current public data rather than preserving a separate public revision log for every score. Per-score source attribution is planned as a larger provenance/data-model project.

Model pages usually include one or more global source links, such as an official release blog, model card, Hugging Face page, or public announcement. For third-party benchmarks, data is collected from the official leaderboard or result page for that benchmark, and many of those benchmark pages are linked from the benchmark index above.

Data access FAQ