{"atlas":{"title":"Synthetic Intelligence Atlas","url":"https://research.mahastrategies.com/atlas/synthetic-intelligence","version":"1.0.0","evidenceCutoff":"2026-06-14","lastReviewed":"2026-07-27","license":"https://creativecommons.org/licenses/by/4.0/","note":"First public edition: established and active-research claims only. Forecast scenarios and local-audit artifacts are intentionally not exposed."},"count":22,"claims":[{"id":"si-001","slug":"transformer-architecture","claim":"The transformer architecture replaces recurrence and convolution with attention as the primary sequence-modelling mechanism.","status":"established","statusLabel":"Established result","explanation":"This is an architectural fact stated by the originating paper and universally adopted since.","limitations":"Architecture alone determines no capability. Nothing about attention implies any particular task competence.","conceptIds":["transformers","large-language-models"],"sourceIds":["vaswani2017"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-001"},{"id":"si-002","slug":"in-context-learning","claim":"Sufficiently large autoregressive language models perform tasks from instructions and examples supplied in context, without gradient updates.","status":"established","statusLabel":"Established result","explanation":"In-context few-shot behaviour was the central reported finding of the GPT-3 paper and is reproduced widely.","limitations":"Reported on the task distributions studied. It is not a claim that in-context learning generalizes to arbitrary tasks, and it says nothing about reliability.","conceptIds":["large-language-models","pre-training"],"sourceIds":["brown2020"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-002"},{"id":"si-003","slug":"scaling-laws-loss","claim":"Language-model loss follows empirical power-law relationships with training compute, parameter count, and dataset size over the studied ranges.","status":"established","statusLabel":"Established result","explanation":"The originating scaling-law work fitted these relations across several orders of magnitude.","limitations":"These are relations on loss within a studied range, not on task capability, and not guaranteed to hold outside it. Treating a loss extrapolation as a capability forecast is not supported by the source.","conceptIds":["scaling-laws","pre-training"],"sourceIds":["kaplan2020"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-003"},{"id":"si-004","slug":"compute-optimal-allocation-contested","claim":"The compute-optimal allocation between parameters and training tokens has been revised in the literature, and deployed practice diverges from compute-optimal training.","status":"active","statusLabel":"Active research","explanation":"The Chinchilla work revised the earlier parameter-heavy allocation toward a more balanced ratio, and large deployed models are trained well past compute-optimal token counts to reduce inference cost.","limitations":"The direction of the revision is sourced; the specific exponents and ratios are not restated here, and no claim is made about what any particular current model used.","conceptIds":["scaling-laws","pre-training"],"sourceIds":["kaplan2020","hoffmann2022","grattafiori2024"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-004"},{"id":"si-005","slug":"rlhf-instruction-following","claim":"Fine-tuning with human feedback substantially changes how closely a model follows instructions relative to its pre-trained base.","status":"established","statusLabel":"Established result","explanation":"The InstructGPT work established this as a distinct post-training stage rather than a property of pre-training.","limitations":"Instruction-following is not correctness, safety, or capability. Improvements on preference judgements do not transfer automatically to task accuracy.","conceptIds":["post-training","large-language-models"],"sourceIds":["ouyang2022"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-005"},{"id":"si-006","slug":"chain-of-thought","claim":"Prompting a model to produce intermediate reasoning steps changes its measured performance on multi-step problems.","status":"established","statusLabel":"Established result","explanation":"Chain-of-thought prompting demonstrated that the inference procedure, not only the weights, determines measured performance.","limitations":"A produced reasoning trace is not evidence of the process that generated the answer. The source measures outputs, not internal reasoning.","conceptIds":["inference-time-compute","post-training"],"sourceIds":["wei2022"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-006"},{"id":"si-007","slug":"test-time-compute-tradeoff","claim":"Allocating additional computation at inference time can, in studied settings, improve results more than spending the equivalent compute on additional parameters.","status":"active","statusLabel":"Active research","explanation":"This trade-off is the basis of reasoning-style systems and is reported with verifier-based and search-based methods.","limitations":"Demonstrated within specific task families and compute budgets. It is not established as a general substitution, and the studied regimes do not license extrapolation.","conceptIds":["inference-time-compute","scaling-laws"],"sourceIds":["snell2024","cobbe2021"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-007"},{"id":"si-008","slug":"rag-external-knowledge","claim":"Retrieval-augmented generation conditions outputs on documents retrieved at inference time rather than on parametric memory alone.","status":"established","statusLabel":"Established result","explanation":"The originating formulation combined a retriever with a generator for knowledge-intensive tasks.","limitations":"Retrieval changes the source of knowledge, not the reliability of reasoning over it. It does not eliminate fabrication.","conceptIds":["retrieval-augmented-generation","tool-use"],"sourceIds":["lewis2020"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-008"},{"id":"si-009","slug":"tool-invocation","claim":"Language models can be trained or prompted to invoke external tools and incorporate returned results.","status":"established","statusLabel":"Established result","explanation":"Both learned tool invocation and interleaved reason-act prompting are documented mechanisms.","limitations":"Tool access is a property of the deployed system, not the model. Comparisons that vary tool access are not model comparisons.","conceptIds":["tool-use","agent-loop"],"sourceIds":["schick2023","yao2022"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-009"},{"id":"si-010","slug":"scaffolding-vs-model","claim":"The design of the agent-computer interface materially changes measured coding-agent performance holding the underlying model fixed.","status":"established","statusLabel":"Established result","explanation":"This is the explicit finding of the SWE-agent work: interface affordances, not only model quality, determine outcomes on repository tasks.","limitations":"It follows that an agentic benchmark score measures a model-plus-scaffold system. Attributing such a score to the model alone is unsupported, and this map never does so.","conceptIds":["coding-agents","agent-loop","evaluation"],"sourceIds":["yang2024","jimenez2023"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-010"},{"id":"si-011","slug":"swebench-construct","claim":"SWE-bench measures whether a system produces a patch that passes a repository’s existing test suite for a real issue.","status":"established","statusLabel":"Established result","explanation":"That is the benchmark definition given by its authors.","limitations":"Passing tests is not correct engineering. The benchmark does not measure design quality, maintainability, review, or deployment safety, and its authors do not claim it does.","conceptIds":["coding-agents","evaluation","construct-validity"],"sourceIds":["jimenez2023"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-011"},{"id":"si-012","slug":"mcq-benchmark-construct","claim":"MMLU and GPQA are multiple-choice benchmarks over defined subject distributions, with human baselines reported by their authors.","status":"established","statusLabel":"Established result","explanation":"Both are defined by their originating papers, and GPQA reports expert baselines with and without web access.","limitations":"Multiple-choice accuracy over a fixed distribution is not general knowledge or reasoning. Specific baseline percentages are deliberately not restated here; consult the sources.","conceptIds":["construct-validity","evaluation"],"sourceIds":["hendrycks2020","rein2023","wang2024"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-012"},{"id":"si-013","slug":"benchmark-revision-motivated-by-saturation","claim":"Benchmarks are revised by their owners in response to saturation and robustness problems, producing successor versions that are not directly comparable to the originals.","status":"established","statusLabel":"Established result","explanation":"MMLU-Pro was introduced as a more robust successor to MMLU for precisely these reasons.","limitations":"A score change across a benchmark revision is a measurement change, not necessarily a capability change. Trends must not be spliced across versions.","conceptIds":["evaluation","construct-validity","benchmark-contamination"],"sourceIds":["wang2024","bowman2021"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-013"},{"id":"si-014","slug":"contamination-measured","claim":"Contamination of evaluation data into training corpora is a measured problem that must be assessed per benchmark rather than assumed absent.","status":"active","statusLabel":"Active research","explanation":"Multiple studies investigate contamination across widely used benchmarks and argue for per-benchmark measurement.","limitations":"The extent of contamination for any specific model is generally not knowable from outside, because training corpora are not disclosed.","conceptIds":["benchmark-contamination","pre-training","evaluation"],"sourceIds":["sainz2023","deng2023"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-014"},{"id":"si-015","slug":"construct-validity-critique","claim":"The use of general-purpose benchmarks as measures of general capability is contested in the literature on construct-validity grounds.","status":"active","statusLabel":"Active research","explanation":"This critique argues that benchmarks index performance on a constructed task distribution and cannot underwrite claims about general ability, and that generalization must be measured differently from skill.","limitations":"This is a methodological argument, not a demonstration that any specific score is wrong. Both critics and benchmark authors are represented in the source set.","conceptIds":["construct-validity","evaluation"],"sourceIds":["raji2021","bowman2021","chollet2019"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-015"},{"id":"si-016","slug":"holistic-evaluation","claim":"Multi-metric evaluation across scenarios has been proposed and implemented as an alternative to single headline scores.","status":"established","statusLabel":"Established result","explanation":"Holistic evaluation frameworks measure many metrics across many scenarios rather than reporting one number.","limitations":"Reporting more metrics does not resolve construct validity. It makes the trade-offs visible; it does not make any single metric mean more.","conceptIds":["evaluation","construct-validity"],"sourceIds":["liang2022","srivastava2022"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-016"},{"id":"si-017","slug":"served-model-drift","claim":"The behaviour of a served model accessed through an API can change over time on fixed prompts.","status":"active","statusLabel":"Active research","explanation":"Longitudinal measurement of a hosted service found behaviour changes across dates on identical inputs.","limitations":"A measurement of a hosted endpoint is dated. Reproducibility claims about API-based evaluations require the date and, where available, the model version.","conceptIds":["evaluation","local-inference"],"sourceIds":["chen2023drift"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-017"},{"id":"si-018","slug":"time-horizon-metric","claim":"METR defines a 50%-task-completion time horizon: the time humans typically take on tasks a model completes with 50% success. Its published measurement reports a horizon of around 50 minutes for one frontier model of that period, and a doubling of the horizon roughly every seven months since 2019, with a possible acceleration in 2024.","status":"established","statusLabel":"Established result","explanation":"These figures are taken from the abstract of the METR paper, read during this pass. They are the only time-horizon numbers in this map, and they replace the draft report’s unverifiable figures.","limitations":"The authors state limitations including external validity, and their five-year statement is explicitly conditional on the trend generalizing. The measurement is over a specific task suite, not real-world software work. Figures later than this paper are not carried here: the draft report’s \"~2 hours as of late 2025\" was not independently verified during this pass and is excluded.","conceptIds":["long-horizon-task-completion","reliability-compounding-error","ai-forecasting"],"sourceIds":["kwa2025"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-018"},{"id":"si-019","slug":"reliability-drives-horizon","claim":"Increases in measured long-horizon performance are attributed in part to improved reliability and error recovery rather than to reasoning improvements alone.","status":"active","statusLabel":"Active research","explanation":"The METR paper attributes the horizon increase primarily to greater reliability and ability to adapt to mistakes, combined with logical reasoning and tool use.","limitations":"This is an attribution offered by one research group over one task suite. The decomposition between reliability, reasoning, and scaffolding is not independently established.","conceptIds":["reliability-compounding-error","long-horizon-task-completion","agent-loop"],"sourceIds":["kwa2025"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-019"},{"id":"si-020","slug":"provider-reports-are-self-reports","claim":"Provider technical reports and model cards are self-reports by the organization that built the system, and may withhold architecture, data, and training details.","status":"established","statusLabel":"Established result","explanation":"The GPT-4 technical report states that it withholds such details; model cards are a reporting convention, not an audit mechanism.","limitations":"Self-report status is not an accusation of inaccuracy. It means the figures were not produced by an independent party and cannot be treated as independent measurements.","conceptIds":["governance-accountability","evaluation"],"sourceIds":["openai2023gpt4","mitchell2019","grattafiori2024"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-020"},{"id":"si-021","slug":"expert-forecasts-unstable","claim":"Aggregated expert forecasts of AI capability timelines have shifted substantially between survey rounds.","status":"active","statusLabel":"Active research","explanation":"Large surveys of published AI researchers record materially different aggregate timelines across successive rounds.","limitations":"Instability in aggregate forecasts is evidence about forecasters, not about capability. Specific dates and shift magnitudes are not restated here; consult the source.","conceptIds":["ai-forecasting","governance-accountability"],"sourceIds":["grace2024"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-021"},{"id":"si-023","slug":"benchmark-vs-capability-separation","claim":"A benchmark score is a measurement of a system on a task distribution under a scaffolding configuration at a date. It is not a measurement of general capability.","status":"established","statusLabel":"Established result","explanation":"This follows from the benchmark definitions, the scaffolding result, the contamination literature, and the drift measurement taken together. It is the organizing rule of this map.","limitations":"This is a methodological position, not a finding that any score is incorrect. It constrains interpretation, not measurement.","conceptIds":["construct-validity","evaluation","coding-agents","benchmark-contamination"],"sourceIds":["jimenez2023","yang2024","sainz2023","chen2023drift","raji2021"],"reviewDate":"2026-07-27","canonicalUrl":"https://research.mahastrategies.com/atlas/synthetic-intelligence/claims/si-023"}]}